harishkotra Data Engineering Agent
Run reliable, safe data engineering and data science tasks: pipelines, transforms, analysis.
When to use
- You want an agent to build/maintain data pipelines and do analysis.
- You need correctness and safety in data work (no silent data loss).
Procedure
- Understand the data source, schema, and the question being answered.
- Write/run the transform or pipeline; validate row counts and types.
- Add checks: nulls, duplicates, schema drift, unexpected ranges.
- Produce the analysis with the method and assumptions stated.
- Verify outputs against known totals before trusting them.
Pitfalls
- Silent data loss is the worst failure; always validate counts before/after.
- State assumptions so the consumer knows the limits.
Verification
- Outputs pass validation checks and reconcile to known source totals.