# Data Pipeline Processor Do

> Sub-skill of data-pipeline-processor: Do (+6).

- Skill: `vamseeachanta/data-pipeline-processor-do` (Agent Skill)
- Install (CLI): `npx skillmds@latest add vamseeachanta/data-pipeline-processor-do`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vamseeachanta/data-pipeline-processor-do/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: vamseeachanta (https://skillmd.com/u/vamseeachanta)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/vamseeachanta/data-pipeline-processor-do

---


# Do (+6)

## Do


1. Always detect encoding before reading CSV
2. Use chunked reading for large files (>100MB)
3. Specify dtypes to reduce memory usage
4. Handle missing values explicitly
5. Validate early in the pipeline
6. Fail fast on critical errors
7. Log warnings for non-critical issues
8. Track validation statistics


## Don't


1. Assume encoding is always UTF-8
2. Load entire large files into memory
3. Skip validation steps
4. Ignore encoding errors
5. Mix transformation and validation


## Data Reading

- Always detect encoding before reading CSV
- Use chunked reading for large files (>100MB)
- Specify dtypes to reduce memory usage
- Handle missing values explicitly


## Validation

- Validate early in the pipeline
- Fail fast on critical errors
- Log warnings for non-critical issues
- Track validation statistics


## Transformation

- Use method chaining for readability
- Apply filters before expensive operations
- Convert types early to catch errors
- Document transformation logic


## Export

- Create output directories automatically
- Use appropriate formats (Parquet for large data)
- Include metadata in output
- Verify output integrity


## File Organization

```
project/
    config/
        pipelines/           # Pipeline configs
            clean_data.yaml
            aggregate.yaml
    data/
        raw/                 # Raw input data
        processed/           # Cleaned data
        results/             # Analysis results
    src/
        data_pipeline/       # Pipeline code
    scripts/
        run_pipeline.sh      # CLI wrapper
```