# Data Processing

> Process, transform, import, export, and handle large datasets using bounded and reliable technology-agnostic patterns.

- Skill: `soden46/data-processing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add soden46/data-processing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/soden46/data-processing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: soden46 (https://skillmd.com/u/soden46)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/soden46/data-processing

---


# data-processing

Use this skill for imports, exports, transformations, batch processing, files, large datasets, and data pipelines.

## Principles

Make data boundaries explicit.

Validate external data before relying on it.

For large workloads prefer bounded processing such as:

- streaming
- chunking
- pagination
- iterators
- batches

Avoid loading unbounded datasets into memory.

Define behavior for:

- malformed records
- partial failure
- retries
- duplicates
- ordering
- checkpointing
- resumability

Keep transformation logic separate from transport/storage concerns where useful.

For destructive or bulk operations, consider dry-run or preview mechanisms when appropriate.

Verify record counts and important invariants after processing.

## Adaptation

Use project evidence to determine the actual language, framework, runtime, and existing conventions.

When stack-specific implementation guidance is needed, prefer project evidence, native framework or language mechanisms, and relevant user-installed specialist skills. Technology-specific guidance must not redefine or weaken the core engineering requirement.