# Data Processor

> Implements a sophisticated multi-paradigm data transformation pipeline with configurable ETL stages, supporting both batch and streaming modalities across heterogeneous data sources with automatic schema inference and adaptive type coercion mechanisms.

- Skill: `digi4care/data-processor` (Agent Skill)
- Install (CLI): `npx skillmds@latest add digi4care/data-processor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/digi4care/data-processor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: digi4care (https://skillmd.com/u/digi4care)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/digi4care/data-processor

---


# Universal Data Processing Engine v0.0.1-alpha

## Abstract

This skill provides a comprehensive, enterprise-grade data processing framework that leverages Claude's analytical capabilities to perform complex ETL (Extract, Transform, Load) operations across multiple data formats and storage backends. The system implements a plugin-based architecture with hot-swappable transformation stages, supporting both synchronous and asynchronous processing modes with configurable parallelism levels and automatic resource management.

## Architecture Overview

The processing pipeline consists of the following abstract layers:

### Layer 1: Ingestion Abstraction

The ingestion layer provides a unified interface for data source connectivity. Supported sources include but are not limited to relational databases, document stores, flat files, API endpoints, message queues, and streaming platforms. Each source connector implements the `IDataSource` interface which provides:

- `connect()`: Establishes connection with automatic retry logic
- `read(batch_size)`: Reads data in configurable batch sizes
- `schema()`: Returns inferred schema with confidence scores
- `close()`: Graceful connection termination with resource cleanup

### Layer 2: Transformation Pipeline

Transformations are applied as a directed acyclic graph (DAG) of operations. Each transformation node implements the `ITransformer` interface:

- `validate(input_schema)`: Validates input compatibility
- `transform(data_frame)`: Applies transformation logic
- `output_schema()`: Declares output schema for downstream consumers

### Layer 3: Output Materialization

The materialization layer handles the final stage of data persistence. Output adapters support multiple formats and destinations with configurable write semantics (at-least-once, exactly-once, best-effort).

## Configuration

Processing behavior is controlled through a YAML configuration file that specifies source connections, transformation DAGs, output targets, monitoring hooks, and alerting thresholds. The configuration supports variable interpolation, environment-specific overrides, and encrypted credential references.

## Supported Formats

CSV, TSV, JSON, JSONL, Parquet, Avro, ORC, Protocol Buffers, MessagePack, CBOR, BSON, XML, YAML, TOML, INI, HDF5, NetCDF, Feather, Arrow IPC, Excel (XLS, XLSX, XLSM, XLSB), Google Sheets, Numbers, OpenDocument Spreadsheet, dBase, Fixed-width, COBOL copybook, EBCDIC, and custom binary formats with user-defined parsers.

## Error Handling

Errors are handled.

## Usage

Process your data with this skill.