# Custom Code Execution In Data Pipeline

> Use when when standard conversion directives (headers, collate, fields_to_headers, exclusion_headers, values_to_str, sort_by, test) cannot express the required transformation logic, or when domain-specific aggregation, conditional logic, or value derivation must be applied to records after field.

- Skill: `holobiomicslab/custom-code-execution-in-data-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/custom-code-execution-in-data-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/custom-code-execution-in-data-pipeline/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/custom-code-execution-in-data-pipeline

---


# Custom Code Execution in Data Pipeline

## Summary

This skill enables the injection and execution of user-defined Python code within a data transformation pipeline to apply custom logic that cannot be expressed through declarative directives alone. It is essential for complex, non-standard data transformations in extract-validate-convert workflows where built-in field mapping, filtering, and collation are insufficient.

## When to use

When standard conversion directives (headers, collate, fields_to_headers, exclusion_headers, values_to_str, sort_by, test) cannot express the required transformation logic, or when domain-specific aggregation, conditional logic, or value derivation must be applied to records after field mapping and filtering but before final output. Trigger: the conversion specification includes a 'code' sub-field, or the output schema requires computed fields or context-dependent values.

## When NOT to use

- Input data does not require custom transformation logic; standard directives (headers, collate, test, sort_by) are sufficient.
- Code is computationally expensive or performs I/O operations; custom code execution blocks the pipeline synchronously and may not scale to large datasets.
- Transformation logic is proprietary or untrusted; executing arbitrary user-supplied Python code introduces security risk without sandboxing or code review.

## Inputs

- Partially-transformed record list (list of dictionaries) after field-to-header mapping, filtering, collation, and sorting
- Python code string from 'code' sub-field of conversion directive
- Conversion directive configuration (headers, fields_to_headers, collate, exclusion_headers, values_to_str, sort_by, sort_order, test)

## Outputs

- Final transformed record list (list of dictionaries) with custom code applied
- Each record may contain computed or modified fields not present in input JSON

## How to apply

After applying field-to-header mapping, filtering (via test conditions and exclusion_headers), collation, and sorting in the matrix directive handler, insert the user-provided Python code from the 'code' sub-field to execute custom transformations on each output record or the full result set. The code receives the partially-transformed records (dictionaries with mapped keys and excluded fields removed) as input and can modify values, add derived fields, or apply domain-specific logic. The code must be syntactically valid Python and should operate on the record structure produced by prior directive steps. Execution order matters: sorting and custom code are applied last, after collation and filtering, ensuring the code operates on the final pre-output state. Validate that the code does not raise exceptions on sample records and that output records match the expected schema after code execution.

## Related tools

- **MESSES (Metadata from Experimental SpreadSheets Extraction System)** (Pipeline framework that orchestrates the extract-validate-convert workflow and executes custom code sub-fields within matrix directive handlers during JSON-to-format conversion) — https://github.com/MoseleyBioinformaticsLab/messes
- **Python** (Runtime environment for parsing, validating, and executing custom code strings supplied in conversion directives)

## Examples

```
messes convert mwtab data.json output_data --directives conversion_directives.json
```

## Evaluation signals

- Custom code executes without runtime exceptions on all records in the input list.
- Output records contain all expected fields from prior directive steps plus any new fields added by custom code.
- Output records conform to the target schema specified in the conversion directive (e.g., mwTab format schema).
- Derived or modified field values in output records are logically consistent with the custom code logic (spot-check a sample of records).
- Record count and field cardinality remain within expected bounds; no records are dropped or duplicated unless explicitly intended by the code.

## Limitations

- Custom code is executed synchronously within the pipeline; performance scales linearly with record count and code complexity.
- No sandboxing or capability restriction; user-supplied code has full access to Python's standard library and system resources, posing security risk in untrusted environments.
- Errors in custom code (e.g., syntax errors, unhandled exceptions, type mismatches) halt the entire conversion pipeline; robust error handling and testing of custom code prior to pipeline execution is essential.
- Custom code sees only the partially-transformed record state after collation and filtering; it cannot access the original input JSON or intermediate transformation states if needed for validation or debugging.

## Evidence

- [other] The matrix directive handler operates by iterating over records in a specified input JSON table to build dictionaries, with capabilities to sort and filter records using sort_by, sort_order, and test fields; collate records by a grouping field; map record fields to dictionary keys via headers; exclude fields with exclusion_headers; convert values to strings with values_to_str; or execute custom Python code for complex transformations.: "or execute custom Python code for complex transformations"
- [other] 9. Validation: verify the output contains the expected number of records, all specified headers are present in each dictionary, and data types match the conversion directives configuration.: "Validation: verify the output contains the expected number of records, all specified headers are present in each dictionary, and data types match"
- [other] 8. Execute any custom transformation code provided in the code sub-field to apply user-defined logic to the output records.: "Execute any custom transformation code provided in the code sub-field to apply user-defined logic to the output records"
- [other] Parse the conversion directives file to extract the matrix-type directive configuration, including headers, collate, fields_to_headers, exclusion_headers, values_to_str, sort_by, sort_order, test, and code sub-fields.: "including headers, collate, fields_to_headers, exclusion_headers, values_to_str, sort_by, sort_order, test, and code sub-fields"
- [intro] To support the JSON-to-JSON conversion a relatively simple set of directives were developed: "To support the JSON-to-JSON conversion a relatively simple set of directives were developed"