# JSON File Parsing

> Use when when you have JSON-formatted curation data organized in a directory structure (e.g., the MIBiG `data` directory) and need to extract a specific nested field (e.g., `cluster.status`) across all entries to build an index, summary table, or quality report.

- Skill: `holobiomicslab/json-file-parsing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add holobiomicslab/json-file-parsing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/holobiomicslab/json-file-parsing/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- License: CC-BY-4.0
- Author: HolobiomicsLab (https://skillmd.com/u/holobiomicslab)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/holobiomicslab/json-file-parsing

---


# json-file-parsing

> **License: restricted** — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->
## Summary

Parse and extract structured fields from JSON files distributed across a directory tree, aggregating results into a tabular format. This skill is essential when source data is curated in JSON format with nested schema (like MIBiG cluster annotations) and you need to index or summarize a particular field across all records.

## When to use

When you have JSON-formatted curation data organized in a directory structure (e.g., the MIBiG `data` directory) and need to extract a specific nested field (e.g., `cluster.status`) across all entries to build an index, summary table, or quality report.

## When NOT to use

- JSON files are already pre-indexed or aggregated into a single database or API endpoint — use direct database/API query instead.
- Target field is deeply nested or conditional on complex schema variants — may require custom schema mapping rather than simple dot-notation extraction.
- File count is extremely large (>1M files) and directory I/O becomes a bottleneck — consider batch processing or stream-based parsing.

## Inputs

- Directory tree containing JSON files (e.g., github.com/mibig-secmet/mibig-json `data` directory)
- Target nested field path as a dot-notation string (e.g., 'cluster.status')

## Outputs

- Tabular data file (CSV) with columns: entry_id, extracted_field_value
- Aggregated summary indexed by entry and field value

## How to apply

Recursively scan the target directory for all JSON files. For each file, parse the JSON structure and navigate to the target nested field using dot notation (e.g., `cluster.status`). Extract the field value along with the entry identifier. Aggregate all extracted records into a structured table (with columns for entry_id and the target field). Export the aggregated table to CSV or other tabular format. Validate the output by spot-checking a sample of entries against the original JSON files to confirm field extraction accuracy and absence of parsing errors.

## Related tools

- **Python json module** (Parse JSON structure and navigate nested fields)
- **pandas** (Aggregate parsed records into tabular data and export to CSV)

## Examples

```
import json, os, pandas as pd; records = []; [records.append({'entry_id': f.split('.')[0], 'cluster_status': json.load(open(os.path.join('data', f))).get('cluster', {}).get('status')}) for f in os.listdir('data') if f.endswith('.json')]; pd.DataFrame(records).to_csv('entry_status_index.csv', index=False)
```

## Evaluation signals

- Output CSV contains no blank rows and row count matches total JSON files scanned
- Spot-check: randomly select 5–10 entries from output and verify extracted field values match the original JSON files
- No parsing errors or exceptions logged during directory traversal and JSON deserialization
- All expected entry identifiers are present in the output; no entries silently dropped
- CSV schema matches specification: entry_id column is unique and non-null; extracted_field_value column contains valid values for the field type (e.g., status is a recognized categorical value)

## Limitations

- Assumes well-formed JSON and consistent nested schema across all files; malformed or schema-variant entries may cause parsing errors or silent omissions.
- Dot-notation field extraction is brittle if schema evolves or if field is conditionally present; requires manual schema validation before applying this skill at scale.
- No changelog or version tracking in the MIBiG repository was found, so field semantics or structure may change without notice between data pulls.

## Evidence

- [readme] The current datasets for [MIBiG] live in the `data` directory, entry status is now tracked via the `cluster.status` field.: "The current datasets for [MIBiG] live in the `data` directory, entry status is now tracked via the `cluster.status` field."
- [other] MIBiG curation data is maintained in JSON format with entry status tracked through the `cluster.status` field within each annotation record.: "MIBiG curation data is maintained in JSON format with entry status tracked through the `cluster.status` field"
- [other] Recursively scan the `data` directory for all JSON files. Parse each JSON file and extract the `cluster.status` field value for each entry. Aggregate entries into a table grouped by status.: "Recursively scan the `data` directory for all JSON files. 2. Parse each JSON file and extract the `cluster.status` field value for each entry. 3. Aggregate entries into a table"
- [other] Python (json module for parsing JSON files), pandas or equivalent tabular data library: "Python (json module for parsing JSON files), pandas or equivalent tabular data library"