Batch Normalize And Package

Batch Normalize and Package

Philosophy

Running normalize-community-skill one file at a time is correct for a single import; running it in a loop without orchestration is how pipelines die mid-batch and leave the catalog in an inconsistent state. This skill exists to make the full community-to-Dojo pipeline — acquire, scan, normalize, package, manifest — repeatable, auditable, and recoverable. Every step produces a durable artifact (the scan catalog, the staging directory, the manifest) so a failure at Step 4 can be resumed from Step 3, not from the beginning. Source attribution is preserved end-to-end because provenance is the only thing that makes a batch import trustworthy enough to act on.

When to Use

• Importing skills from one or more community repositories into the Dojo ecosystem
• Running the weekly or periodic supply-chain refresh on a set of watched repos
• Onboarding a new external skill source for the first time
• Producing a signed distribution manifest for audit or sharing with another operator

Do not use for single-file imports — use normalize-community-skill directly for that.

I. Workflow

This is a 6-step DAG with dependency structure:

Step 1: Acquire sources (parallel per repo)
    |
Step 2: Scan all sources (invokes scan-community-repos)
    |
Step 3: Filter catalog (select normalizable + ready skills)
    |
Step 4: Normalize (parallel per skill, invokes normalize-community-skill)
    |
Step 5: Package (invokes dojo skill package-all on normalized directory)
    |
Step 6: Generate distribution manifest

Step 1: Acquire Sources

For each input repo:

1. If GitHub URL: shallow clone (git clone --depth=1) to temp directory
2. If local path: validate exists, use directly
3. If already cloned (cache hit): git pull --ff-only to update
4. Record: repo name, local path, acquisition timestamp

Parallelization: All repos can be acquired simultaneously. No dependencies.

Step 2: Scan All Sources

Invoke scan-community-repos with the list of acquired paths.

Input: list of {repo_name, local_path} pairs Output: JSON catalog with per-file classification (ready / normalizable / incompatible)

Step 3: Filter Catalog

From the scan catalog, build the work queue:

Decision criteria for "normalizable":

• Has name field in frontmatter (required — cannot be inferred)
• Has description field (required — cannot be inferred)
• Has markdown body with at least one section heading
• File size > 100 bytes and < 50KB (too small = stub, too large = not a skill)

Step 4: Normalize

For each skill in the normalization queue, invoke normalize-community-skill:

Input: path to the community SKILL.md Output: enriched SKILL.md with Dojo-compatible frontmatter

Parallelization: All normalizations are independent. Run in parallel batches of 10.

Error handling:

• If normalization fails (malformed YAML, encoding issues): log error, mark as "failed" in manifest, continue
• If normalized skill still fails IsValid() simulation: log warning, mark as "needs-manual-review"

Step 5: Package

Run dojo skill package-all <staging-directory> on the combined staging directory containing:

• Ready skills (copied as-is from Step 3)
• Normalized skills (output from Step 4)

This produces CAS entries: config blob (SkillManifest JSON) + content blob (tar archive) for each skill.

Validation: After packaging, verify each skill has both CAS tags:

• skill/{name}:config @ {version}
• skill/{name}:content @ {version}

Step 6: Generate Distribution Manifest

Produce a manifest document with:

# Skill Distribution Manifest
Generated: {timestamp}
Source repos: {count}

## Summary
- Scanned: {total_files}
- Ready (no normalization needed): {count}
- Normalized successfully: {count}
- Failed normalization: {count}
- Incompatible (skipped): {count}
- Packaged into CAS: {count}

## Packaged Skills
| Name | Source Repo | Tier | CAS Config Hash | CAS Content Hash |
|------|-----------|------|-----------------|------------------|
| ... | ... | ... | sha256:... | sha256:... |

## Skipped / Failed
| Name | Source Repo | Reason |
|------|-----------|--------|
| ... | ... | "missing name field" / "normalization error: ..." |

Also generate machine-readable manifest.json:

{
  "generated": "2026-04-05T...",
  "source_repos": [...],
  "skills": [
    {
      "name": "skill-name",
      "source_repo": "org/repo",
      "tier": 2,
      "cas_config_hash": "sha256:...",
      "cas_content_hash": "sha256:...",
      "status": "packaged"
    }
  ]
}

II. Best Practices

1. Run scan before normalize. Never normalize blindly — the scan catalog prevents wasted effort on incompatible files.

2. Preserve source attribution. Every packaged skill must retain its source repo in the manifest. Provenance is non-negotiable for trust.

3. Fail gracefully, report completely. A single malformed skill should never abort the entire pipeline. Log it, skip it, include it in the manifest as failed, continue.

4. Idempotent runs. Running the pipeline twice on the same repos should produce the same CAS hashes (content-addressed storage guarantees this). Use this property for verification.

5. Respect rate limits. When cloning from GitHub, introduce a small delay between clones to avoid API rate limiting. Use shallow clones to minimize bandwidth.

6. Stage before packaging. Always copy/normalize into a clean staging directory. Never modify source repos in-place.

III. Quality Checklist

• All source repos acquired successfully (or failures logged)
• Scan catalog generated with correct classifications
• No "normalizable" skills skipped without attempting normalization
• All normalized skills pass IsValid() simulation
• dojo skill package-all completes without errors
• Every packaged skill has both CAS tags (config + content)
• Distribution manifest includes ALL skills (packaged, failed, and skipped)
• manifest.json is valid JSON and parseable
• Source attribution preserved for every entry
• Pipeline is idempotent (re-run produces same hashes)

Output

• Normalized skill files in a clean staging directory (source repos untouched)
• CAS entries for each packaged skill: skill/{name}:config and skill/{name}:content tags at the resolved version hash
• skill-distribution-manifest.md — human-readable summary of scanned, normalized, failed, and skipped counts
• manifest.json — machine-readable manifest with CAS hashes and source attribution for every entry
• Pipeline run logged with timestamps for each step

Examples

Scenario 1: "Import community skills from alirezarezvani/claude-skills and slavingia/skills" → Shallow-clone both repos in parallel, invoke scan-community-repos on the combined paths, filter catalog, normalize all normalizable skills in parallel batches of 10, run dojo skill package-all on staging, generate manifest.

Scenario 2: "Run the weekly supply chain refresh on our watched repos" → Pull latest from cached clones (git pull --ff-only), re-scan, normalize only skills not already in CAS (idempotent check), package new additions, append to existing manifest.

Edge Cases

• A single repo clone fails (network error, private repo) — log the failure, skip that source, continue with the rest; name the failed repo explicitly in the manifest's skipped section
• A skill fails normalization (malformed YAML, encoding issues) — mark as "failed" in the manifest with the error message, continue; do not abort the pipeline
• A normalized skill still fails IsValid() after normalization — mark as "needs-manual-review" in the manifest; do not attempt to package it
• Re-run on the same repos — content-addressed storage guarantees the same input produces the same hash; use this as a verification check, not a guard against re-running

Anti-Patterns

• Modifying source repos in place: All normalization must happen in the staging directory — never edit the cloned source repos directly
• Aborting on a single failure: One malformed skill file should never stop the pipeline; log, skip, and continue
• Omitting source attribution: Every entry in the manifest must name its source repo — provenance is required for trust and for rollback decisions
• Running without a prior scan: Normalizing all files blindly wastes compute on incompatible files; always let scan-community-repos filter the work queue first

IV. Related Skills

• scan-community-repos — Run this first to produce the JSON catalog that this skill consumes; batch-normalize-and-package depends on the catalog to know what to normalize vs. skip
• normalize-community-skill — The per-file normalization skill invoked in Step 4 of this pipeline; use it directly for single-file imports
• skill-creation — Use when a community skill is so incomplete it needs a full rewrite rather than normalization; route incompatible skills here after manual review
• skill-maintenance — Use after a batch import to rename or refactor skills that don't follow verb-object naming conventions