TBC Schema Updater

TBC Schema Updater

This skill manages the extraction and regeneration of JSON schemas from the TBC Kicker source, keeping the validation schemas synchronized with new To-Be-Continuous releases.

Overview

The TBC Kicker skill uses JSON schemas to validate .gitlab-ci.yml inputs against real template variables. When TBC releases new versions with updated variables, features, or templates, regenerate the schemas to keep validation accurate.

Prerequisites

Before regenerating schemas, ensure the kicker source is available:

# Clone kicker repository if not present
git clone https://gitlab.com/to-be-continuous/kicker.git /tmp/kicker

# Or update existing clone
cd /tmp/kicker && git pull

The source file is located at:

/tmp/kicker/src/assets/kicker-aggregated.json

Schema Regeneration Workflow

Step 1: Verify Source

Check that the kicker source is up to date:

cd /tmp/kicker
git log -1 --oneline

Step 2: Run Extraction

Execute the extraction script:

cd /path/to/gitlab-tbc/skills/tbc-kicker
python3 scripts/extract-schemas.py \
  --input /tmp/kicker/src/assets/kicker-aggregated.json \
  --output-dir schemas/

Step 3: Verify Results

Check the generated schemas:

# Count schemas (should be ~50)
ls schemas/*.json | wc -l

# Verify a specific schema
python3 scripts/validate-inputs.py --list-valid aws

Step 4: Validate Examples

Run validation on existing examples:

for f in examples/*.yml; do
  python3 scripts/validate-inputs.py "$f"
done

Extraction Script

The scripts/extract-schemas.py script handles:

1. Variable name transformation: AWS_CLI_IMAGE → cli-image
2. Type inference: string, boolean, enum, number
3. Feature variables: Grouped under x-feature metadata
4. Variant variables: Grouped under x-variant metadata
5. Schema metadata: prefix, kind, features, variants lists
6. Component names: Extracts valid component names from template_path (base + variants)

Script Options

python3 scripts/extract-schemas.py --help

Options:
  --input, -i     Path to kicker-aggregated.json
  --output-dir, -o   Output directory for schemas
  --template, -t     Extract only a specific template

Single Template Extraction

To extract or update a single template:

python3 scripts/extract-schemas.py \
  --input /tmp/kicker/src/assets/kicker-aggregated.json \
  --output-dir schemas/ \
  --template python

Schema Format

Each generated schema follows this structure:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Template Name",
  "description": "to-be-continuous/project-path",
  "type": "object",
  "properties": {
    "inputs": {
      "type": "object",
      "properties": {
        "input-name": {
          "type": "string",
          "description": "...",
          "default": "...",
          "x-feature": "Feature Name",
          "x-variant": "Variant Name",
          "x-advanced": true
        }
      },
      "additionalProperties": false
    }
  },
  "_meta": {
    "prefix": "PREFIX_",
    "kind": "build|hosting|analysis",
    "features": ["Feature1", "Feature2"],
    "variants": ["Variant1", "Variant2"],
    "variable_count": 25
  }
}

Handling New Templates

When TBC adds new templates:

1. Run full extraction (new templates auto-discovered)
2. Verify new schema is correct
3. Update tbc-kicker SKILL.md if needed (new template categories)
4. Add example configuration if significant

Validation Levels

The validate-inputs.py script performs 3-level validation:

Validation Order

1. Parse component string: to-be-continuous/{project}/{component}@{version}
2. Find template by project.path in _meta.json
3. Check component exists in components[]
4. Check version exists in project.tags[]
5. Validate inputs against schema

Troubleshooting

Schema Not Found

If validation reports "No schema found for template":

1. Check if template uses different project path
2. Verify schema exists in schemas/ directory
3. Re-run extraction if missing

Variable Name Mismatch

The script transforms variable names:

• Strips prefix: PYTHON_IMAGE → image
• Lowercase: IMAGE → image
• Hyphens: BUILD_SYSTEM → build-system

If mismatch occurs, check the _meta.prefix in schema.

Missing Features/Variants

Features and variants are extracted from the kicker JSON. If missing:

1. Verify kicker source is updated
2. Check feature/variant structure in kicker-aggregated.json

Additional Resources

Reference Files

• references/kicker-structure.md - Structure of kicker-aggregated.json

Scripts

• scripts/extract-schemas.py - Main extraction script (shared with tbc-kicker)

_meta.json Structure

The extraction script generates schemas/_meta.json with template metadata:

{
  "generated_from": "/tmp/kicker/src/assets/kicker-aggregated.json",
  "template_count": 50,
  "templates": {
    "python": {
      "name": "Python",
      "project": {
        "path": "to-be-continuous/python",
        "tag": "7.5.2",
        "tags": ["7.5.2", "7.5", "7", ...]
      },
      "variable_count": 48,
      "components": ["gitlab-ci-python", "gitlab-ci-python-vault", "gitlab-ci-python-gcp"]
    }
  }
}

Key Fields

Component Name Format

Component names are extracted from template_path:

• templates/gitlab-ci-python.yml → gitlab-ci-python
• templates/gitlab-ci-python-vault.yml → gitlab-ci-python-vault

The validation script uses this to verify component names in .gitlab-ci.yml:

# Correct format
- component: $CI_SERVER_FQDN/to-be-continuous/python/gitlab-ci-python@7

# NOT: python/python@7 (missing gitlab-ci- prefix)

Recommended Workflow

1. Watch TBC releases: Monitor https://gitlab.com/to-be-continuous for updates
2. Update kicker clone: git pull in /tmp/kicker
3. Regenerate schemas: Run extraction script
4. Validate examples: Ensure existing examples still pass
5. Fix breaking changes: Update examples if variables renamed/removed
6. Document changes: Note significant schema changes