Kelly Local Model Lab
Use Busabase as the durable data and control plane while a trusted local Worker
performs compute on Apple Silicon. Fine-tuning changes behavior; it is not a
replacement for RAG when the goal is to inject current knowledge.
Mandatory Dependencies
Before creating, provisioning, or changing the AirApp, read and follow:
$kelly-app-skill-creator for product behavior and UI quality;
$busabase for Space selection, node discovery, ChangeRequests, and trusted writes;
$busabase-app-creator for AirApp runtime, SDK, security, validation, and deployment;
$kelly-app-skill-creator-tests before claiming full conformance.
If a dependency is unavailable, keep the canonical local artifact and training
pipeline intact, stop before the unavailable Busabase operation, and report the
missing dependency. Never replace Busabase with SQLite, local JSON, or another
persistent workflow backend.
Operating Boundary
- The AirApp reviews examples, runs, evaluations, and registered adapters. It
never launches MLX, reads local model files, or performs training itself.
- A trusted local Agent/Worker claims one eligible
training-runs record,
downloads its immutable Drive snapshot, trains locally, writes heartbeats,
uploads the adapter and report, and then marks the run terminal.
- Busabase Base records hold structured state and provenance. Drive/File holds
dataset snapshots, adapters, reports, and model cards. Vault holds only named
credential requirements; values never enter the browser.
- Base-model caches are disposable local compute caches. They are not canonical
workflow state and need not be uploaded to Busabase.
- A deployed AirApp cannot call the Mac's
localhost. Inference or training
requests cross the same reviewed Busabase job boundary.
Read references/lab-schema.md before operating on
resources, exporting data, claiming work, evaluating, or registering an adapter.
Default Experiment
For a first Apple Silicon validation, use:
- base model:
mlx-community/Qwen3-0.6B-4bit for a fast smoke run;
- method: QLoRA through MLX-LM;
- task: short App-in-Skill brief to strict
app_spec JSON;
- metrics: JSON validity, schema validity, exact-field accuracy, and latency;
- output: a LoRA adapter, never a rewritten foundation checkpoint.
For customer-support training, use task support_qa. Approved QA pairs from
$kelly-support cross the trusted bridge into training-examples with their
source article and pair ids, but return to needs_review before they can enter
a snapshot. The evaluator reports normalized exact-answer match and Unicode
character F1 instead of the app_spec JSON/schema metrics.
Treat the bundled training/fixture/ data as a pipeline smoke fixture only. It
never impersonates reviewed Busabase data. Production experiments must carry a
Drive snapshot reference and SHA-256 hash on the run record.
Workflow
- Inspect runtime readiness, the explicit Space, resource versions, and local
Worker readiness. Do not read or mutate app resources before Space selection.
- Review examples individually. Use
approve, request_changes, or block;
preserve the note, reviewer, timestamp, source, and content hash.
- Export only approved examples. Split by source or another leakage-resistant
grouping, write
train.jsonl, valid.jsonl, and locked test.jsonl, upload
one immutable snapshot to Drive, and record its SHA-256 digest.
- Create a
ready training run containing the exact base model and revision,
method, snapshot reference/hash, and config. Never train from a mutable live
query whose contents cannot be reproduced.
- Claim the run atomically. Record claimant, attempt, claimed time, and
heartbeat. Re-read eligibility before starting MLX.
- Evaluate the untouched base model first, train the adapter, then evaluate the
adapter on the identical locked test file and decoding settings.
- Upload the adapter, full evaluation report, and model card. Record terminal
status and sanitized errors. Do not mark a run done merely because MLX exited.
- Let a human choose
promote, hold, or reject on the adapter evaluation.
Promotion creates or updates a model-registry candidate/active record with
base revision, dataset hash, run id, and artifact reference.
Local Smoke Command
Use an isolated virtual environment. The installed MLX-LM version must be pinned
in requirements-mlx.txt before a result is treated as reproducible.
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-mlx.txt
.venv/bin/python scripts/mlx_smoke.py
Support QA handoff and training:
pnpm sync:support-qa
pnpm sync:support-qa -- --apply
.venv/bin/python scripts/mlx_smoke.py --task support_qa --data <snapshot-directory>
The bridge groups split assignment by source article so QA from one material
cannot leak across train and locked test. A single source can populate a review
queue, but it is not sufficient evidence for model promotion; collect at least
three independent source groups and keep one or more groups locked for testing.
Outputs go under the ignored .cache/smoke-run/ directory. The script validates
the fixture, hashes all splits, records the exact package version and command,
evaluates the baseline, trains, evaluates the adapter, and writes report.json.
UI
#/overview: readiness, approved split counts, active runs, candidate adapters,
pipeline stages, latest baseline/adapter delta, and recent runs.
#/examples[/<id>]: desktop list/detail review queue and separate phone list
and detail panes with stable example ids and review actions.
#/runs: base model, method, claim/status, heartbeat, artifacts, and failures.
#/evaluations: locked baseline/adapter comparisons and promotion verdicts.
#/registry: active, candidate, and retired adapters with complete lineage.
#/settings: responsive Help & Settings modal with guide, resources, and
sanitized training configuration.
Demo mode is explicit and in-memory only. ?demo=1 opens the overview;
?demo=dataset, ?demo=evaluations, and ?demo=registry open named screenshot
scenes. lang=en or lang=zh forces the UI language. Demo never claims a real
connection, writes Busabase, or proves the local Worker is online.
Completion
Do not claim a production fine-tune until:
- the snapshot came from reviewed Busabase examples and has a stable Drive ref/hash;
- the base model revision, MLX-LM version, command, and config are recorded;
- baseline and adapter used the same locked cases and decoding settings;
- adapter artifacts and reports exist and are referenced by a terminal run;
- the human promotion verdict is recorded against the evaluated version;
- app checks, local/OSS conformance, responsive browser checks, and available
Cloud/AirApp suites passed or are explicitly reported as skipped.
Stop when the Space or dataset is ambiguous, examples lack review/provenance,
the run changed after claim, test leakage is detected, the model license is
incompatible, an artifact hash fails, or a secret would cross into the browser.
1---2name: kelly-local-model-lab3description: Curate Busabase training examples, create immutable dataset snapshots, run reproducible LoRA or QLoRA fine-tunes on a local Apple Silicon Mac with MLX-LM, compare the untouched baseline and adapter on locked evaluations, and register approved adapters. Use when the user wants to train, fine-tune, evaluate, compare, promote, or manage a local language model with Busabase as the workflow source of truth. Do not use for RAG-only knowledge retrieval, cloud training, or training a foundation model from scratch.4license: MIT5metadata:6 source: https://skillmd.com/skills/mr-kelly/kelly-local-model-lab7 category: platform8 tags:9 - risk:local-write10 - surface:busabase11 busabase:12 template: true13 folderSlug: kelly-local-model-lab14 resources:15 - training-examples16 - training-runs17 - evaluations18 - model-registry19 - settings20 risk: local-write21---2223# Kelly Local Model Lab2425Use Busabase as the durable data and control plane while a trusted local Worker26performs compute on Apple Silicon. Fine-tuning changes behavior; it is not a27replacement for RAG when the goal is to inject current knowledge.2829## Mandatory Dependencies3031Before creating, provisioning, or changing the AirApp, read and follow:32331. `$kelly-app-skill-creator` for product behavior and UI quality;342. `$busabase` for Space selection, node discovery, ChangeRequests, and trusted writes;353. `$busabase-app-creator` for AirApp runtime, SDK, security, validation, and deployment;364. `$kelly-app-skill-creator-tests` before claiming full conformance.3738If a dependency is unavailable, keep the canonical local artifact and training39pipeline intact, stop before the unavailable Busabase operation, and report the40missing dependency. Never replace Busabase with SQLite, local JSON, or another41persistent workflow backend.4243## Operating Boundary4445- The AirApp reviews examples, runs, evaluations, and registered adapters. It46 never launches MLX, reads local model files, or performs training itself.47- A trusted local Agent/Worker claims one eligible `training-runs` record,48 downloads its immutable Drive snapshot, trains locally, writes heartbeats,49 uploads the adapter and report, and then marks the run terminal.50- Busabase Base records hold structured state and provenance. Drive/File holds51 dataset snapshots, adapters, reports, and model cards. Vault holds only named52 credential requirements; values never enter the browser.53- Base-model caches are disposable local compute caches. They are not canonical54 workflow state and need not be uploaded to Busabase.55- A deployed AirApp cannot call the Mac's `localhost`. Inference or training56 requests cross the same reviewed Busabase job boundary.5758Read [references/lab-schema.md](references/lab-schema.md) before operating on59resources, exporting data, claiming work, evaluating, or registering an adapter.6061## Default Experiment6263For a first Apple Silicon validation, use:6465- base model: `mlx-community/Qwen3-0.6B-4bit` for a fast smoke run;66- method: QLoRA through MLX-LM;67- task: short App-in-Skill brief to strict `app_spec` JSON;68- metrics: JSON validity, schema validity, exact-field accuracy, and latency;69- output: a LoRA adapter, never a rewritten foundation checkpoint.7071For customer-support training, use task `support_qa`. Approved QA pairs from72`$kelly-support` cross the trusted bridge into `training-examples` with their73source article and pair ids, but return to `needs_review` before they can enter74a snapshot. The evaluator reports normalized exact-answer match and Unicode75character F1 instead of the `app_spec` JSON/schema metrics.7677Treat the bundled `training/fixture/` data as a pipeline smoke fixture only. It78never impersonates reviewed Busabase data. Production experiments must carry a79Drive snapshot reference and SHA-256 hash on the run record.8081## Workflow82831. Inspect runtime readiness, the explicit Space, resource versions, and local84 Worker readiness. Do not read or mutate app resources before Space selection.852. Review examples individually. Use `approve`, `request_changes`, or `block`;86 preserve the note, reviewer, timestamp, source, and content hash.873. Export only approved examples. Split by source or another leakage-resistant88 grouping, write `train.jsonl`, `valid.jsonl`, and locked `test.jsonl`, upload89 one immutable snapshot to Drive, and record its SHA-256 digest.904. Create a `ready` training run containing the exact base model and revision,91 method, snapshot reference/hash, and config. Never train from a mutable live92 query whose contents cannot be reproduced.935. Claim the run atomically. Record claimant, attempt, claimed time, and94 heartbeat. Re-read eligibility before starting MLX.956. Evaluate the untouched base model first, train the adapter, then evaluate the96 adapter on the identical locked test file and decoding settings.977. Upload the adapter, full evaluation report, and model card. Record terminal98 status and sanitized errors. Do not mark a run done merely because MLX exited.998. Let a human choose `promote`, `hold`, or `reject` on the adapter evaluation.100 Promotion creates or updates a model-registry candidate/active record with101 base revision, dataset hash, run id, and artifact reference.102103## Local Smoke Command104105Use an isolated virtual environment. The installed MLX-LM version must be pinned106in `requirements-mlx.txt` before a result is treated as reproducible.107108```bash109python3 -m venv .venv110.venv/bin/python -m pip install -r requirements-mlx.txt111.venv/bin/python scripts/mlx_smoke.py112```113114Support QA handoff and training:115116```bash117pnpm sync:support-qa118pnpm sync:support-qa -- --apply119.venv/bin/python scripts/mlx_smoke.py --task support_qa --data <snapshot-directory>120```121122The bridge groups split assignment by source article so QA from one material123cannot leak across train and locked test. A single source can populate a review124queue, but it is not sufficient evidence for model promotion; collect at least125three independent source groups and keep one or more groups locked for testing.126127Outputs go under the ignored `.cache/smoke-run/` directory. The script validates128the fixture, hashes all splits, records the exact package version and command,129evaluates the baseline, trains, evaluates the adapter, and writes `report.json`.130131## UI132133- `#/overview`: readiness, approved split counts, active runs, candidate adapters,134 pipeline stages, latest baseline/adapter delta, and recent runs.135- `#/examples[/<id>]`: desktop list/detail review queue and separate phone list136 and detail panes with stable example ids and review actions.137- `#/runs`: base model, method, claim/status, heartbeat, artifacts, and failures.138- `#/evaluations`: locked baseline/adapter comparisons and promotion verdicts.139- `#/registry`: active, candidate, and retired adapters with complete lineage.140- `#/settings`: responsive Help & Settings modal with guide, resources, and141 sanitized training configuration.142143Demo mode is explicit and in-memory only. `?demo=1` opens the overview;144`?demo=dataset`, `?demo=evaluations`, and `?demo=registry` open named screenshot145scenes. `lang=en` or `lang=zh` forces the UI language. Demo never claims a real146connection, writes Busabase, or proves the local Worker is online.147148## Completion149150Do not claim a production fine-tune until:151152- the snapshot came from reviewed Busabase examples and has a stable Drive ref/hash;153- the base model revision, MLX-LM version, command, and config are recorded;154- baseline and adapter used the same locked cases and decoding settings;155- adapter artifacts and reports exist and are referenced by a terminal run;156- the human promotion verdict is recorded against the evaluated version;157- app checks, local/OSS conformance, responsive browser checks, and available158 Cloud/AirApp suites passed or are explicitly reported as skipped.159160Stop when the Space or dataset is ambiguous, examples lack review/provenance,161the run changed after claim, test leakage is detected, the model license is162incompatible, an artifact hash fails, or a secret would cross into the browser.