Omni Bench

omni-bench development

Embeddable benchmark framework, not a model runner: hosts own inference via adapter seams; omni-bench owns datasets, scoring, metrics, and the JSON spec. North star: quality + speed tied to specific hardware; parity (Python↔Swift, CUDA↔MLX) is a derived view. Primary consumer: a Swift host wrapping in-process models (e.g. mlx-omni).

Layout

• schemas/*.schema.json — the SSOT (open spec, JSON Schema draft 2020-12). Everything else follows it. Pinned schema_version (0.1.0, re-cut 2026-07-04 — the pre-release 0.1.0/0.2.0 iterations are retired).
• python/src/omni_bench/ — core/ (prepare, producer, scorer, diff, validate, identity, hardware), asr/ (normalizers, metrics, registry — the registry is the modality-agnostic task table despite the package name), textgen/ (synthetic shared-prefix task), models/ (generated — never hand-edit; regenerate with just codegen-python).
• swift/ — producer SDK mirroring the Python producer (Transcriber, Generator, Producer, ArtifactWriter, JCS identity).
• fixtures/ — committed goldens; each dir has a generate.py to regenerate.

Pipeline & artifacts

prepare → run → score → diff: manifest.json (+references.jsonl) → run-artifact.jsonl → result.json → parity-report.json.

• Producer never aborts: a failing sample becomes an error record.
• Scorer is the trust anchor: scoring exists only in Python, once. Never reimplement scoring in a host (measurer drift is the enemy this design kills).
• Identity is a JCS (RFC 8785) hash over a fixed subset (model, backend, hardware, os, task, dataset, run_profile.decode). Identity fields are host-declared and trusted — a bogus value silently breaks the parity join.

Adapter seams (structural, no inheritance; Swift mirrors Python 1:1)

• ASR: Transcriber.transcribe(audio, language, task) -> Transcript.
• Text generation: Generator.generate(prompt, task) -> Generation + reset_cache(). All latency/token metrics (ttft_s, prefill_s, prompt_tokens, generated_tokens, reused_tokens, cache_hit, decode_s) are host-reported via backend_native — the seam is non-streaming, the producer cannot observe TTFT. The scorer derives the rates: tok_per_s (micro) and prefill_tok_per_s (cold samples only in cache mode — cache-free).
• Shared-prefix cold/warm scenario: mode: "shared_prefix_cold_warm" + per-sample prefix_group. Producer resets the cache at every group boundary; first sample after reset is phase: "cold", rest "warm"; nothing is warmup-discarded — the scorer's cache block reports cold/warm medians and ttft_speedup = cold / warm. Built-in offline task: textgen.shared_prefix.en.v1.

Schema evolution rules (bite hard)

• Objects are closed (additionalProperties: false) ⇒ ANY additive field is a breaking change ⇒ bump the minor schema_version in lockstep across all schemas, then update SCHEMA_VERSION in core/{prepare,producer,scorer,diff}.py, Swift ArtifactWriter.writeHeader, regenerate models AND all fixtures.
• After any schema change: just codegen-python (commit models/), rerun fixtures/{scoring,parity,textgen}/generate.py, hand-bump fixtures/examples/* and fixtures/xlang/manifest.json.
• Determinism is a hard invariant: sort_keys JSON, samples sorted by id, monotonic clocks, no timestamps in WAVs — prepare twice ⇒ identical hashes. Nothing under omni_bench/ may use wall-clock or randomness in artifacts.

Commands

just check                 # validate-schemas + pytest + swift test — run before commit
just codegen-python        # schemas -> pydantic models (stages *.schema.json to tmp)
uv run --project python pytest python/tests -q
cd swift && swift test
uv run --project python python -m omni_bench.cli prepare textgen.shared_prefix.en.v1  # offline, CI-safe

just/uv may live outside the default PATH (e.g. /opt/homebrew/bin). Only textgen.shared_prefix.en.v1 is downloads-free; the ASR tasks hit HuggingFace — tests monkeypatch get_task instead.

Gotchas

• datamodel-codegen parses EVERY file in its input dir — stage *.schema.json into a temp dir named schemas (the dir name lands in the generated header; keep it stable), which is why the justfile does exactly that.
• Required-nullable is the repo idiom ("type": ["x", "null"] + required): Swift writes explicit NSNull(), omission fails validation. phase is the exception (optional) so ASR artifacts keep their v1 shape.
• ASR hyp-identity in diff uses the language normalizer; text-generation uses NFC-exact comparison — deterministic decoding makes exact identity the gate.
• Hardware keys the identity: off macOS the collector can't see the accelerator; pass explicit hardware={"soc": ..., "mem_gb": ...} (producer warns otherwise).
• Keep contract docs in sync on any schema change: schemas/README.md, docs/integration.md, and the README CLI/task lists.