Review vendor coverage
The principle (per CLAUDE.md): anything that decides a score is
vendored verbatim from NeMo-Skills. SE code is allowed only for
transport, glue, default config, and CLI / output formatting.
This skill audits whether SE code has leaked scoring logic. It does NOT
auto-fix; vendoring more is a design call the user makes after seeing
the report.
1. Enumerate SE files
find sgl_eval scripts tests -name "*.py" \
| grep -v _vendored | grep -v __pycache__ | sort
2. Classify each into one of four buckets
For each file (and each function within larger files), ask:
"Does this code decide a score?"
| Bucket |
Description |
Examples |
Action |
| Transport |
Wraps an external service or a Python primitive. No scoring decisions. |
sampler, runner, cli, types, registry, metrics formatter |
OK as SE. |
| Glue |
Bridges between vendored API and our runtime model. Necessary because of architectural difference (sync vs async, in-memory vs file, single vs batch). |
_predictions.py, _loader.py, _eval_single_sync, _score_via_eval_mcq |
OK as SE. Document the upstream API it bridges. |
| Default config |
Per-benchmark sampling / repeat / thinking defaults. Model-dependent, not a scoring decision. |
_TABLE rows in _registry.py |
OK as SE. Confirm tracks NS-aligned defaults. |
| Score-deciding |
Code that parses generation, compares answers, computes pass@k, renders prompts beyond str.format. |
(should be empty in SE!) |
RED FLAG. See step 4. |
3. Specific spots to scrutinize (grey areas)
Known borderline pieces -- not violations, but watch for drift:
| Location |
What it does |
Why grey |
evals/_prompts.py:render_prompt |
yaml.safe_load + str.format on vendored yaml |
Re-implements upstream's prompt subsystem (~500 lines). Equivalent today because templates only use {problem} / {examples}. Drift if upstream adds richer templating. |
evals/_math.py:aggregate_with_math_metrics (padding) |
while len < n_repeats: append dup |
NS doesn't pad; we pad to keep MathMetrics happy under partial sample failure. Affects score only when sampler fails partway. |
evals/_math.py:_flatten_math_metrics and _multichoice.py:_flatten |
Picks fields out of MathMetrics.get_metrics() nested dict |
Display selection, not computation. Safe if upstream keeps the same field names. |
_TABLE thinking / n_repeats |
Per-benchmark sgl-eval defaults |
Not a violation; runtime config. CLI override always wins. |
If a new grey area surfaces during the audit, add a row.
4. If you find score-deciding code in SE
Don't auto-edit. Report to the user:
- What it computes (the scoring decision in plain words).
- Where in NS upstream the equivalent lives (or "not in NS").
- What it would take to vendor it: file path(s), import rewrites,
any drop_imports/drop_functions needed.
Then ask the user whether to:
- vendor (run
vendor-update workflow with extended SOURCES.yaml),
- keep as SE (document the deviation, e.g., in CLAUDE.md grey-area list),
- or remove the logic entirely.
5. Output format
Write a short report (under 300 words):
SE files: <count> total
transport : <n> (<list>)
glue : <n> (<list>)
default config : <n> (<list>)
score-deciding : <n> (RED if > 0)
Grey areas reviewed:
- <location>: <still acceptable / has drifted because X>
New findings:
- <none> | <list>
Recommendation: <one sentence>
1---2name: review-vendor-coverage3description: Audit whether all score-deciding logic in sgl-eval is vendored from NeMo-Skills, or whether some has crept into SE code. Use when the user asks "are we vendoring enough", "review vendor coverage", "audit vendoring", "is our vendoring complete", or before a release.4---56# Review vendor coverage78The principle (per `CLAUDE.md`): **anything that decides a score is9vendored verbatim from NeMo-Skills**. SE code is allowed only for10transport, glue, default config, and CLI / output formatting.1112This skill audits whether SE code has leaked scoring logic. It does NOT13auto-fix; vendoring more is a design call the user makes after seeing14the report.1516## 1. Enumerate SE files1718```bash19find sgl_eval scripts tests -name "*.py" \20 | grep -v _vendored | grep -v __pycache__ | sort21```2223## 2. Classify each into one of four buckets2425For each file (and each function within larger files), ask:26**"Does this code decide a score?"**2728| Bucket | Description | Examples | Action |29|---|---|---|---|30| **Transport** | Wraps an external service or a Python primitive. No scoring decisions. | sampler, runner, cli, types, registry, metrics formatter | OK as SE. |31| **Glue** | Bridges between vendored API and our runtime model. Necessary because of architectural difference (sync vs async, in-memory vs file, single vs batch). | `_predictions.py`, `_loader.py`, `_eval_single_sync`, `_score_via_eval_mcq` | OK as SE. Document the upstream API it bridges. |32| **Default config** | Per-benchmark sampling / repeat / thinking defaults. Model-dependent, not a scoring decision. | `_TABLE` rows in `_registry.py` | OK as SE. Confirm tracks NS-aligned defaults. |33| **Score-deciding** | Code that parses generation, compares answers, computes pass@k, renders prompts beyond `str.format`. | (should be empty in SE!) | RED FLAG. See step 4. |3435## 3. Specific spots to scrutinize (grey areas)3637Known borderline pieces -- not violations, but watch for drift:3839| Location | What it does | Why grey |40|---|---|---|41| `evals/_prompts.py:render_prompt` | `yaml.safe_load + str.format` on vendored yaml | Re-implements upstream's prompt subsystem (~500 lines). Equivalent today because templates only use `{problem}` / `{examples}`. Drift if upstream adds richer templating. |42| `evals/_math.py:aggregate_with_math_metrics` (padding) | `while len < n_repeats: append dup` | NS doesn't pad; we pad to keep `MathMetrics` happy under partial sample failure. Affects score only when sampler fails partway. |43| `evals/_math.py:_flatten_math_metrics` and `_multichoice.py:_flatten` | Picks fields out of `MathMetrics.get_metrics()` nested dict | Display selection, not computation. Safe if upstream keeps the same field names. |44| `_TABLE` `thinking` / `n_repeats` | Per-benchmark sgl-eval defaults | Not a violation; runtime config. CLI override always wins. |4546If a new grey area surfaces during the audit, add a row.4748## 4. If you find score-deciding code in SE4950Don't auto-edit. Report to the user:51521. **What** it computes (the scoring decision in plain words).532. **Where** in NS upstream the equivalent lives (or "not in NS").543. **What it would take to vendor it**: file path(s), import rewrites,55 any drop_imports/drop_functions needed.5657Then ask the user whether to:58- vendor (run `vendor-update` workflow with extended `SOURCES.yaml`),59- keep as SE (document the deviation, e.g., in CLAUDE.md grey-area list),60- or remove the logic entirely.6162## 5. Output format6364Write a short report (under 300 words):6566```67SE files: <count> total68 transport : <n> (<list>)69 glue : <n> (<list>)70 default config : <n> (<list>)71 score-deciding : <n> (RED if > 0)7273Grey areas reviewed:74 - <location>: <still acceptable / has drifted because X>7576New findings:77 - <none> | <list>7879Recommendation: <one sentence>80```
sgl-project