Benchmark Representativeness Experimental Design Lens
Philosophical Mode: Generalizability
Primary Question: "Does this generalize beyond the test bed?"
Focus: Task Distribution, Scenario Coverage, Missing Regions, Dataset Selection, Generalization Claims
Arguments
/autoskillit:exp-lens-benchmark-representativeness [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Evaluating claims that extend beyond specific benchmarks
- Checking coverage of evaluation suite
- Assessing dataset diversity
- User invokes
/autoskillit:exp-lens-benchmark-representativeness or /autoskillit:make-experiment-diag benchmark
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Focus on GENERALIZATION GAP between benchmark coverage and claimed scope
Show which regions of the target space are untested
Document the relationship between benchmark selection and generalization claims
Include a coverage matrix mapping scenarios to metrics
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Benchmark & Dataset Inventory
- Find all datasets, benchmarks, test suites used
- Look for:
benchmark, dataset, test_suite, eval, corpus, split, GLUE, ImageNet
Task & Scenario Coverage
- Find what scenarios, conditions, and domains are tested
- Look for:
task, scenario, domain, category, difficulty, subset
Metric Coverage
- Find all evaluation metrics used
- Look for:
metric, accuracy, f1, bleu, rouge, latency, cost, fairness
Claimed Generalization Scope
- Find claims about generality in docs, papers, READMEs
- Look for:
generalize, real-world, production, deploy, robust, transfer, domain
Distribution Characteristics
- Find data distribution analysis, class balance, domain stats
- Look for:
distribution, balance, skew, size, demographics, diversity
Step 2: Build the Coverage Matrix
Build the coverage matrix: rows = scenarios/domains tested, columns = metrics measured. Identify which cells are populated and which are gaps. Compare the coverage to the stated generalization claims.
Step 3: CRITICAL — Analyze Generalization Gap
For every generalization claim:
- Target population: What is the full population the claim extends to?
- Benchmark representation: What subset of that population is represented in the benchmark?
- Untested regions: What regions of the space are untested?
- Coverage ratio: Is the coverage sufficient to support the claim?
Distinguish clearly:
- Strong claims (e.g., "production-ready"): require broad, diverse coverage
- Scoped claims (e.g., "best on GLUE"): only require benchmark-specific coverage
- Implicit claims: claims made in framing but not stated explicitly
Step 4: Create the Diagram
Use flowchart with:
Direction: TB (claims flow from benchmarks up to generalization)
Subgraphs:
BENCHMARKS USED
SCENARIOS TESTED
METRICS MEASURED
GENERALIZATION CLAIMS
UNTESTED REGIONS
Node Styling:
stateNode class: benchmarks/datasets
handler class: tested scenarios
output class: measured metrics
cli class: generalization claims
gap class: untested regions/missing coverage
detector class: validation of generalization
Step 5: Write Output
Write the diagram to: {{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Benchmark Representativeness Diagram: {System Name}
**Lens:** Benchmark Representativeness (Generalizability)
**Question:** Does this generalize beyond the test bed?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Coverage Matrix
| Scenario / Domain | {Metric A} | {Metric B} | {Metric C} | Coverage |
|-------------------|-----------|-----------|-----------|----------|
| {Scenario 1} | ✓ | ✓ | ✗ | Partial |
| {Scenario 2} | ✗ | ✗ | ✗ | None |
| {Scenario 3} | ✓ | ✓ | ✓ | Full |
## Benchmark Representativeness Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Benchmarks ["BENCHMARKS USED"]
direction TB
B1["{Benchmark 1}<br/>━━━━━━━━━━<br/>{size}, {domain}"]
B2["{Benchmark 2}<br/>━━━━━━━━━━<br/>{size}, {domain}"]
end
subgraph Scenarios ["SCENARIOS TESTED"]
direction TB
S1["{Scenario 1}<br/>━━━━━━━━━━<br/>{conditions}"]
S2["{Scenario 2}<br/>━━━━━━━━━━<br/>{conditions}"]
end
subgraph Metrics ["METRICS MEASURED"]
direction TB
M1["{Metric A}<br/>━━━━━━━━━━<br/>{what it captures}"]
M2["{Metric B}<br/>━━━━━━━━━━<br/>{what it captures}"]
end
subgraph Claims ["GENERALIZATION CLAIMS"]
direction TB
C1["{Claim 1}<br/>━━━━━━━━━━<br/>{source of claim}"]
C2["{Claim 2}<br/>━━━━━━━━━━<br/>{source of claim}"]
end
subgraph Gaps ["UNTESTED REGIONS"]
direction TB
G1["{Gap 1}<br/>━━━━━━━━━━<br/>{why it matters}"]
G2["{Gap 2}<br/>━━━━━━━━━━<br/>{why it matters}"]
end
VALIDATE["{Generalization Validity Check}<br/>━━━━━━━━━━<br/>Coverage vs. Claim scope"]
B1 --> S1
B2 --> S2
S1 --> M1
S2 --> M2
M1 --> C1
M2 --> C2
C1 --> VALIDATE
C2 --> VALIDATE
G1 -.->|missing| VALIDATE
G2 -.->|missing| VALIDATE
%% CLASS ASSIGNMENTS %%
class B1,B2 stateNode;
class S1,S2 handler;
class M1,M2 output;
class C1,C2 cli;
class G1,G2 gap;
class VALIDATE detector;
Color Legend:
| Color |
Category |
Description |
| Dark Teal |
Benchmarks |
Datasets and test suites used |
| Orange |
Scenarios |
Tested scenarios and conditions |
| Teal |
Metrics |
Measured evaluation metrics |
| Dark Blue |
Claims |
Generalization claims made |
| Yellow/Amber |
Gaps |
Untested regions of target space |
| Red |
Validation |
Generalization validity check |
Generalization Gap Analysis
| Claim |
Evidence (Benchmarks) |
Gap (Untested) |
Risk |
| {Claim 1} |
{What covers it} |
{What is missing} |
High/Med/Low |
| {Claim 2} |
{What covers it} |
{What is missing} |
High/Med/Low |
Representativeness Assessment
| Dimension |
Current Coverage |
Required for Claim |
Verdict |
| Domain diversity |
{count} domains |
{needed} |
✓/✗ |
| Task variety |
{count} tasks |
{needed} |
✓/✗ |
| Scale range |
{min}–{max} |
{needed} |
✓/✗ |
| Distribution shift |
{tested?} |
{needed} |
✓/✗ |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-measurement-validity` - For metric quality analysis
- `/autoskillit:exp-lens-validity-threats` - For systematic threat inventory
1---2name: exp-lens-benchmark-representativeness3categories: [exp-lens]4activate_deps: [mermaid]5description: Create Benchmark Representativeness experimental design diagram showing coverage matrix, generalization gaps, and untested regions. Generalizability lens answering "Does this generalize beyond the test bed?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Benchmark Representativeness Lens - Analyzing generalization coverage...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-benchmark-representativeness15---1617# Benchmark Representativeness Experimental Design Lens1819**Philosophical Mode:** Generalizability20**Primary Question:** "Does this generalize beyond the test bed?"21**Focus:** Task Distribution, Scenario Coverage, Missing Regions, Dataset Selection, Generalization Claims2223## Arguments2425`/autoskillit:exp-lens-benchmark-representativeness [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Evaluating claims that extend beyond specific benchmarks38- Checking coverage of evaluation suite39- Assessing dataset diversity40- User invokes `/autoskillit:exp-lens-benchmark-representativeness` or `/autoskillit:make-experiment-diag benchmark`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/`48- Run subagents in the background (`run_in_background: true` is prohibited)4950**ALWAYS:**51- Focus on GENERALIZATION GAP between benchmark coverage and claimed scope52- Show which regions of the target space are untested53- Document the relationship between benchmark selection and generalization claims54- Include a coverage matrix mapping scenarios to metrics55- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY56- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.57- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md`58- After writing the file, emit the structured output token as **literal plain text** with no59 markdown formatting on the token name (the adjudicator performs a regex match):6061 ```62 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{...}.md63 ```6465---6667## Analysis Workflow6869### Step 0: Parse optional arguments7071If positional arg 1 (context_path) is provided and the file exists, read it to obtain72IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional73arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full74methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD75exploration for these fields if the context file supplies them.7677### Step 1: Launch Parallel Exploration Subagents7879Spawn Explore subagents to investigate:8081**Benchmark & Dataset Inventory**82- Find all datasets, benchmarks, test suites used83- Look for: `benchmark`, `dataset`, `test_suite`, `eval`, `corpus`, `split`, `GLUE`, `ImageNet`8485**Task & Scenario Coverage**86- Find what scenarios, conditions, and domains are tested87- Look for: `task`, `scenario`, `domain`, `category`, `difficulty`, `subset`8889**Metric Coverage**90- Find all evaluation metrics used91- Look for: `metric`, `accuracy`, `f1`, `bleu`, `rouge`, `latency`, `cost`, `fairness`9293**Claimed Generalization Scope**94- Find claims about generality in docs, papers, READMEs95- Look for: `generalize`, `real-world`, `production`, `deploy`, `robust`, `transfer`, `domain`9697**Distribution Characteristics**98- Find data distribution analysis, class balance, domain stats99- Look for: `distribution`, `balance`, `skew`, `size`, `demographics`, `diversity`100101### Step 2: Build the Coverage Matrix102103Build the coverage matrix: rows = scenarios/domains tested, columns = metrics measured. Identify which cells are populated and which are gaps. Compare the coverage to the stated generalization claims.104105### Step 3: CRITICAL — Analyze Generalization Gap106107For every generalization claim:108- **Target population**: What is the full population the claim extends to?109- **Benchmark representation**: What subset of that population is represented in the benchmark?110- **Untested regions**: What regions of the space are untested?111- **Coverage ratio**: Is the coverage sufficient to support the claim?112113Distinguish clearly:114- **Strong claims** (e.g., "production-ready"): require broad, diverse coverage115- **Scoped claims** (e.g., "best on GLUE"): only require benchmark-specific coverage116- **Implicit claims**: claims made in framing but not stated explicitly117118### Step 4: Create the Diagram119120Use flowchart with:121122**Direction:** `TB` (claims flow from benchmarks up to generalization)123124**Subgraphs:**125- `BENCHMARKS USED`126- `SCENARIOS TESTED`127- `METRICS MEASURED`128- `GENERALIZATION CLAIMS`129- `UNTESTED REGIONS`130131**Node Styling:**132- `stateNode` class: benchmarks/datasets133- `handler` class: tested scenarios134- `output` class: measured metrics135- `cli` class: generalization claims136- `gap` class: untested regions/missing coverage137- `detector` class: validation of generalization138139### Step 5: Write Output140141Write the diagram to: `{{AUTOSKILLIT_TEMP}}/exp-lens-benchmark-representativeness/exp_diag_benchmark_representativeness_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)142143---144145## Output Template146147```markdown148# Benchmark Representativeness Diagram: {System Name}149150**Lens:** Benchmark Representativeness (Generalizability)151**Question:** Does this generalize beyond the test bed?152**Date:** {YYYY-MM-DD}153**Scope:** {What was analyzed}154155## Coverage Matrix156157| Scenario / Domain | {Metric A} | {Metric B} | {Metric C} | Coverage |158|-------------------|-----------|-----------|-----------|----------|159| {Scenario 1} | ✓ | ✓ | ✗ | Partial |160| {Scenario 2} | ✗ | ✗ | ✗ | None |161| {Scenario 3} | ✓ | ✓ | ✓ | Full |162163## Benchmark Representativeness Diagram164165```mermaid166%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%167flowchart TB168 %% CLASS DEFINITIONS %%169 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;170 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;171 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;172 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;173 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;174 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;175 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;176 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;177 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;178179 subgraph Benchmarks ["BENCHMARKS USED"]180 direction TB181 B1["{Benchmark 1}<br/>━━━━━━━━━━<br/>{size}, {domain}"]182 B2["{Benchmark 2}<br/>━━━━━━━━━━<br/>{size}, {domain}"]183 end184185 subgraph Scenarios ["SCENARIOS TESTED"]186 direction TB187 S1["{Scenario 1}<br/>━━━━━━━━━━<br/>{conditions}"]188 S2["{Scenario 2}<br/>━━━━━━━━━━<br/>{conditions}"]189 end190191 subgraph Metrics ["METRICS MEASURED"]192 direction TB193 M1["{Metric A}<br/>━━━━━━━━━━<br/>{what it captures}"]194 M2["{Metric B}<br/>━━━━━━━━━━<br/>{what it captures}"]195 end196197 subgraph Claims ["GENERALIZATION CLAIMS"]198 direction TB199 C1["{Claim 1}<br/>━━━━━━━━━━<br/>{source of claim}"]200 C2["{Claim 2}<br/>━━━━━━━━━━<br/>{source of claim}"]201 end202203 subgraph Gaps ["UNTESTED REGIONS"]204 direction TB205 G1["{Gap 1}<br/>━━━━━━━━━━<br/>{why it matters}"]206 G2["{Gap 2}<br/>━━━━━━━━━━<br/>{why it matters}"]207 end208209 VALIDATE["{Generalization Validity Check}<br/>━━━━━━━━━━<br/>Coverage vs. Claim scope"]210211 B1 --> S1212 B2 --> S2213 S1 --> M1214 S2 --> M2215 M1 --> C1216 M2 --> C2217 C1 --> VALIDATE218 C2 --> VALIDATE219 G1 -.->|missing| VALIDATE220 G2 -.->|missing| VALIDATE221222 %% CLASS ASSIGNMENTS %%223 class B1,B2 stateNode;224 class S1,S2 handler;225 class M1,M2 output;226 class C1,C2 cli;227 class G1,G2 gap;228 class VALIDATE detector;229```230231**Color Legend:**232| Color | Category | Description |233|-------|----------|-------------|234| Dark Teal | Benchmarks | Datasets and test suites used |235| Orange | Scenarios | Tested scenarios and conditions |236| Teal | Metrics | Measured evaluation metrics |237| Dark Blue | Claims | Generalization claims made |238| Yellow/Amber | Gaps | Untested regions of target space |239| Red | Validation | Generalization validity check |240241## Generalization Gap Analysis242243| Claim | Evidence (Benchmarks) | Gap (Untested) | Risk |244|-------|----------------------|----------------|------|245| {Claim 1} | {What covers it} | {What is missing} | High/Med/Low |246| {Claim 2} | {What covers it} | {What is missing} | High/Med/Low |247248## Representativeness Assessment249250| Dimension | Current Coverage | Required for Claim | Verdict |251|-----------|-----------------|-------------------|---------|252| Domain diversity | {count} domains | {needed} | ✓/✗ |253| Task variety | {count} tasks | {needed} | ✓/✗ |254| Scale range | {min}–{max} | {needed} | ✓/✗ |255| Distribution shift | {tested?} | {needed} | ✓/✗ |256```257258---259260## Pre-Diagram Checklist261262Before creating the diagram, verify:263264- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool265- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)266- [ ] Diagram will include a color legend table267268---269270## Related Skills271272- `/autoskillit:make-experiment-diag` - Parent skill for lens selection273- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram274- `/autoskillit:exp-lens-measurement-validity` - For metric quality analysis275- `/autoskillit:exp-lens-validity-threats` - For systematic threat inventory