# Microsoft Benchmark Qed Benchmark Qed > Benchmark-QED Question Generation (autoq) - Skill: `tomevault-io/microsoft-benchmark-qed-benchmark-qed` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/microsoft-benchmark-qed-benchmark-qed` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/microsoft-benchmark-qed-benchmark-qed/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/microsoft-benchmark-qed-benchmark-qed --- # Benchmark-QED Question Generation (autoq) Generate benchmark questions and assertions from input data for RAG evaluation. ## Prerequisites - A configured workspace with valid `settings.yaml` (use the `/benchmark-qed-setup` skill first) - A configured workspace with valid `settings.yaml` (use the `benchmark-qed-setup` skill to initialize and configure) - Input data (CSV or JSON) in the workspace `input/` directory - Valid LLM API key in `.env` Run all commands with: ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed ``` ## Commands ### 1. Generate Questions (`autoq`) The main question generation pipeline. Generates benchmark questions from input data. ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq [OPTIONS] ``` **Options:** | Option | Description | |--------|-------------| | `--generation-types` | Specific types to generate (repeatable). CLI default: all except `data_linked`, but this skill always includes `data_linked` | | `--print-model-usage` | Print LLM token usage stats | | `--account-url` | Azure Blob Storage account URL (managed-identity auth). Use when the config path is a `blob://` URI. Falls back to `$AZURE_STORAGE_ACCOUNT_URL`. | | `--connection-string` | Azure Blob Storage connection string. Use when the config path is a `blob://` URI. Falls back to `$AZURE_STORAGE_CONNECTION_STRING`. | **Generation types and dependencies:** ``` data_local ← runs first (no dependencies) ├── data_global ← requires data_local candidates └── data_linked ← requires data_local candidates (not in CLI default, but this skill always includes it) activity_local ← auto-generates activity_context first └── activity_global ← requires activity_local ``` > **Important**: `data_linked` is NOT included in the CLI's default generation types, but this skill always generates it by passing all types explicitly. If running the CLI manually, you must add `--generation-types data_linked`. > **Gotcha**: `data_global` and `data_linked` silently return empty results if `data_local` hasn't been run first. Always run `data_local` before these types. **Examples:** ```bash # Run from the workspace directory (paths resolve relative to settings.yaml location) cd ./workspace # Generate all types including data_linked (skill default) uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq settings.yaml ./output \ --generation-types data_local --generation-types data_global --generation-types data_linked \ --generation-types activity_local --generation-types activity_global # Generate only local questions uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq settings.yaml ./output --generation-types data_local # Generate local + linked questions uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq settings.yaml ./output \ --generation-types data_local --generation-types data_linked # Use a config stored in Azure Blob Storage uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq \ blob://my-container/configs/settings.yaml ./output \ --account-url https://myaccount.blob.core.windows.net \ --generation-types data_local ``` **Output structure:** ``` output_dir/ ├── sample_texts.parquet # Intermediate: clustered text samples ├── data_local_questions/ │ ├── selected_questions.json # Final curated questions │ ├── selected_questions_text.json # Human-readable version │ └── candidate_questions.json # All generated candidates ├── data_global_questions/ # Same structure ├── data_linked_questions/ # Same structure + question_stats.json ├── activity_local_questions/ # Same structure ├── activity_global_questions/ # Same structure ├── context/ │ └── activity_context_full.json # Generated activity context └── model_usage.json # LLM token/cost tracking ``` ### 2. Generate Assertions (`generate-assertions`) Generate ground-truth assertions for existing questions (decoupled from question generation). This is a **top-level** command, not a subcommand of `autoq`. ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed generate-assertions [OPTIONS] ``` **Options:** | Option | Description | |--------|-------------| | `--type` / `-t` | Assertion type: `local`, `global`, or `linked` (default: `local`) | | `--print-model-usage` | Print LLM token usage stats | | `--account-url` | Azure Blob Storage account URL (managed-identity auth). Use when the config path is a `blob://` URI. | | `--connection-string` | Azure Blob Storage connection string. Use when the config path is a `blob://` URI. | **Examples:** ```bash # Run from the workspace directory (paths resolve relative to settings.yaml location) cd ./workspace uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed generate-assertions \ settings.yaml \ ./output/data_local_questions/candidate_questions.json \ ./output/data_local_questions/ \ --type local uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed generate-assertions \ settings.yaml \ ./output/data_global_questions/candidate_questions.json \ ./output/data_global_questions/ \ --type global ``` ### 3. Assertion Statistics (`assertion-stats`) Compute quality statistics for assertion files. This is a **top-level** command, not a subcommand of `autoq`. ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed assertion-stats [OPTIONS] ``` **Options:** | Option | Description | |--------|-------------| | `--output` / `-o` | Output path for stats JSON (auto-generated if omitted) | | `--type` / `-t` | `global`, `map`, or `local` (auto-inferred if omitted) | | `--quiet` / `-q` | Suppress console output | **Examples:** ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed assertion-stats ./output/assertions.json uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed assertion-stats ./output/data_global_questions/ -q ``` ## Workflow ### Standard Question Generation Flow - [ ] Step 1: Initialize workspace if needed — use the `benchmark-qed-setup` skill to create and configure the workspace. Verify `settings.yaml`, `.env`, and `input/` exist. - [ ] Step 2: `cd ` then run question generation — `uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq settings.yaml ./output --generation-types data_local --generation-types data_global --generation-types data_linked --generation-types activity_local --generation-types activity_global` - [ ] Step 3: Verify output artifacts — list `` and confirm the per-type `selected_questions.json` files (see "Output structure" above) plus `model_usage.json` exist. - [ ] Step 4: (Optional) Generate additional assertions — use `generate-assertions` - [ ] Step 5: (Optional) Check assertion quality — use `assertion-stats` ## Azure Blob Storage All `autoq` and `generate-assertions` commands support reading config files from Azure Blob Storage using `blob://` URIs: ```bash # Config file in blob storage — the CLI downloads the config (and sibling prompt files) to a temp directory uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoq \ blob://my-container/project/settings.yaml ./output \ --connection-string "$AZURE_STORAGE_CONNECTION_STRING" ``` In addition, `settings.yaml` supports `input.storage` and `output_storage` blocks to read/write data from Azure Blob Storage: ```yaml # Read input data from blob storage input: dataset_path: data/input.csv storage: type: blob container_name: my-datasets connection_string: ${AZURE_STORAGE_CONNECTION_STRING} # Or use managed identity: # account_url: https://myaccount.blob.core.windows.net # Write output to blob storage output_storage: type: blob container_name: my-output connection_string: ${AZURE_STORAGE_CONNECTION_STRING} ``` See [references/config-reference.md](../benchmark-qed-setup/references/config-reference.md) for full `StorageConfig` fields. ## Gotchas - **Path resolution**: The `autoq` and `generate-assertions` commands resolve `output_dir` (and other relative paths) **relative to the settings.yaml file's directory**, not the current working directory. Always `cd` into the workspace directory first, or use absolute paths. For example, running `benchmark-qed autoq workspace/settings.yaml workspace/output` from the repo root creates output at `workspace/workspace/output/` (not `workspace/output/`). - **Stale outputs**: The pipeline skips steps if output files already exist (`sample_texts.parquet`, `activity_context_full.json`). Use a fresh output directory for clean runs, or delete specific files to re-run a step. - **Long-running**: Question generation with large datasets can take hours. Use background execution and monitor via `model_usage.json` presence. - **Output is in files, not stdout**: All results are written to JSON/CSV/Parquet files. Parse the output files, not CLI stdout. - **Generation ordering**: `data_global` and `data_linked` depend on `data_local`. `activity_global` depends on `activity_local`. Running dependent types without their prerequisites produces silent empty results. - **`data_linked` CLI opt-in**: The CLI excludes `data_linked` by default, but this skill always includes it. If running the CLI manually outside this skill, add `--generation-types data_linked`. - **Blob URI format**: Use `blob:///` for config paths. The CLI downloads the config and all sibling files (prompt templates) to a temp directory so relative paths resolve correctly. --- > Source: [microsoft/benchmark-qed](https://github.com/microsoft/benchmark-qed) — distributed by [TomeVault](https://tomevault.io).