flashinfer-ai Validate Dataset
Run structural, schema, and coverage checks on a FlashInfer Trace dataset. Produces a JSON report and optional text summary.
Usage
# Full validation (including GPU benchmark)
/validate-dataset
# Skip GPU benchmark
/validate-dataset --disable-gpu
# Validate specific op_type or definition
/validate-dataset --op-types mla_paged --disable-gpu
/validate-dataset --definitions rmsnorm_h4096 --disable-gpu
Parameters
--dataset: Dataset root (default:FIB_DATASET_PATHenv var)--op-types: Filter by op_type (union with--definitions)--definitions: Filter by definition name--checks: Comma-separated subset of:layout,definition,workload,solution,trace,baseline,benchmark--disable-gpu: Skip benchmark checks (default: GPU enabled)--outputs: Comma-separated output targets:stdout,json,text(default: all three)--output-folder: Report output directory (default:<dataset>/reports/)
Implementation
Files
| File | Purpose |
|---|---|
flashinfer_bench/data/validate.py |
Core validation logic: validate_dataset() |
flashinfer_bench/data/validate_render.py |
Report models and rendering: render_report() |
flashinfer_bench/cli/main.py |
CLI entry: flashinfer-bench validate, flashinfer-bench validate-render |
Steps
Run validation via CLI:
flashinfer-bench validate --dataset tmp/flashinfer-trace --disable-gpuOr call Python API directly:
from flashinfer_bench.data.validate import validate_dataset report = validate_dataset(dataset="tmp/flashinfer-trace", disable_gpu=True)Re-render an existing report:
flashinfer-bench validate-render reports/report-20260413-120000.json
Check Categories
| Category | What it checks |
|---|---|
layout |
Duplicate definition names, directory structure, path-field consistency, blob file existence |
definition |
Pydantic schema, reference Python syntax, axis references, descriptions, build reference (GPU) |
workload |
Schema, axes > 0, shape inference, safetensors blobs, no solution/evaluation |
solution |
Pydantic schema, entry_point format, sources, path-field consistency |
trace |
Schema, must have solution+evaluation, solution existence, workload coverage |
baseline |
Baseline solution/trace existence, build baseline solutions (GPU), PASSED status, workload coverage |
benchmark |
GPU: run baseline + reference via Benchmark.run_all() (lightweight config) |
Report Output
- JSON and text reports are timestamped (
report-YYYYMMDD-HHMMSS.json/.txt) to prevent overwriting - Reports go to
<dataset>/reports/by default - See
docs/flashinfer-trace/validate.mdxfor full report structure and examples
See Also
- docs/flashinfer-trace/validate.mdx — full documentation
- collect-workloads — collect workloads before validation
- add-reference-tests — add reference tests for definitions