Golden Dataset
Comprehensive patterns for building, managing, and validating golden datasets for AI/ML evaluation. Each category has individual rule files in rules/ loaded on-demand.
Quick Reference
| Category |
Rules |
Impact |
When to Use |
| Curation |
3 |
HIGH |
Content collection, annotation pipelines, diversity analysis |
| Management |
3 |
HIGH |
Versioning, backup/restore, CI/CD automation |
| Validation |
3 |
CRITICAL |
Quality scoring, drift detection, regression testing |
| Add Workflow |
1 |
HIGH |
9-phase curation, quality scoring, bias detection, silver-to-gold |
Total: 10 rules across 4 categories
Curation
Content collection, multi-agent annotation, and diversity analysis for golden datasets.
| Rule |
File |
Key Pattern |
| Collection |
rules/curation-collection.md |
Content type classification, quality thresholds, duplicate prevention |
| Annotation |
rules/curation-annotation.md |
Multi-agent pipeline, consensus aggregation, Langfuse tracing |
| Diversity |
rules/curation-diversity.md |
Difficulty stratification, domain coverage, balance guidelines |
Management
Versioning, storage, and CI/CD automation for golden datasets.
| Rule |
File |
Key Pattern |
| Versioning |
rules/management-versioning.md |
JSON backup format, embedding regeneration, disaster recovery |
| Storage |
rules/management-storage.md |
Backup strategies, URL contract, data integrity checks |
| CI Integration |
rules/management-ci.md |
GitHub Actions automation, pre-deployment validation, weekly backups |
Validation
Quality scoring, drift detection, and regression testing for golden datasets.
| Rule |
File |
Key Pattern |
| Quality |
rules/validation-quality.md |
Schema validation, content quality, referential integrity |
| Drift |
rules/validation-drift.md |
Duplicate detection, semantic similarity, coverage gap analysis |
| Regression |
rules/validation-regression.md |
Difficulty distribution, pre-commit hooks, full dataset validation |
Add Workflow
Structured workflow for adding new documents to the golden dataset.
| Rule |
File |
Key Pattern |
| Add Document |
rules/curation-add-workflow.md |
9-phase curation, parallel quality analysis, bias detection |
Quick Start Example
from app.shared.services.embeddings import embed_text
async def validate_before_add(document: dict, source_url_map: dict) -> dict:
"""Pre-addition validation for golden dataset entries."""
errors = []
# 1. URL contract check
if "placeholder" in document.get("source_url", ""):
errors.append("URL must be canonical, not a placeholder")
# 2. Content quality
if len(document.get("title", "")) < 10:
errors.append("Title too short (min 10 chars)")
# 3. Tag requirements
if len(document.get("tags", [])) < 2:
errors.append("At least 2 domain tags required")
return {"valid": len(errors) == 0, "errors": errors}
Key Decisions
| Decision |
Recommendation |
| Backup format |
JSON (version controlled, portable) |
| Embedding storage |
Exclude from backup (regenerate on restore) |
| Quality threshold |
>= 0.70 quality score for inclusion |
| Confidence threshold |
>= 0.65 for auto-include |
| Duplicate threshold |
>= 0.90 similarity blocks, >= 0.85 warns |
| Min tags per entry |
2 domain tags |
| Min test queries |
3 per document |
| Difficulty balance |
Trivial 3, Easy 3, Medium 5, Hard 3 minimum |
| CI frequency |
Weekly automated backup (Sunday 2am UTC) |
Common Mistakes
- Using placeholder URLs instead of canonical source URLs
- Skipping embedding regeneration after restore
- Not validating referential integrity between documents and queries
- Over-indexing on articles (neglecting tutorials, research papers)
- Missing difficulty distribution balance in test queries
- Not running verification after backup/restore operations
- Testing restore procedures in production instead of staging
- Committing SQL dumps instead of JSON (not version-control friendly)
Evaluations
See test-cases.json for 9 test cases across all categories.
Related Skills
ork:rag-retrieval - Retrieval evaluation using golden datasetlangfuse-observability - Tracing patterns for curation workflowsork:testing-unit - Unit testing patterns and strategiesai-native-development - Embedding generation for restore
Capability Details
curation
Keywords: golden dataset, curation, content collection, annotation, quality criteria
Solves:
- Classify document content types for golden dataset
- Run multi-agent quality analysis pipelines
- Generate test queries for new documents
management
Keywords: golden dataset, backup, restore, versioning, disaster recovery
Solves:
- Backup and restore golden datasets with JSON
- Regenerate embeddings after restore
- Automate backups with CI/CD
validation
Keywords: golden dataset, validation, schema, duplicate detection, quality metrics
Solves:
- Validate entries against document schema
- Detect duplicate or near-duplicate entries
- Analyze dataset coverage and distribution gaps
Converted and distributed by TomeVault — claim your Tome and manage your conversions.
1---2name: golden-dataset3description: Golden dataset lifecycle patterns for curation, versioning, quality validation, and CI integration. Use when building evaluation datasets, managing dataset versions, validating quality scores, or integrating golden tests into pipelines. Use when this capability is needed.4---56# Golden Dataset78Comprehensive patterns for building, managing, and validating golden datasets for AI/ML evaluation. Each category has individual rule files in `rules/` loaded on-demand.910## Quick Reference1112| Category | Rules | Impact | When to Use |13| -------- | ----- | ------ | ----------- |14| [Curation](#curation) | 3 | HIGH | Content collection, annotation pipelines, diversity analysis |15| [Management](#management) | 3 | HIGH | Versioning, backup/restore, CI/CD automation |16| [Validation](#validation) | 3 | CRITICAL | Quality scoring, drift detection, regression testing |17| [Add Workflow](#add-workflow) | 1 | HIGH | 9-phase curation, quality scoring, bias detection, silver-to-gold |1819Total: 10 rules across 4 categories2021## Curation2223Content collection, multi-agent annotation, and diversity analysis for golden datasets.2425| Rule | File | Key Pattern |26| ---- | ---- | ----------- |27| Collection | `rules/curation-collection.md` | Content type classification, quality thresholds, duplicate prevention |28| Annotation | `rules/curation-annotation.md` | Multi-agent pipeline, consensus aggregation, Langfuse tracing |29| Diversity | `rules/curation-diversity.md` | Difficulty stratification, domain coverage, balance guidelines |3031## Management3233Versioning, storage, and CI/CD automation for golden datasets.3435| Rule | File | Key Pattern |36| ---- | ---- | ----------- |37| Versioning | `rules/management-versioning.md` | JSON backup format, embedding regeneration, disaster recovery |38| Storage | `rules/management-storage.md` | Backup strategies, URL contract, data integrity checks |39| CI Integration | `rules/management-ci.md` | GitHub Actions automation, pre-deployment validation, weekly backups |4041## Validation4243Quality scoring, drift detection, and regression testing for golden datasets.4445| Rule | File | Key Pattern |46| ---- | ---- | ----------- |47| Quality | `rules/validation-quality.md` | Schema validation, content quality, referential integrity |48| Drift | `rules/validation-drift.md` | Duplicate detection, semantic similarity, coverage gap analysis |49| Regression | `rules/validation-regression.md` | Difficulty distribution, pre-commit hooks, full dataset validation |5051## Add Workflow5253Structured workflow for adding new documents to the golden dataset.5455| Rule | File | Key Pattern |56| ---- | ---- | ----------- |57| Add Document | `rules/curation-add-workflow.md` | 9-phase curation, parallel quality analysis, bias detection |5859## Quick Start Example6061```python62from app.shared.services.embeddings import embed_text6364async def validate_before_add(document: dict, source_url_map: dict) -> dict:65 """Pre-addition validation for golden dataset entries."""66 errors = []6768 # 1. URL contract check69 if "placeholder" in document.get("source_url", ""):70 errors.append("URL must be canonical, not a placeholder")7172 # 2. Content quality73 if len(document.get("title", "")) < 10:74 errors.append("Title too short (min 10 chars)")7576 # 3. Tag requirements77 if len(document.get("tags", [])) < 2:78 errors.append("At least 2 domain tags required")7980 return {"valid": len(errors) == 0, "errors": errors}81```8283## Key Decisions8485| Decision | Recommendation |86| -------- | -------------- |87| Backup format | JSON (version controlled, portable) |88| Embedding storage | Exclude from backup (regenerate on restore) |89| Quality threshold | >= 0.70 quality score for inclusion |90| Confidence threshold | >= 0.65 for auto-include |91| Duplicate threshold | >= 0.90 similarity blocks, >= 0.85 warns |92| Min tags per entry | 2 domain tags |93| Min test queries | 3 per document |94| Difficulty balance | Trivial 3, Easy 3, Medium 5, Hard 3 minimum |95| CI frequency | Weekly automated backup (Sunday 2am UTC) |9697## Common Mistakes98991. Using placeholder URLs instead of canonical source URLs1002. Skipping embedding regeneration after restore1013. Not validating referential integrity between documents and queries1024. Over-indexing on articles (neglecting tutorials, research papers)1035. Missing difficulty distribution balance in test queries1046. Not running verification after backup/restore operations1057. Testing restore procedures in production instead of staging1068. Committing SQL dumps instead of JSON (not version-control friendly)107108## Evaluations109110See `test-cases.json` for 9 test cases across all categories.111112## Related Skills113114- `ork:rag-retrieval` - Retrieval evaluation using golden dataset115- `langfuse-observability` - Tracing patterns for curation workflows116- `ork:testing-unit` - Unit testing patterns and strategies117- `ai-native-development` - Embedding generation for restore118119## Capability Details120121### curation122123**Keywords:** golden dataset, curation, content collection, annotation, quality criteria124125**Solves:**126127- Classify document content types for golden dataset128- Run multi-agent quality analysis pipelines129- Generate test queries for new documents130131### management132133**Keywords:** golden dataset, backup, restore, versioning, disaster recovery134135**Solves:**136137- Backup and restore golden datasets with JSON138- Regenerate embeddings after restore139- Automate backups with CI/CD140141### validation142143**Keywords:** golden dataset, validation, schema, duplicate detection, quality metrics144145**Solves:**146147- Validate entries against document schema148- Detect duplicate or near-duplicate entries149- Analyze dataset coverage and distribution gaps150151---152> Converted and distributed by [TomeVault](https://tomevault.io/claim/yonatangross) — claim your Tome and manage your conversions.153<!-- tomevault:4.0:skill_md:2026-04-11 -->
tomevault-io