Golden Dataset Curation

Golden Dataset Curation

Curate high-quality documents for the golden dataset with multi-agent validation

Overview

This skill provides patterns and workflows for adding new documents to the golden dataset with thorough quality analysis. It complements golden-dataset-management which handles backup/restore.

When to use this skill:

• Adding new documents to the golden dataset
• Classifying content types and difficulty levels
• Generating test queries for new documents
• Running multi-agent quality analysis

Content Types

Difficulty Levels

Quality Dimensions

Evaluation focuses:

• Accuracy: Technical correctness, code validity, up-to-date info
• Coherence: Logical structure, clear flow, consistent terminology
• Depth: Comprehensive coverage, edge cases, appropriate detail
• Relevance: Alignment with AI/ML, backend, frontend, DevOps domains

Multi-Agent Pipeline

INPUT: URL/Content
        |
        v
+------------------+
|   FETCH AGENT    |  Extract structure, detect type
+--------+---------+
         |
         v
+-----------------------------------------------+
|  PARALLEL ANALYSIS AGENTS                      |
|  Quality | Difficulty | Domain  | Query Gen   |
+-----------------------------------------------+
         |
         v
+------------------+
| CONSENSUS        |  Weighted score + confidence
| AGGREGATOR       |  -> include/review/exclude
+--------+---------+
         |
         v
+------------------+
|  USER APPROVAL   |  Show scores, confirm
+--------+---------+
         |
         v
OUTPUT: Curated document entry

Decision Thresholds

Quality Thresholds

# Recommended thresholds for golden dataset inclusion
minimum_quality_score: 0.70
minimum_confidence: 0.65
required_tags: 2          # At least 2 domain tags
required_queries: 3       # At least 3 test queries

Coverage Balance Guidelines

Maintain balanced coverage across:

• Content types: Don't over-index on articles
• Difficulty levels: Need trivial AND hard queries
• Domains: Spread across AI/ML, backend, frontend, etc.

Duplicate Prevention Checklist

Before adding:

1. Check URL against existing source_url_map.json
2. Run semantic similarity against existing document embeddings
3. Warn if >80% similar to existing document

Provenance Tracking

Always record:

• Source URL (canonical)
• Curation date
• Agent scores (for audit trail)
• Langfuse trace ID

Langfuse Integration

Trace Structure

trace = langfuse.trace(
    name="golden-dataset-curation",
    metadata={"source_url": url, "document_id": doc_id}
)

# Log individual dimension scores
trace.score(name="accuracy", value=0.85)
trace.score(name="coherence", value=0.90)
trace.score(name="depth", value=0.78)
trace.score(name="relevance", value=0.92)

# Final aggregated score
trace.score(name="quality_total", value=0.87)
trace.event(name="curation_decision", metadata={"decision": "include"})

Managed Prompts

References

For detailed implementation patterns, see:

• references/selection-criteria.md - Content type classification, difficulty stratification, quality evaluation dimensions, and best practices
• references/annotation-patterns.md - Multi-agent pipeline architecture, agent specifications, consensus aggregation logic, and Langfuse integration

Related Skills

• golden-dataset-management - Backup/restore operations
• golden-dataset-validation - Validation rules and checks
• langfuse-observability - Tracing patterns
• pgvector-search - Duplicate detection

Version: 1.0.0 (December 2025) Issue: #599

Capability Details

content-classification

Keywords: content type, classification, document type, golden dataset Solves:

• Classify document content types for golden dataset
• Categorize entries by domain and purpose
• Identify content requiring special handling

difficulty-stratification

Keywords: difficulty, stratification, complexity level, challenge rating Solves:

• Assign difficulty levels to golden dataset entries
• Ensure balanced difficulty distribution
• Identify edge cases and challenging examples

quality-evaluation

Keywords: quality, evaluation, quality dimensions, quality criteria Solves:

• Evaluate entry quality against defined criteria
• Score entries on multiple quality dimensions
• Identify entries needing improvement

multi-agent-analysis

Keywords: multi-agent, parallel analysis, consensus, agent evaluation Solves:

• Run parallel agent evaluations on entries
• Aggregate consensus from multiple analysts
• Resolve disagreements in classifications