Source-Type Registry
A research corpus outgrows "papers" — it catalogs preprints, code repos, blog
posts, lab announcements, vendor docs, books, RFCs, discussions, and more. This
skill is the canonical, extensible source-type registry that makes source
type a first-class, config-driven dimension.
The problem it solves
Corpora accumulate three drifting type vocabularies:
| Surface |
Example values |
Drift |
frontmatter type: |
book, reference, gap-note, internal-research |
mixes source type with doc role |
frontmatter source_type: |
conference-paper / conference_paper, book_chapter |
hyphen vs underscore |
| body "Source Type" |
paper, maintainer-doc, discussion |
a third overlapping enum |
The registry folds all three (plus a venue-classification fallback) into one
canonical source type per artifact.
How to use
# List the registry — canonical types + per-type rules
aiwg corpus source-types
aiwg corpus source-types --json
# The by-source-type index view groups the corpus by normalized type
aiwg index build --graph by-source-type # → indices/by-source-type.md
What each type declares
Per canonical type: template, required-sections, citation-format,
acquisition, storage, quality-rules, default-radar-cadence. Example:
| type |
template |
citation |
acquisition |
storage |
cadence |
| paper |
reference-academic |
doi-bibtex |
pdf-download |
sources/pdfs/full |
quarterly |
| preprint |
reference-academic |
arxiv-id |
pdf-download |
sources/pdfs/full |
quarterly |
| blog |
reference-web |
url-venue-retrieved |
web-snapshot |
sources/web |
biannual |
| repo |
reference-repo |
repo-url-commit |
git-clone |
sources/repos |
on-demand |
| standard |
reference-web |
standard-id |
web-snapshot |
sources/web |
annual |
| video |
reference-media |
timestamp-transcript |
media-curator |
media/video |
on-demand |
| podcast |
reference-media |
timestamp-transcript |
media-curator |
media/audio |
on-demand |
(17 canonical types ship by default; aiwg corpus source-types lists them all.)
Adding a new source type is config, not code
Create documentation/source-types.yaml in your corpus to replace the
default registry (include the defaults you keep). Adding a podcast, dataset, or
talk type is a registry entry:
version: 1
types:
podcast:
description: Podcast episode.
aliases: [podcast, episode]
template: reference-media
required_sections: [Citation, Media Profile, Summary, Key Timestamps]
citation_format: timestamp-transcript
acquisition: media-curator
storage: media/audio
quality_rules: interview-hedged-grade
default_radar_cadence: on-demand
venue_fallback: { … }
meta_roles: [redirect, stub, gap-note]
Normalization rules
source_type: → type: → body "Source Type" are checked in that order; the
first that matches a canonical type or alias wins.- Doc-role values (
redirect, stub, gap-note, merged, index) map to
the meta pseudo-type (excluded from source-type analytics).
- If no explicit type matches, the classified venue falls back to a type
(academic venues →
paper, arXiv → preprint, GitHub → repo, RFC →
standard, lab/vendor research posts → blog, Wikipedia → encyclopedia).
- Otherwise →
other.
Validated on a real 1,273-ref corpus: the registry normalizes ~93% (paper,
preprint, blog, repo, standard, book, chapter, encyclopedia, internal-review),
with meta excluding redirects/stubs and other capturing genuinely untyped refs.
Consumers
The registry is the foundation other subsystems read:
- by-source-type index view (this framework) — groups refs by normalized type.
- Per-type induction audit (
induction-audit / #1504) — required-section + depth checks vary by type (don't flag a blog for missing Ablation Studies).
- Acquisition dispatch (
research-acquire / #1507) — PDF download vs web snapshot vs git clone by type.
- Templates (#1497) — per-type reference templates selected by source type.
- Quality/GRADE — non-peer-reviewed types carry different hedging expectations.
Triggers
- "source type registry"
- "normalize source types"
- "what source types does the corpus have"
- "add a new source type"
- "by source type"
Notes
- Authoritative runtime default:
src/artifacts/corpus-tools/source-types.ts
(DEFAULT_SOURCE_TYPES); human-readable + override form:
agentic/code/frameworks/research-complete/config/source-types.yaml. A drift
test keeps them in sync.
- The venue fallback reuses the existing
VENUE_PATTERNS classifier
(src/artifacts/corpus-views/taxonomies.ts).
1---2name: source-types3description: The canonical, extensible source-type registry for a research corpus — paper, preprint, blog, repo, book, chapter, standard, doc, discussion, encyclopedia, expert-material, video, audio, podcast, lecture, internal-review. Replaces the drifting type / source_type / "Source Type" vocabularies with one registry that declares per-type template, required sections, citation format, acquisition method, storage, quality rules, and radar cadence. Surfaced via `aiwg corpus source-types`; consumed by the by-source-type index view, per-type induction audit, and acquisition dispatch.4---56# Source-Type Registry78A research corpus outgrows "papers" — it catalogs preprints, code repos, blog9posts, lab announcements, vendor docs, books, RFCs, discussions, and more. This10skill is the **canonical, extensible source-type registry** that makes source11type a first-class, config-driven dimension.1213## The problem it solves1415Corpora accumulate **three drifting type vocabularies**:1617| Surface | Example values | Drift |18|---------|---------------|-------|19| frontmatter `type:` | `book`, `reference`, `gap-note`, `internal-research` | mixes source type with doc role |20| frontmatter `source_type:` | `conference-paper` / `conference_paper`, `book_chapter` | hyphen vs underscore |21| body "Source Type" | `paper`, `maintainer-doc`, `discussion` | a third overlapping enum |2223The registry folds all three (plus a venue-classification fallback) into **one24canonical source type** per artifact.2526## How to use2728```bash29# List the registry — canonical types + per-type rules30aiwg corpus source-types31aiwg corpus source-types --json3233# The by-source-type index view groups the corpus by normalized type34aiwg index build --graph by-source-type # → indices/by-source-type.md35```3637## What each type declares3839Per canonical type: `template`, `required-sections`, `citation-format`,40`acquisition`, `storage`, `quality-rules`, `default-radar-cadence`. Example:4142| type | template | citation | acquisition | storage | cadence |43|------|----------|----------|-------------|---------|---------|44| paper | reference-academic | doi-bibtex | pdf-download | sources/pdfs/full | quarterly |45| preprint | reference-academic | arxiv-id | pdf-download | sources/pdfs/full | quarterly |46| blog | reference-web | url-venue-retrieved | web-snapshot | sources/web | biannual |47| repo | reference-repo | repo-url-commit | git-clone | sources/repos | on-demand |48| standard | reference-web | standard-id | web-snapshot | sources/web | annual |49| video | reference-media | timestamp-transcript | media-curator | media/video | on-demand |50| podcast | reference-media | timestamp-transcript | media-curator | media/audio | on-demand |5152(17 canonical types ship by default; `aiwg corpus source-types` lists them all.)5354## Adding a new source type is config, not code5556Create `documentation/source-types.yaml` in your corpus to **replace** the57default registry (include the defaults you keep). Adding a podcast, dataset, or58talk type is a registry entry:5960```yaml61version: 162types:63 podcast:64 description: Podcast episode.65 aliases: [podcast, episode]66 template: reference-media67 required_sections: [Citation, Media Profile, Summary, Key Timestamps]68 citation_format: timestamp-transcript69 acquisition: media-curator70 storage: media/audio71 quality_rules: interview-hedged-grade72 default_radar_cadence: on-demand73venue_fallback: { … }74meta_roles: [redirect, stub, gap-note]75```7677## Normalization rules78791. `source_type:` → `type:` → body "Source Type" are checked in that order; the80 first that matches a canonical type or alias wins.812. Doc-role values (`redirect`, `stub`, `gap-note`, `merged`, `index`) map to82 the **`meta`** pseudo-type (excluded from source-type analytics).833. If no explicit type matches, the classified **venue** falls back to a type84 (academic venues → `paper`, arXiv → `preprint`, GitHub → `repo`, RFC →85 `standard`, lab/vendor research posts → `blog`, Wikipedia → `encyclopedia`).864. Otherwise → `other`.8788Validated on a real 1,273-ref corpus: the registry normalizes ~93% (paper,89preprint, blog, repo, standard, book, chapter, encyclopedia, internal-review),90with `meta` excluding redirects/stubs and `other` capturing genuinely untyped refs.9192## Consumers9394The registry is the foundation other subsystems read:9596- **by-source-type index view** (this framework) — groups refs by normalized type.97- **Per-type induction audit** (`induction-audit` / #1504) — required-section + depth checks vary by type (don't flag a blog for missing Ablation Studies).98- **Acquisition dispatch** (`research-acquire` / #1507) — PDF download vs web snapshot vs git clone by type.99- **Templates** (#1497) — per-type reference templates selected by source type.100- **Quality/GRADE** — non-peer-reviewed types carry different hedging expectations.101102## Triggers103104- "source type registry"105- "normalize source types"106- "what source types does the corpus have"107- "add a new source type"108- "by source type"109110## Notes111112- Authoritative runtime default: `src/artifacts/corpus-tools/source-types.ts`113 (`DEFAULT_SOURCE_TYPES`); human-readable + override form:114 `agentic/code/frameworks/research-complete/config/source-types.yaml`. A drift115 test keeps them in sync.116- The venue fallback reuses the existing `VENUE_PATTERNS` classifier117 (`src/artifacts/corpus-views/taxonomies.ts`).
jmagly