Deep Research Reports

Deep Research Report Processor

Processes deep research reports (from ChatGPT, Claude, or other platforms) generated by the deep-research-genelist skill. Cleans platform-specific artifacts, generates publication-quality PDF and/or HTML, extracts the YAML header, and maintains a growing annotation summary table.

Workflow

Step 1: Identify Input

Accept one of:

• A specific file path provided by the user
• "all" — glob outs/deep_research*/**/*.md (note the * after deep_research — catches deep_research/, deep_research_platynereis/, etc.), skip files ending in _clean.md, _raw.md, or _prompt.md
• "new" — glob outs/deep_research*/**/*.md, same exclusions, then keep only files that don't yet have a corresponding _clean.md

If no outs/deep_research*/ directories exist, tell the user and stop.

Directory structure

Each module gets its own subdirectory under outs/deep_research/:

outs/deep_research/
├── annotation_summary.tsv          ← cross-cluster summary (top level)
├── clade6sub25/                    ← per-module subdirectory
│   ├── clade6sub25_prompt.md       ← shared prompt (no date/platform prefix)
│   ├── 260304_chatgpt_clade6sub25_raw.md
│   ├── 260304_chatgpt_clade6sub25_clean.md
│   ├── 260304_chatgpt_clade6sub25_report.pdf
│   ├── 260304_chatgpt_clade6sub25_report.html
│   ├── 260304_claude_clade6sub25_clean.md
│   ├── 260304_claude_clade6sub25_report.pdf
│   └── 260304_claude_clade6sub25_report.html
└── another_cluster/
    └── ...

Create the subdirectory outs/deep_research/{module_id}/ if it doesn't exist. All output files for that module go inside it.

Step 2: Detect Platform and Clean

Read the file as raw bytes. Detect the platform from artifact signatures:

Artifact-free reports (no PUA, no entity tags) could be from Claude OR ChatGPT Pro (extended thinking). These two platforms produce clean output with no artifacts. When no artifacts are detected, ask the user which platform generated the report using AskUserQuestion:

• chatgpt_pro — ChatGPT Pro / extended thinking / o-series models
• claude — Claude (Anthropic)

This matters for the canonical filename (260330_chatgpt_pro_family_C vs 260330_claude_family_C) and for the summary table platform column.

Report detection to user: "Detected {platform} report." (or "No artifacts detected — asking platform.")

Parse YAML header first

Before cleaning, extract the YAML front matter (between first --- line and next --- line) to get module_id and date_generated. These are needed for the canonical filename.

Construct canonical base name

Pattern depends on report type:

• Cluster reports: YYMMDD_platform_moduleID (e.g., 260304_chatgpt_clade6sub25)
• Family reports: YYMMDD_platform_family_FAMILYID (e.g., 260330_claude_family_C)
• Nonmetazoan: YYMMDD_platform_kingdom (e.g., 260315_chatgpt_prokaryote)

Components:

• YYMMDD from query.date_generated (format: YYYY-MM-DD → YYMMDD). Fall back to today's date.
• platform: chatgpt, chatgpt_pro, or claude (lowercase)
• moduleID: from query.module_id (cluster reports)
• FAMILYID: from query.family (family reports — note: NOT query.module_id)
• kingdom: from query.kingdom (nonmetazoan reports)

Platform-specific cleaning

ChatGPT reports (have artifacts):

Cleaning order is critical — PUA characters must be stripped first or other patterns won't match.

import re

# 1. Strip all PUA characters (U+E000–U+F8FF) — MUST be first
text = re.sub(r'[\ue000-\uf8ff]', '', text)

# 2. Replace entity["type","name","desc"] → name
text = re.sub(r'entity\["[^"]*","([^"]*)","[^"]*"\]', r'\1', text)

# 3. Remove citeturn... strings (the [N] refs are already inline)
text = re.sub(r'\s*citeturn\S+', '', text)

# 4. Remove image_group{...} lines
text = re.sub(r'^.*image_group\{.*\}.*$', '', text, flags=re.MULTILINE)

# 5. Normalize whitespace
text = re.sub(r'  +', ' ', text)           # collapse double spaces
text = re.sub(r'\n{3,}', '\n\n', text)     # collapse triple+ newlines
text = re.sub(r' +$', '', text, flags=re.MULTILINE)  # strip trailing whitespace

Output files:

• Rename raw file to {base}_raw.md (preserves original with artifacts for reference)
• Write cleaned text to {base}_clean.md

Report artifact counts: "Removed X entity tags, Y citation markers, Z PUA characters."

Claude reports (already clean):

Claude deep research outputs have no known artifacts (verified: zero PUA chars, zero entity tags, zero citeturn markers).

• Rename the original file directly to {base}_clean.md (no separate _raw.md since there are no artifacts to preserve)
• Apply only whitespace normalization (collapse blank lines, strip trailing spaces)

Report: "Claude report — no artifacts detected. Renamed to {base}_clean.md."

Step 3: Detect Report Type and Validate YAML Header

Parse the YAML front matter from the clean file using Python yaml.safe_load().

Report type detection: Check query.report_type to determine which parser to use:

• "cluster" (or absent) → cluster cell-type-annotation parser
• "family" → legacy family cell-type-annotation parser (see Step 3c below) — used by Platynereis 2026-04-04 batch
• "family_report1" → Family Report 1 (WGCNA-module pipeline) Validation (see Step 3d below) — the v2/v3 chiton family pipeline (scripts/17_family_report_prompts.py)
• "family_report2" → Family Report 2 (per-triage-group) Validation (see Step 3e below) — per-triage-group cluster-level deep dives
• "nonmetazoan_characterization" → nonmetazoan parser (see Step 3b below)

ChatGPT YAML indentation fix: ChatGPT inconsistently outputs YAML with 1-space or 2-space indentation across runs. 1-space indent is valid YAML for simple key-value mappings, but breaks for two specific constructs:

1. Folded block scalars (summary: >) — the continuation text lands at the same indent as sibling keys, so the parser can't distinguish block scalar content from a new key
2. List item mapping continuations (- cell_type: "..." followed by organism: "...") — continuation keys land at the same indent as the - marker, so they're parsed as siblings rather than part of the list item's mapping

Important: Do NOT "double all indentation" — this breaks normal mapping keys by making children indent past their siblings. Instead, apply a targeted fix that only touches the two broken patterns:

The fix: try yaml.safe_load() first. If it fails, apply fix_flat_yaml() (for zero-indent ChatGPT Pro output), then fix_chatgpt_yaml() (for 1-space indent issues), then retry.

fix_flat_yaml — zero-indent ChatGPT Pro / extended thinking output

ChatGPT Pro with extended thinking outputs YAML with ALL keys at column 0 (zero indent under section headers). This fixer detects and corrects this pattern before the existing fix_chatgpt_yaml runs.

def fix_flat_yaml(yaml_text):
    """Fix YAML where all keys are at column 0 (ChatGPT Pro / extended thinking).

    Detects this pattern: top-level section keys (query:, annotation:, markers:)
    followed by child keys at the same indent level. Adds proper indent to all
    child keys under each section. Distinguishes mapping list items ("- key: val"
    → continuations get 6-space indent) from simple list items ('- "value"' →
    next sibling key resets to 2-space).
    """
    lines = yaml_text.split('\n')
    # Quick check: if we see 'query:' at col 0 followed by 'organism:' at col 0,
    # this is a flat YAML that needs fixing
    has_flat_pattern = False
    for i, line in enumerate(lines):
        if line.strip() == 'query:' and i + 1 < len(lines):
            next_line = lines[i + 1]
            if next_line and not next_line.startswith(' ') and ':' in next_line:
                has_flat_pattern = True
                break
    
    if not has_flat_pattern:
        return yaml_text  # Already properly indented
    
    fixed = []
    section = None
    section_keys = {'query', 'annotation', 'markers', 'classification',
                    'functional_categories', 'hgt_candidates', 'expression_patterns',
                    'biology',
                    # family_report1 (WGCNA-pipeline Report 1)
                    'report2_triage',
                    # family_report2 (per-triage-group)
                    'intra_group_differences', 'cross_family_implications',
                    'atlas_synthesis_inputs'}
    in_block_scalar = False
    in_mapping_list = False  # inside "- key: val" list item (continuations need 6-space)
    in_simple_list = False   # inside "- value" list (next sibling key resets to 2-space)

    for line in lines:
        stripped = line.strip()
        n_spaces = len(line) - len(line.lstrip())

        if not stripped:
            in_block_scalar = False
            in_mapping_list = False
            in_simple_list = False
            fixed.append('')
            continue

        # Block scalar continuation
        if in_block_scalar and n_spaces == 0 and not any(
            stripped.startswith(k + ':') for k in section_keys
        ):
            fixed.append('    ' + stripped)
            continue
        elif in_block_scalar:
            in_block_scalar = False

        # Top-level section key
        key_candidate = stripped.split(':')[0].strip()
        if n_spaces == 0 and key_candidate in section_keys and stripped.endswith(':'):
            section = key_candidate
            in_mapping_list = False
            in_simple_list = False
            fixed.append(line)
            continue

        # Detect block scalar start
        if stripped.endswith(': >') or stripped.endswith(': |'):
            in_block_scalar = True

        # Child content under a section at col 0
        if section and n_spaces == 0:
            if stripped.startswith('---'):
                section = None
                in_mapping_list = False
                in_simple_list = False
                fixed.append(line)
                continue
            if key_candidate in section_keys:
                section = key_candidate
                in_mapping_list = False
                in_simple_list = False
                fixed.append(line)
                continue
            # List items
            if stripped.startswith('- '):
                after_dash = stripped[2:].strip()
                if re.match(r'[\w_-]+\s*:', after_dash):
                    in_mapping_list = True
                    in_simple_list = False
                else:
                    in_simple_list = True
                    in_mapping_list = False
                fixed.append('    ' + stripped)
                continue
            # Continuation key inside a mapping list item
            if in_mapping_list and re.match(r'[\w_-]+\s*:', stripped):
                fixed.append('      ' + stripped)
                continue
            # Regular key (not inside a list, or after simple list ended)
            in_mapping_list = False
            in_simple_list = False
            fixed.append('  ' + stripped)
            continue

        # Already indented — add 2 more spaces for section nesting
        if section and n_spaces > 0:
            fixed.append('  ' + line)
            continue

        fixed.append(line)

    
    return '\n'.join(fixed)

fix_chatgpt_yaml — 1-space indent issues

This existing fixer handles the two specific constructs broken by 1-space indentation:

• After a line ending in : > or : |, adds 1 extra space to continuation lines (non-key, non-blank lines at the same indent) until a blank line or a real key is encountered
• After a line matching - key: value (list item mapping start), adds 2 extra spaces to subsequent non-- key lines at the same indent (these are continuation keys inside the mapping)
• Leaves all other indentation unchanged

def fix_chatgpt_yaml(yaml_text):
    """Fix ChatGPT 1-space indent YAML: folded block scalars + list item mapping continuations."""
    lines = yaml_text.split('\n')
    fixed = []
    in_block_scalar = False
    block_scalar_key_indent = -1
    in_list_item = False
    list_item_indent = -1

    for i, line in enumerate(lines):
        stripped = line.lstrip(' ')
        n_spaces = len(line) - len(stripped)

        if stripped == '':
            in_block_scalar = False
            fixed.append(line)
            continue

        # FIX 1: Folded/literal block scalar continuation
        if in_block_scalar:
            if (n_spaces == block_scalar_key_indent
                    and not re.match(r'[\w-]+\s*:', stripped)
                    and not stripped.startswith('- ')):
                fixed.append(' ' * (n_spaces + 1) + stripped)
                continue
            else:
                in_block_scalar = False

        # FIX 2: List item mapping continuation
        if in_list_item:
            if (n_spaces == list_item_indent
                    and re.match(r'[\w_-]+\s*:', stripped)
                    and not stripped.startswith('- ')):
                fixed.append(' ' * (n_spaces + 2) + stripped)
                continue
            elif n_spaces <= list_item_indent and not (
                    n_spaces == list_item_indent and stripped.startswith('- ')):
                in_list_item = False

        if re.search(r':\s*[>|]\s*$', stripped):
            in_block_scalar = True
            block_scalar_key_indent = n_spaces

        if re.match(r'- [\w_-]+\s*:', stripped):
            in_list_item = True
            list_item_indent = n_spaces

        fixed.append(line)

    return '\n'.join(fixed)

This handles both 1-space and 2-space ChatGPT output without breaking valid YAML.

Validate required fields:

Required:

• query.module_id, query.organism, query.n_genes
• annotation.proposed_name, annotation.confidence, annotation.one_line
• markers.top_diagnostic (non-empty list)

Optional but checked:

• query.source_object, query.clustering_column, query.marker_file — warn if missing (these are provenance fields added in v2 of the genelist skill; older prompts won't have them)
• query.comparison_mode, query.clade_family — present when merged marker mode was used; absent for single-list prompts or older prompts. Default to empty string in summary table.
• query.report_type — "cluster" (default) or "family". Family reports are generated by the family-aware mode of the genelist skill.
• query.member_clusters — list of fine cluster names (family reports only). Default to empty list.
• query.n_member_clusters — integer count (family reports only). Default to 0.

If validation fails, report which fields are missing and ask whether to proceed.

Step 3b: Nonmetazoan Report Validation

When query.report_type == "nonmetazoan_characterization", use this validation instead:

Detect template variant from which classification fields are present:

• Has classification.n_candidate_hgt → prokaryote variant (Bacteria, Archaea, combined)
• Has classification.n_database_bias → eukaryote variant (Fungi, Viridiplantae, etc.)
• Has both or neither → warn and proceed with best guess

Required fields (both variants):

• query.kingdom, query.organism, query.n_genes, query.n_expressed
• classification.assessment_confidence
• At least one entry in functional_categories

Prokaryote-specific required:

• classification.n_symbiont_transcript, classification.n_candidate_hgt, classification.n_conserved, classification.n_ambiguous (can be 0)

Eukaryote-specific required:

• classification.n_database_bias, classification.n_conserved, classification.n_symbiont_transcript, classification.n_lateral_transfer, classification.n_ambiguous (can be 0)

Optional but checked:

• hgt_candidates — warn if absent for prokaryote reports
• expression_patterns.cell_type_enriched_genes — warn if empty
• biology.symbiosis_relevant — prokaryote only
• biology.evolutionary_insight — eukaryote only

Canonical base name for nonmetazoan reports: Pattern: YYMMDD_platform_kingdom (e.g., 260315_chatgpt_prokaryote, 260315_claude_fungi)

• kingdom from query.kingdom, lowercased, spaces replaced with underscores

Directory: Reports go in outs/deep_research/nonmetazoan/{kingdom}/ (note: under deep_research/nonmetazoan/, not the script-04 outs/scmicrobiome/ directory — the processed reports live alongside cell-type-annotation reports).

Step 3c: Family Report Validation

When query.report_type == "family", use this validation:

Required:

• query.family, query.organism, query.n_modules or query.n_member_clusters
• annotation.proposed_family_name, annotation.confidence, annotation.one_line
• annotation.per_cluster (non-empty list with at least cluster and proposed_name per entry)

Optional but checked:

• query.subfamilies — subfamily structure. Default empty.
• annotation.best_matches — comparative matches. Default empty list.
• annotation.family_conservation — conservation level. Default empty.
• markers.family_defining — family marker list. Warn if empty.
• markers.family_specifying_tfs, markers.cluster_specifying_tfs — TF annotations.

Canonical base name: YYMMDD_platform_family_FAMILYID

Directory: Reports go in the same directory as the input file (family reports already live in their own subdirectory, e.g., outs/deep_research_platynereis/family_C/).

Step 3d: Family Report 1 Validation (WGCNA-pipeline v2/v3)

Used when query.report_type == "family_report1". Generated by scripts/17_family_report_prompts.py in the WGCNA-module family-level pipeline. Design rationale lives in each project's .claude/REPORT_PIPELINE_DECISIONS.md (see References at the end of this skill).

Required fields:

• query.organism, query.family, query.member_clusters, query.date_generated
• annotation.proposed_family_name, annotation.confidence, annotation.one_line
• annotation.family_specifying_tfs (warn if empty)
• report2_triage.groups (warn if absent — means the LLM didn't follow the triage policy)

Optional but parsed:

• annotation.confidence_rationale
• annotation.subfamily_structure
• annotation.evolutionary_classification — one of {pan-animal | clade-specific | lineage-specific-innovation | lineage-specific-variant}. Added 2026-05; v3 prompts emit this explicitly. v2 prompts may have annotation.family_conservation instead — treat both as the same column in the summary.
• annotation.evolutionary_rationale — one-sentence justification
• annotation.best_matches[] — comparative matches (cell_type_family, organism, shared_tfs, shared_effectors, conservation)
• report2_triage.notes

Version detection: Extract v{N} from input filename if present (e.g., user uploads shell_v3.md); otherwise default to v3 (the current prompt template version going forward).

Canonical base name: {YYMMDD}_{platform}_family_{family}_v{N} — matches existing v2 outputs.

Directory: Same directory as input file (typically outs/17_family_report_prompts/{species}/{family}/).

Step 3e: Family Report 2 Validation (per-triage-group)

Used when query.report_type == "family_report2". Generated by the Report 2 branch of scripts/17_family_report_prompts.py (run with --report 2). One Report 2 covers one triage group from Report 1 (singleton or multi-cluster).

Required fields:

• query.organism, query.family, query.triage_group_id, query.cluster_set, query.priority, query.date_generated
• annotation.per_cluster — non-empty list, each entry with at least cluster and proposed_name
• cross_family_implications — list (can be empty list [])
• atlas_synthesis_inputs — list of high-importance findings (can be empty list)

Per-cluster entry fields (each annotation.per_cluster[i]):

• Required: cluster, proposed_name, confidence, one_line
• display_name_short — condensed plot label ≤ 35 chars, for use in single-cell dotplots / UMAPs / heatmaps. Required for v3+ prompts; warn if absent (older runs).
• Optional: confidence_rationale, cluster_specifying_tfs[], key_distinguishing_markers[], subfamily_role, cross_species_match (with sub-fields cell_type, organism, shared_tfs, shared_effectors, conservation)

Multi-cluster groups (when len(query.cluster_set) >= 2) also have:

• intra_group_differences.interpretation ∈ {distinct_cell_types | different_states | transitional_stages | uncertain}
• intra_group_differences.distinguishing_functional_signatures[]
• intra_group_differences.distinguishing_regulatory_signatures[]
• intra_group_differences.interpretation_rationale
• intra_group_differences.algorithmic_verdict_from_panel, agree_with_panel, optionally disagreement_explanation

Legacy / variable fields (accept but don't require):

• merge_recommendation — some Pro outputs include it even though the v3 prompt explicitly dropped the field; accept it as informational, don't validate values.

Group-type derivation (for the subtitle): infer from triage context, not from a YAML field:

• len(cluster_set) >= 2 → merge-candidate
• len(cluster_set) == 1 and the cluster name contains "transitional" or "outlier" tags → outlier
• len(cluster_set) == 1 and the cluster is described as "bridge" or "hybrid" in its subfamily_role → bridge
• Other singletons → singleton or context-specific label (e.g., "shell-field anchor")

When inferring, prefer the subfamily_role field text over heuristics.

Canonical base name: {YYMMDD}_{platform}_family_{family}_{group_id}_report2 where group_id comes from query.triage_group_id (multi-cluster groups already use +-joined names like shell_5+shell_14_1).

Directory: Same directory as input file (typically outs/17_family_report_prompts/{species}/{family}/).

Step 4: Ask Output Format

Use AskUserQuestion:

• Both (Recommended) — PDF for sharing, HTML for browsing with clickable DOIs
• PDF only — LaTeX-rendered, good for sharing/printing
• HTML only — Standalone, nice table styling, clickable links

Step 5: Generate Outputs

Create a temporary _report_for_render.md from {base}_clean.md:

1. Strip the data YAML header — the large query:/annotation:/markers: (or classification:/functional_categories:) block is for programmatic parsing only, not for display

2. Insert a pandoc formatting YAML. The title format depends on report type:

   • Cell-type annotation: "Gene Module Interpretation: *{organism}* {module_id}"
   • Family annotation (legacy family): "Cell Type Family Report: *{organism}* Family {family}"
   • family_report1: "Cell Type Family Report 1: *{organism}* Family {family}" + subtitle includes (v{N})
   • family_report2: "Cell Type Family Report 2: *{organism}* {family} → {group_id}" + subtitle includes ({group_type} group) (group_type derived per Step 3e)
   • Nonmetazoan: "Non-Metazoan Gene Characterization: *{organism}* — {kingdom}"

   ---
   title: "<title per report type>"
   subtitle: "Deep Research Report — {Platform} ({date_generated})"
   geometry: margin=1in
   fontsize: 11pt
   header-includes:
     - \renewcommand{\arraystretch}{1.4}
     - \usepackage{booktabs}
     - \usepackage{longtable}
   ---
   

   The \arraystretch{1.4} adds 40% vertical padding to every table row — critical for readability.

3. Remove the H1 title from the body (it's now in the YAML title: field)

4. Normalize heading levels — ChatGPT reports nest headings too deep, producing tiny headers at H4/H5 in PDFs. Apply these fixes in order:

   a. Remove redundant heading pairs. ChatGPT often outputs a generic ## heading immediately followed by the lettered ### section (e.g., ## Comparative biological analysis then ### H. Comparative Biological Analysis). Remove the redundant ## line:

   body = re.sub(r'^##\s+[^\n]+\n\n(###\s+[A-L]\.)', r'\1', body, flags=re.MULTILINE)
   

   b. Shift all headings up one level. Since the H1 title was removed, the remaining hierarchy is too deep (##→####). Shift ##→#, ###→##, ####→###, etc. (never shift H1):

   def shift_heading(m):
       hashes = m.group(1)
       rest = m.group(2)
       if len(hashes) > 1:
           return '#' * (len(hashes) - 1) + rest
       return m.group(0)
   body = re.sub(r'^(#{2,})([ \t].*)$', shift_heading, body, flags=re.MULTILINE)
   

   After normalization, the typical hierarchy is: # (sections A–K), ## (subsections like "Core identity modules"), ### (sub-subsections).

5. Escape backslashes in gene names for LaTeX. ChatGPT reports sometimes include gene names with literal backslashes (e.g., Dmel\cg5579), which LaTeX interprets as control sequences. Escape them in the body only:

   body = re.sub(r'\\(?=[a-zA-Z])', r'', body)
   

Detect rendering tools: Find quarto on PATH or common locations (~/.local/bin/quarto, /usr/local/bin/quarto). Check for xelatex availability — if not on PATH, try module load texlive (HPC cluster). If no LaTeX available, skip PDF and inform user. On cluster, shell commands for PDF rendering must include module load texlive && prefix.

Important: When writing the pandoc YAML header from Python, write the LaTeX commands directly to the file (not through Python f-strings). The backslashes in \renewcommand, \arraystretch, \usepackage must appear literally in the markdown file — do NOT double-escape them in Python. Use f.write() with raw strings or explicit line writes.

Generate PDF:

{quarto_path} pandoc \
  "{base}_report_for_render.md" \
  -o "{base}_report.pdf" \
  --pdf-engine=xelatex \
  -V colorlinks=true -V linkcolor=blue -V urlcolor=blue

On cluster, prepend module load texlive && before the quarto command.

Generate HTML:

{quarto_path} pandoc \
  "{base}_report_for_render.md" \
  -o "{base}_report.html" \
  --standalone \
  --css="$HOME/.claude/skills/deep-research-reports/templates/report-style.css" \
  --embed-resources

The --embed-resources flag inlines the CSS so the HTML is fully self-contained.

Delete _report_for_render.md after successful conversion.

If pandoc fails (e.g., LaTeX error), fall back to HTML only and notify the user.

Step 6: Update Summary Table

Extract fields from validated YAML and update outs/deep_research/annotation_summary.tsv.

Composite key: module_id + platform + date_generated. This allows the table to hold both ChatGPT and Claude annotations for the same cluster side by side.

Logic:

1. Read existing TSV if it exists (using Python csv with delimiter='\t')
2. Check if a row with matching composite key exists
3. If yes: show old vs new proposed_name, ask to update or skip
4. If no: append new row
5. Sort by module_id, then platform
6. Write back TSV

Columns — parse ALL YAML fields into the summary table:

module_id, organism, common_name, clade, dataset, module_type, report_type, source_object, clustering_column, marker_file, comparison_mode, clade_family, member_clusters, n_member_clusters, biological_context, n_genes, proposed_name, alternative_names, confidence, confidence_rationale, one_line, summary, cell_type_family, family_conservation, top_tfs, top_markers, top_diagnostic_ids, top_diagnostic_roles, receptors_channels, signaling_ligands, adhesion_molecules, secreted_products, key_pathways, metabolic_signature, n_uncharacterized, best_match_1, best_match_2, best_match_3, date_generated, platform, report_file, date_processed

Field mapping from YAML:

Fields may be absent in older reports — default to empty string.

Family report field mappings: For family reports, the summary table uses the same annotation_summary.tsv but with these field mappings:

All other columns map the same as cluster reports. The per_cluster annotations are NOT added to the summary table row — they are available in the clean markdown file for detailed parsing.

Step 6b: Nonmetazoan Summary Table

For nonmetazoan reports (report_type == "nonmetazoan_characterization"), update a separate summary table at outs/deep_research/nonmetazoan_summary.tsv. Do NOT mix with annotation_summary.tsv — the column semantics are fundamentally different.

Composite key: kingdom + platform + date_generated

Columns:

Step 6c: Family Report 1 Summary Table

For family_report1 reports (Step 3d), update outs/17_family_report_prompts/{species}/family_report1_summary.tsv (top-level per-species, aggregates across all families). Do NOT mix with annotation_summary.tsv (cluster-level schema) or family_report2_summary.tsv (per-cluster schema).

Composite key: family + platform + version + date_generated

Columns:

Step 6d: Family Report 2 Summary Table

For family_report2 reports (Step 3e), update outs/17_family_report_prompts/{species}/family_report2_summary.tsv. One row per (family, group_id, cluster) — atlas synthesis (Pass 3) consumes per-cluster rows. Multi-cluster groups expand into multiple rows; group-level fields (group_id, cluster_set, intra_group_*) repeat across cluster rows.

Composite key: family + group_id + cluster + platform + date_generated

Columns:

Step 7: Report Results

Cell-type-annotation reports:

Processed: clade6sub25_annotation_report.md
  Platform: Claude
  Renamed: clade6sub25/260304_claude_clade6sub25_clean.md
  PDF: outs/deep_research/clade6sub25/260304_claude_clade6sub25_report.pdf
  HTML: outs/deep_research/clade6sub25/260304_claude_clade6sub25_report.html
  Summary: annotation_summary.tsv (new row added)

  Annotation: "hemocyte-like immune/scavenger c

…(truncated)