Analysis Campaign
Use this skill when one or more follow-up runs are needed and the quest needs a coordinated evidence campaign.
This is the shared DeepScientist protocol for supplementary experiments after a durable result.
Use the same route for:
- ordinary ablations / robustness / sensitivity work
- review-driven evidence gaps
- rebuttal-driven extra experiments
- writing-driven evidence gaps
For paper-facing work, treat “analysis campaign” broadly:
- not only post-hoc interpretation
- also ablations, sensitivity checks, robustness checks, efficiency or cost checks, highlight-validation runs, and limitation-boundary work beyond the main result
Do not assume a writing-facing campaign means “analysis only”.
Do not invent a separate experiment system for those cases.
Interaction discipline
- Follow the shared interaction contract injected by the system prompt.
- For ordinary active work, prefer a concise progress update once work has crossed roughly 6 tool calls with a human-meaningful delta, and do not drift beyond roughly 12 tool calls or about 8 minutes without a user-visible update.
- Hard execution rule: every terminal command in this stage must go through
bash_exec; do not use any other terminal path for slice execution, smoke tests, Git, Python, package-manager, or file-inspection commands.
- Prefer
bash_exec for campaign slice commands so each run has a durable session id, quest-local log folder, and later read/list/kill control.
- Keep ordinary subtask completions concise. When an analysis campaign or a stage-significant campaign checkpoint is complete, upgrade to a richer
artifact.interact(kind='milestone', reply_mode='threaded', ...) report.
- That richer campaign milestone report should normally cover: which slices completed, the main takeaway, whether the claim got stronger or weaker, and the exact recommended next route.
- That richer milestone report is still normally non-blocking. If the post-campaign route is already clear, continue automatically after reporting instead of waiting for explicit acknowledgment.
- If the active communication surface is QQ and QQ milestone media is enabled in config, prefer at most one aggregated campaign summary PNG on a meaningful campaign milestone.
- That attachment should summarize the campaign as a whole; do not auto-send one image per slice.
- Treat connector-facing campaign PNGs as report charts, not draft paper figures.
- Preferred connector-chart palettes are Morandi-like and restrained:
sage-clay: #E7E1D6, #B7A99A, #7F8F84 for the default aggregated campaign summary
mist-stone: #F3EEE8, #D8D1C7, #8A9199 for conservative or uncertainty-heavy summaries
dust-rose: #F2E9E6, #D8C3BC, #B88C8C only as a secondary accent when an extra comparison is necessary
- Connector-facing campaign chart requirements:
- one campaign-level message, not a crowded slice dashboard
- low saturation and limited color count
- clear aggregation labels and direct comparison against the main run or baseline
- prefer one summary figure that communicates the boundary change honestly
- Preferred campaign summaries are:
- point-range or bar summaries for slice-to-slice endpoint comparisons
- line plots only when the x-axis is truly ordered and comparable across slices
- small multiples instead of one rainbow figure when slices answer different questions
- If a campaign view uses continuous color, keep it sequential for ordered magnitude and diverging only for signed deltas around a meaningful center.
- Avoid rainbow / jet-like maps and decorative heatmaps when a simpler comparison plot would communicate the result better.
- Keep the same muted palette semantics across the full campaign so the same color means the same role in every slice summary.
- If a campaign figure is milestone-facing, paper-facing, or otherwise durable, open
figure-polish/SKILL.md and complete its render-inspect-revise pass before treating the figure as final.
- If plotting in Python, reuse the fixed Morandi plotting starter from the system prompt and keep the same palette discipline across the whole campaign.
- If the runtime starts an auto-continue turn with no new user message, resume from the current campaign state and active requirements instead of replaying the previous user turn.
- Progress message templates are references only. Adapt to the actual context and vary wording so messages feel human, respectful, and non-robotic.
- If a threaded user reply arrives, interpret it relative to the latest campaign progress update before assuming the task changed completely.
Stage purpose
The analysis-campaign stage exists to test the strength, boundaries, and failure modes of a result.
It preserves the core old DeepScientist analysis-experimenter discipline:
- each analysis run should correspond to one clear question
- campaign runs should stay isolated and comparable
- negative results must remain visible
- campaign-level conclusions should be aggregated explicitly
The campaign should behave like a disciplined evidence program, not an unstructured pile of extra runs.
For campaign prioritization and writing-facing slice design, read references/campaign-design.md.
When the campaign is paper-facing and the mapping fields are not obvious, also read references/writing-facing-slice-examples.md.
Quick workflow
Treat this as the compressed campaign map. The authoritative slice protocol and aggregation rules remain in Workflow.
- Bind the campaign to the parent run or idea and, when writing-facing, to the selected outline.
- When the campaign is writing-facing, refresh
paper/paper_experiment_matrix.* before freezing the slice frontier.
- Before launching slices, create
PLAN.md and CHECKLIST.md.
- Use
PLAN.md as the durable charter and CHECKLIST.md as the living execution surface while launching, monitoring, recording, and aggregating slices.
- Run claim-critical slices first and smoke-test long slices before their real runs.
- Revise the plan and matrix if slice feasibility, ordering, comparators, or campaign interpretation changes materially, and record every slice durably, including honest non-success states.
- Close meaningful campaign milestones with a concise
1-2 sentence summary that says whether the claim gained stable support, partial support, contradiction, or unresolved ambiguity, what the matrix frontier now looks like, and what happens next.
Non-negotiable rules
- Every analysis run must be code-based and fully automatable.
- Do not introduce human evaluation or subjective assessment into a campaign.
- Do not bring in a new dataset unless the quest scope explicitly changed.
- Every analysis slice must have a specific research question and a falsifiable or at least decision-relevant expectation.
- If the campaign is supporting a paper or paper-like report, do not launch it until a selected outline exists.
- When a selected outline exists, every slice should map to a named
research_question and experimental_design from that outline.
- When the campaign is supporting a paper or paper-like report, do not launch or reorder the slice set without first reading
paper/paper_experiment_matrix.md when it exists.
- For writing-facing campaigns, every slice should correspond to a stable matrix row such as
exp_id, not just a free-form note.
- For writing-facing campaigns, every todo item must also carry
section_id, item_id, claim_links, and paper_role; otherwise the slice is not paper-ready.
- Do not aggregate campaign conclusions without per-run evidence.
- Do not bury null or contradictory findings.
Use when
- writing reveals evidence gaps
- a main result needs ablations
- robustness or sensitivity needs to be checked
- a failure mode needs explanation
- efficiency or environment variation matters to the claim
Do not use when
- the quest still lacks a credible main run or accepted baseline
- the next step is obviously another main experiment rather than follow-up evidence work
Preconditions and gate
Before launching a campaign, confirm:
- the reference main run or accepted idea line
- the claim or question being tested
- the comparison target
- the metric or observable of interest
- the list of specific analysis questions
- the current quest / user-provided assets that each planned slice will actually use
- whether each slice is executable with the current assets, tooling, and available credentials
- for paper-facing campaigns, the current paper experiment matrix frontier and which rows are actually feasible now
- if durable state exposes
active_baseline_metric_contract_json, read that JSON file before defining slice success criteria or comparison tables
- treat
active_baseline_metric_contract_json as the default baseline comparison contract unless a slice is explicitly testing a different evaluation contract
If the question list is fuzzy, sharpen it before running anything.
Treat quest files, attached user assets, checkpoints, configs, extracted texts, baselines, and existing code paths as the first-choice asset pool.
Do not design slices around hypothetical resources that the current system cannot actually access or run.
If a slice cannot be executed with the current system, redesign it around available assets or explicitly report that the task cannot currently be completed.
If infeasibility appears mid-run, attempt bounded recovery first; if still blocked, record the slice with a non-success status and explain why.
If ids, active refs, or current quest state are unclear after restart, call artifact.get_quest_state(detail='summary') and artifact.resolve_runtime_refs(...) before launching or recording slices.
If the exact quest brief / plan / status wording matters for campaign scope, call artifact.read_quest_documents(...).
If earlier user instructions materially affect campaign scope or ordering, call artifact.get_conversation_context(...) before changing the slice set.
For concrete paper-facing cases:
- if the slice is the only thing keeping a main-text section unsupported, make it
main_required / main_text
- if the slice is useful but non-blocking, make it
appendix
- if the slice is informative but not meant for the manuscript, keep it durable and mark it
reference_only with a reason
- after every completed paper-facing slice, verify the return path immediately:
- the matching outline
result_table row is updated
- the section notes are updated when the outline folder exists
paper/evidence_ledger.json reflects the new mapping
- the active paper line summary no longer treats that slice as missing
Do not leave a slice "completed" while the paper contract still looks stale.
Required plan and checklist
Before launching any real campaign slice, create a quest-visible PLAN.md and CHECKLIST.md.
- Use
references/campaign-plan-template.md as the canonical structure for PLAN.md.
- Use
references/campaign-checklist-template.md as the canonical structure for CHECKLIST.md.
PLAN.md is the durable campaign charter and should cover the claim under test, slice table, comparability boundary, available assets, required comparators, smoke and main-run strategy, monitoring and sleep rules, reporting expectations, and a revision log.
CHECKLIST.md is the living campaign execution list; update it during launch, asset preparation, slice execution, aggregation, and route changes.
- If slice ordering, feasibility, required baselines, campaign interpretation, or the writing-facing outline mapping changes materially, revise
PLAN.md before continuing.
- The later charter report, slice artifacts, and aggregate report remain required, but
PLAN.md and CHECKLIST.md should be the canonical campaign-control surface during execution.
Truth sources
Use:
- main experiment artifacts
- baseline artifacts
active_baseline_metric_contract_json when available
- recent decisions and milestone reports
- code and configs used in the accepted main line
- actual analysis outputs and logs
bash_exec session ids and managed shell logs for campaign runs
Do not summarize a campaign from impressions alone.
Required durable outputs
A campaign should usually leave behind:
- a campaign identifier
- a selected outline reference when the campaign is writing-facing
- a refreshed
paper/paper_experiment_matrix.md
- a refreshed
paper/paper_experiment_matrix.json
- one directory per analysis run
- any supplementary baseline reproduced for analysis under
baselines/local/<baseline_id>/ or attached under baselines/imported/<baseline_id>/
- one quest-level supplementary baseline inventory at
artifacts/baselines/analysis_inventory.json
- one run artifact per analysis slice
- one outline-bound todo manifest when the campaign is writing-facing
- an aggregated campaign report
- a decision about the next move
In the current runtime, represent that with existing artifact actions only:
- one
decision artifact with action='launch_analysis_campaign'
- one charter
report
- one
run artifact per slice
- optional
progress artifacts during execution
- one aggregated
report
- one closing
decision
Workflow
0. Launch the campaign durably
Before launching any slice, record the campaign start through artifacts:
- write a
decision artifact with:
action='launch_analysis_campaign'
campaign_id
parent_run_id or parent_idea_id
- why the campaign is needed now
- write a charter
report with the planned slice list
- update
plan.md if the campaign materially changes the quest path
Do not start a multi-slice campaign from chat-only intent.
Do not start it from chat-only intent plus vague notes either: write PLAN.md and CHECKLIST.md first, using references/campaign-plan-template.md and references/campaign-checklist-template.md as the default structures.
After the charter and launch decision are durably recorded, send one threaded artifact.interact(kind='milestone', ...) update naming:
- why the campaign exists now
- the claim-critical slices that will run first
- the first thing the user should expect from the campaign
- the first real checkpoint for the user
- if the active surface is QQ, keep that campaign-launch milestone text-first unless a single summary image is already genuinely useful
0.1 Bind the campaign to the selected outline when writing-facing
If the campaign exists to support a paper or paper-like report:
- do not proceed until one selected outline exists
- if no selected outline exists yet, route to
write or decision first so the outline can be created and selected durably
- before deciding the slice list, create or refresh
paper/paper_experiment_matrix.md when it is missing or stale
- treat that matrix as the upstream paper experiment contract, not
todo_items alone
- use the matrix to decide:
- which rows are
main_required
- which are
main_optional
- which are appendix-only
- which are optional or should be dropped
- do not start stable experiments-section drafting while currently feasible non-optional matrix rows remain unresolved
- call
artifact.create_analysis_campaign(...) with:
selected_outline_ref
research_questions
experimental_designs
todo_items
- ensure each todo item names at least:
exp_id
todo_id
slice_id
title
research_question
experimental_design
tier
paper_placement
completion_condition
For writing-facing campaigns, every slice should also carry paper-contract identity, not just free-form text:
section_id
item_id
claim_links
paper_role
Do not treat a completed analysis slice as paper-ready until those fields exist and the slice is mappable back into the selected outline or paper experiment matrix.
Use references/writing-facing-slice-examples.md when the correct field values are not obvious.
This keeps the analysis campaign aligned with the paper plan instead of becoming a free-floating batch of slices.
1. Define the campaign charter
State:
- campaign id
- parent run or parent idea
- main claim under test
- list of analysis questions
- what will be held fixed
- what may vary
The charter should also include:
- campaign type priority order
- expected slice count
- dependency structure between slices
- the matrix path and current execution frontier
- whether any slice requires isolated code changes or only reruns/config changes
- the top-level success condition for ending the campaign
- the top-level abandonment condition for stopping it early
Prefer to keep this charter in PLAN.md first and mirror the execution frontier in CHECKLIST.md.
For each analysis question, also state:
- why it matters to the main claim
- whether it exists mainly to support a core claim, validate a highlight, answer an efficiency or cost concern, or bound a limitation
- what result would strengthen the claim
- what result would weaken or complicate the claim
- whether the run is:
- ablation
- robustness
- sensitivity
- error analysis
- efficiency
- environment variation
If there are many possible slices, order them by decision value:
- most claim-critical ablation or contradiction check
- strongest robustness or sensitivity checks
- failure-mode explanation
- efficiency or secondary supporting analyses
Do not spend half the campaign budget on secondary slices before the claim-critical ones run.
When the parent line is still below solid evidence quality, use the campaign first to move it from minimum to solid before chasing broader polish.
2. Split into isolated analysis runs
Each analysis run should correspond to one need, such as:
- remove one component
- vary one hyperparameter family
- run additional seeds
- inspect one failure bucket
- test one environment variation
- measure one efficiency or cost dimension
- validate one highlight hypothesis
Avoid changing many factors at once unless the campaign is explicitly exploratory.
For each slice, define at minimum:
- research question
- hypothesis or expected pattern
- intervention
- controls or fixed conditions
- metric or observable
- stop condition
- evidence path expectations
required_baselines when the slice depends on an extra comparator that is not yet available in the quest
Recommended extra per-slice fields:
exp_id
slice_id
run_kind
slice_class, such as auxiliary, claim-carrying, or supporting
tier, such as main_required, main_optional, appendix, or optional
paper_placement
highlight_ids
required_baselines, where each item records at least baseline_id plus the reason, benchmark, and split when known
If a slice needs an extra comparator baseline:
- reproduce it under
baselines/local/<baseline_id>/ unless it is attached under baselines/imported/<baseline_id>/
- keep the usual durable baseline notes there, including
analysis_plan.md, setup.md, execution.md, and verification.md
- do not overwrite the canonical quest baseline gate just because an analysis slice needed a supplementary baseline
- after the comparator is ready, record it back through
record_analysis_slice(..., comparison_baselines=[...]) with its baseline_id, path, benchmark/split, and metrics summary
parent_run_id
- whether a code diff is required
- whether an isolated branch/worktree is required
- quantitative success criteria
- quantitative abandonment criteria
- contingency trigger for the next slice
Recommended run_kind naming in the current runtime:
analysis.ablation
analysis.robustness
analysis.sensitivity
analysis.error
analysis.efficiency
analysis.environment
Create the campaign with artifact.create_analysis_campaign(...) before starting any slice.
Even one extra experiment should still be represented as a one-slice campaign so Git and Canvas show a real child node.
Branch that campaign from the current workspace/result node rather than mutating the completed parent node in place.
That tool should receive the full slice list, and each returned slice worktree becomes the required execution location for that slice.
Only create the campaign after you have verified that the listed slices are actually executable with the current quest assets and runtime.
When the campaign is writing-facing, the same call should also carry selected_outline_ref, research_questions, experimental_designs, and todo_items.
If ids or refs are unclear, recover them first with artifact.resolve_runtime_refs(...), artifact.get_analysis_campaign(...), or artifact.list_paper_outlines(...) instead of guessing.
Treat campaign_id as system-owned, and treat slice_id / todo_id as agent-authored semantic ids.
Do not replace the normal campaign flow with repeated manual artifact.prepare_branch(...) calls.
After each slice finishes, call artifact.record_analysis_slice(...) immediately so the result is mirrored back to the parent branch and the next slice can be activated.
If a slice fails or becomes infeasible, still call artifact.record_analysis_slice(...) with an honest non-success status plus the real blocker and next recommendation; do not leave the campaign state ambiguous.
After every completed, excluded, or blocked writing-facing slice:
- reopen
paper/paper_experiment_matrix.md
- update the row status, feasibility, and result artifacts
- update whether the row now belongs in main text, appendix, or omission
- update the remaining execution frontier before choosing the next slice
Do not keep launching writing-facing slices from stale memory when the matrix has changed.
For slice recording, deviations and evidence_paths are optional context fields, not mandatory ceremony; include them only when they materially help explanation or auditability.
Each artifact.record_analysis_slice(...) call should also include an evaluation_summary with exactly these six fields:
takeaway
claim_update
baseline_relation
comparability
failure_mode
next_action
Use those six fields to keep each slice readable at a glance from Canvas, stage tabs, review, and rebuttal.
The longer prose still matters, but the six-field summary is the stable routing summary.
For writing-facing campaigns, prefer running claim-carrying slices before supporting slices unless an auxiliary check is required to make the main slice interpretable.
For slices that run longer than a quick smoke check:
- first run a bounded smoke test so the slice command, outputs, and metric path are validated cheaply
- once the smoke test passes, launch the real slice with
bash_exec(mode='detach', ...) and normally leave timeout_seconds unset for that long run
bash_exec(mode='read', id=...) returns the full rendered log when it is 2000 lines or fewer; for longer logs it returns the first 500 lines plus the last 1500 lines and a hint to inspect omitted sections with start and tail
- if you need a middle section that was omitted from that default preview, use
bash_exec(mode='read', id=..., start=..., tail=...)
- monitor them with
bash_exec(mode='list') and bash_exec(mode='read', id=..., tail_limit=..., order='desc')
- after the first read, prefer
bash_exec(mode='read', id=..., after_seq=last_seen_seq, tail_limit=..., order='asc') for incremental monitoring
- if ids become unclear, recover them through
bash_exec(mode='history')
- launch long slices with a structured
comment such as {stage, goal, action, expected_signal, next_check}
- use
silent_seconds, progress_age_seconds, signal_age_seconds, and watchdog_overdue from bash_exec(mode='list'|'read', ...) as the default stall checks
- use an explicit wait-and-check cadence of about
60s, 120s, 300s, 600s, 1800s, then every 1800s while still running
- if needed, use an explicit bounded wait such as
bash_exec(command='sleep 60', mode='await', timeout_seconds=70) or bash_exec(mode='await', id=..., timeout_seconds=...) between checks
- canonical sleep choice:
- if you only need wall-clock waiting between checks, use
bash_exec(command='sleep N', mode='await', timeout_seconds=N+buffer, ...)
- keep a real buffer on that sleep timeout; do not set
timeout_seconds exactly equal to N
- if you are waiting on an already running managed session, prefer
bash_exec(mode='await', id=..., timeout_seconds=...) instead of starting a new sleep command
- after the first meaningful signal and then at real checkpoints (e.g., completion, blocker, recovery, or a materially changed evidence frontier), send
artifact.interact(kind='progress', ...) so the user sees the newest real state
- after each completed sleep / await monitoring cycle for an active slice, inspect state first; only send another
artifact.interact(kind='progress', ...) update if the user-visible state materially changed
- include the estimated next reply time or next check time in those monitoring updates
- stop them with
bash_exec(mode='kill', id=..., wait=true, timeout_seconds=...) if the slice is invalid, wedged, or superseded; add force=true when immediate termination is required
- when you control the slice code, prefer a throttled
tqdm progress reporter and, when feasible, pair it with concise __DS_PROGRESS__ lines carrying phase and ETA
- do not mark a slice complete until the managed log and outputs both confirm completion
3. Keep comparability
Comparability rules:
- keep the same evaluation contract unless the variation is the point
- when
active_baseline_metric_contract_json exists, keep slice comparisons aligned with it unless the slice explicitly records why it differs
- state exactly what changed
- state exactly what stayed fixed
- keep naming and output paths clean so multiple runs can coexist
For code-modifying slices, the default durable layout should stay interpretable:
- working surface:
.ds/worktrees/<slice_id>/ when isolated worktrees are used
- experiment surface:
experiments/analysis/<campaign_id>/<slice_id>/
- artifact surface:
artifacts/runs/<artifact_id>.json
artifacts/reports/<artifact_id>.json
If the variation itself changes the evaluation setup, record that explicitly and do not present the run as a direct apples-to-apples comparison.
4. Record each analysis slice
Before a long slice starts, emit a progress artifact or artifact.interact(kind='progress', ...) update so the quest shows that the slice is active.
For each run, record:
- analysis question
- intervention
- metric or qualitative evidence
- whether the result strengthens, weakens, or complicates the claim
- paths to the evidence
Preferred per-slice summary shape:
- question
- implementation change
- main metric delta
- interpretation
- caveats
Each completed slice should also leave a run artifact containing at least:
campaign_id
slice_id
run_kind
parent_run_id
analysis_question
fixed_conditions
changed_factors
metrics_summary
metric_deltas
success_criteria
abandonment_criteria
verdict
reason
paths
If a slice fails before producing evidence, still record it as a failed or partial run artifact rather than silently skipping it.
When a slice materially changes the recommended route or weakens the main claim, do not wait until the final synthesis to mention it.
Send a threaded artifact.interact(kind='milestone', ...) update at that point with the new boundary or risk.
5. Aggregate the campaign
The campaign report should explain:
- which findings are stable
- which findings are fragile
- what changed the interpretation of the main result
- which open questions still remain
Campaign reporting rules:
- focus on the highest-impact findings first
- results matter more than process narration
- if using tables, show only the most decision-relevant rows
- separate:
- stable support
- partial support
- contradiction
- unresolved ambiguity
When there are many slices, summarize the top 3-5 most important ones first, then point to the full evidence paths.
The aggregated report should also answer:
- should the main claim be strengthened, weakened, narrowed, or abandoned?
- which slice changed the interpretation most?
- which slice is still worth rerunning, and why?
- which planned slices were intentionally skipped because earlier results made them low value?
When the aggregated campaign report is complete, send a richer threaded artifact.interact(kind='milestone', ...) update.
Lead that milestone with a concise 1-2 sentence campaign outcome summary before expanding into slice-level detail.
If QQ milestone media is enabled and the aggregated report materially changes the claim boundary, you may attach one campaign summary PNG to that closing milestone update.
That update should explicitly classify the campaign outcome in the same language as the report:
- stable support
- partial support
- contradiction
- unresolved ambiguity
6. Route the next step
A campaign should end with an explicit next move:
- continue the campaign
- return to
experiment
- move to
write
- stop or reset the current line
Record the post-campaign route as a decision artifact.
When helpful, include a reflection block with:
what_worked
what_failed
learned_constraints
and a next_direction block that states:
- objective
- key steps
- success criteria
- abandonment criteria
This makes the next stage executable without guesswork.
Analysis-quality rules
Good campaign behavior:
- one clear question per run
- one-factor-at-a-time changes when possible
- clear comparison against the accepted reference line
- visibility of null and negative findings
- a logically ordered suite rather than a random batch
Strong campaign ordering usually looks like:
- most claim-critical ablation or comparison
- strongest robustness or sensitivity checks
- failure-mode or error analysis
- efficiency or secondary analysis
The exact order can vary, but the most claim-relevant evidence should appear first.
Weak campaign behavior:
- hidden scope expansion
- many untracked simultaneous changes
- campaign summary without per-run evidence
- ignoring contradictory analysis results
- reporting every minor slice with equal weight instead of prioritizing the important ones
Memory rules
Stage-start requirement:
- begin every analysis campaign pass with
memory.list_recent(scope='quest', limit=5)
- then run at least one analysis-relevant
memory.search(...) before launching or resuming slices
- if several campaigns, parent runs, or idea lines exist, narrow retrieval to the current
campaign_id, parent_run_id, idea_id, or branch instead of mixing unrelated slice memory
Write to memory only when the campaign yields reusable lessons, such as:
- robust failure patterns
- evaluation caveats
- reproducible sensitivity findings
Stage-end requirement:
- if the campaign produced a durable cross-slice lesson, failure pattern, or comparability caveat, write at least one
memory.write(...) before leaving the stage
The campaign’s main record belongs in run artifacts and the aggregated report.
When synthesizing the campaign, read the per-slice evaluation_summary fields first, then expand into longer evidence only where the short summaries are still ambiguous.
Artifact rules
Typical artifact sequence:
- decision artifact to launch the campaign
- report artifact for the charter
- progress artifacts during long campaigns
- run artifacts per analysis slice
- report artifact for the aggregated campaign summary
- decision artifact for the next anchor
Failure and blocked handling
Record blocked or failed campaign states explicitly, such as:
- missing parent run
- analysis question under-specified
- campaign run failed before evidence was produced
- metrics not comparable
- campaign conclusion still ambiguous
A blocked campaign should still name the next best action.
Exit criteria
Exit the analysis-campaign stage once one of the following is durably true:
- the campaign produced enough evidence for writing or decision-making
- the campaign exposed a problem that requires returning to
experiment or idea
- the campaign is blocked and the blocker is durably recorded
1---
2name: ds-analysis-campaign
3description: Use when a quest needs one or more follow-up runs such as ablations, robustness checks, error analysis, or failure analysis after a main experiment.
4skill_role: stage
5license: MIT
6metadata:
7 author: ResearAI/DeepScientist
8 version: "1.0.0"
9---
10
11# Analysis Campaign
12
13Use this skill when one or more follow-up runs are needed and the quest needs a coordinated evidence campaign.
14
15This is the shared DeepScientist protocol for supplementary experiments after a durable result.
16Use the same route for:
17
18- ordinary ablations / robustness / sensitivity work
19- review-driven evidence gaps
20- rebuttal-driven extra experiments
21- writing-driven evidence gaps
22
23For paper-facing work, treat “analysis campaign” broadly:
24
25- not only post-hoc interpretation
26- also ablations, sensitivity checks, robustness checks, efficiency or cost checks, highlight-validation runs, and limitation-boundary work beyond the main result
27
28Do not assume a writing-facing campaign means “analysis only”.
29
30Do not invent a separate experiment system for those cases.
31
32## Interaction discipline
33
34- Follow the shared interaction contract injected by the system prompt.
35- For ordinary active work, prefer a concise progress update once work has crossed roughly 6 tool calls with a human-meaningful delta, and do not drift beyond roughly 12 tool calls or about 8 minutes without a user-visible update.
36- Hard execution rule: every terminal command in this stage must go through `bash_exec`; do not use any other terminal path for slice execution, smoke tests, Git, Python, package-manager, or file-inspection commands.
37- Prefer `bash_exec` for campaign slice commands so each run has a durable session id, quest-local log folder, and later `read/list/kill` control.
38- Keep ordinary subtask completions concise. When an analysis campaign or a stage-significant campaign checkpoint is complete, upgrade to a richer `artifact.interact(kind='milestone', reply_mode='threaded', ...)` report.
39- That richer campaign milestone report should normally cover: which slices completed, the main takeaway, whether the claim got stronger or weaker, and the exact recommended next route.
40- That richer milestone report is still normally non-blocking. If the post-campaign route is already clear, continue automatically after reporting instead of waiting for explicit acknowledgment.
41- If the active communication surface is QQ and QQ milestone media is enabled in config, prefer at most one aggregated campaign summary PNG on a meaningful campaign milestone.
42- That attachment should summarize the campaign as a whole; do not auto-send one image per slice.
43- Treat connector-facing campaign PNGs as report charts, not draft paper figures.
44- Preferred connector-chart palettes are Morandi-like and restrained:
45 - `sage-clay`: `#E7E1D6`, `#B7A99A`, `#7F8F84` for the default aggregated campaign summary
46 - `mist-stone`: `#F3EEE8`, `#D8D1C7`, `#8A9199` for conservative or uncertainty-heavy summaries
47 - `dust-rose`: `#F2E9E6`, `#D8C3BC`, `#B88C8C` only as a secondary accent when an extra comparison is necessary
48- Connector-facing campaign chart requirements:
49 - one campaign-level message, not a crowded slice dashboard
50 - low saturation and limited color count
51 - clear aggregation labels and direct comparison against the main run or baseline
52 - prefer one summary figure that communicates the boundary change honestly
53- Preferred campaign summaries are:
54 - point-range or bar summaries for slice-to-slice endpoint comparisons
55 - line plots only when the x-axis is truly ordered and comparable across slices
56 - small multiples instead of one rainbow figure when slices answer different questions
57- If a campaign view uses continuous color, keep it sequential for ordered magnitude and diverging only for signed deltas around a meaningful center.
58- Avoid rainbow / jet-like maps and decorative heatmaps when a simpler comparison plot would communicate the result better.
59- Keep the same muted palette semantics across the full campaign so the same color means the same role in every slice summary.
60- If a campaign figure is milestone-facing, paper-facing, or otherwise durable, open `figure-polish/SKILL.md` and complete its render-inspect-revise pass before treating the figure as final.
61- If plotting in Python, reuse the fixed Morandi plotting starter from the system prompt and keep the same palette discipline across the whole campaign.
62- If the runtime starts an auto-continue turn with no new user message, resume from the current campaign state and active requirements instead of replaying the previous user turn.
63- Progress message templates are references only. Adapt to the actual context and vary wording so messages feel human, respectful, and non-robotic.
64- If a threaded user reply arrives, interpret it relative to the latest campaign progress update before assuming the task changed completely.
65
66## Stage purpose
67
68The analysis-campaign stage exists to test the strength, boundaries, and failure modes of a result.
69It preserves the core old DeepScientist analysis-experimenter discipline:
70
71- each analysis run should correspond to one clear question
72- campaign runs should stay isolated and comparable
73- negative results must remain visible
74- campaign-level conclusions should be aggregated explicitly
75
76The campaign should behave like a disciplined evidence program, not an unstructured pile of extra runs.
77
78For campaign prioritization and writing-facing slice design, read `references/campaign-design.md`.
79When the campaign is paper-facing and the mapping fields are not obvious, also read `references/writing-facing-slice-examples.md`.
80
81## Quick workflow
82
83Treat this as the compressed campaign map. The authoritative slice protocol and aggregation rules remain in `Workflow`.
84
851. Bind the campaign to the parent run or idea and, when writing-facing, to the selected outline.
862. When the campaign is writing-facing, refresh `paper/paper_experiment_matrix.*` before freezing the slice frontier.
873. Before launching slices, create `PLAN.md` and `CHECKLIST.md`.
884. Use `PLAN.md` as the durable charter and `CHECKLIST.md` as the living execution surface while launching, monitoring, recording, and aggregating slices.
895. Run claim-critical slices first and smoke-test long slices before their real runs.
906. Revise the plan and matrix if slice feasibility, ordering, comparators, or campaign interpretation changes materially, and record every slice durably, including honest non-success states.
917. Close meaningful campaign milestones with a concise `1-2` sentence summary that says whether the claim gained stable support, partial support, contradiction, or unresolved ambiguity, what the matrix frontier now looks like, and what happens next.
92
93## Non-negotiable rules
94
95- Every analysis run must be code-based and fully automatable.
96- Do not introduce human evaluation or subjective assessment into a campaign.
97- Do not bring in a new dataset unless the quest scope explicitly changed.
98- Every analysis slice must have a specific research question and a falsifiable or at least decision-relevant expectation.
99- If the campaign is supporting a paper or paper-like report, do not launch it until a selected outline exists.
100- When a selected outline exists, every slice should map to a named `research_question` and `experimental_design` from that outline.
101- When the campaign is supporting a paper or paper-like report, do not launch or reorder the slice set without first reading `paper/paper_experiment_matrix.md` when it exists.
102- For writing-facing campaigns, every slice should correspond to a stable matrix row such as `exp_id`, not just a free-form note.
103- For writing-facing campaigns, every todo item must also carry `section_id`, `item_id`, `claim_links`, and `paper_role`; otherwise the slice is not paper-ready.
104- Do not aggregate campaign conclusions without per-run evidence.
105- Do not bury null or contradictory findings.
106
107## Use when
108
109- writing reveals evidence gaps
110- a main result needs ablations
111- robustness or sensitivity needs to be checked
112- a failure mode needs explanation
113- efficiency or environment variation matters to the claim
114
115## Do not use when
116
117- the quest still lacks a credible main run or accepted baseline
118- the next step is obviously another main experiment rather than follow-up evidence work
119
120## Preconditions and gate
121
122Before launching a campaign, confirm:
123
124- the reference main run or accepted idea line
125- the claim or question being tested
126- the comparison target
127- the metric or observable of interest
128- the list of specific analysis questions
129- the current quest / user-provided assets that each planned slice will actually use
130- whether each slice is executable with the current assets, tooling, and available credentials
131- for paper-facing campaigns, the current paper experiment matrix frontier and which rows are actually feasible now
132- if durable state exposes `active_baseline_metric_contract_json`, read that JSON file before defining slice success criteria or comparison tables
133- treat `active_baseline_metric_contract_json` as the default baseline comparison contract unless a slice is explicitly testing a different evaluation contract
134
135If the question list is fuzzy, sharpen it before running anything.
136Treat quest files, attached user assets, checkpoints, configs, extracted texts, baselines, and existing code paths as the first-choice asset pool.
137Do not design slices around hypothetical resources that the current system cannot actually access or run.
138If a slice cannot be executed with the current system, redesign it around available assets or explicitly report that the task cannot currently be completed.
139If infeasibility appears mid-run, attempt bounded recovery first; if still blocked, record the slice with a non-success status and explain why.
140If ids, active refs, or current quest state are unclear after restart, call `artifact.get_quest_state(detail='summary')` and `artifact.resolve_runtime_refs(...)` before launching or recording slices.
141If the exact quest brief / plan / status wording matters for campaign scope, call `artifact.read_quest_documents(...)`.
142If earlier user instructions materially affect campaign scope or ordering, call `artifact.get_conversation_context(...)` before changing the slice set.
143
144For concrete paper-facing cases:
145
146- if the slice is the only thing keeping a main-text section unsupported, make it `main_required` / `main_text`
147- if the slice is useful but non-blocking, make it `appendix`
148- if the slice is informative but not meant for the manuscript, keep it durable and mark it `reference_only` with a reason
149- after every completed paper-facing slice, verify the return path immediately:
150 - the matching outline `result_table` row is updated
151 - the section notes are updated when the outline folder exists
152 - `paper/evidence_ledger.json` reflects the new mapping
153 - the active paper line summary no longer treats that slice as missing
154
155Do not leave a slice "completed" while the paper contract still looks stale.
156
157## Required plan and checklist
158
159Before launching any real campaign slice, create a quest-visible `PLAN.md` and `CHECKLIST.md`.
160
161- Use `references/campaign-plan-template.md` as the canonical structure for `PLAN.md`.
162- Use `references/campaign-checklist-template.md` as the canonical structure for `CHECKLIST.md`.
163- `PLAN.md` is the durable campaign charter and should cover the claim under test, slice table, comparability boundary, available assets, required comparators, smoke and main-run strategy, monitoring and sleep rules, reporting expectations, and a revision log.
164- `CHECKLIST.md` is the living campaign execution list; update it during launch, asset preparation, slice execution, aggregation, and route changes.
165- If slice ordering, feasibility, required baselines, campaign interpretation, or the writing-facing outline mapping changes materially, revise `PLAN.md` before continuing.
166- The later charter report, slice artifacts, and aggregate report remain required, but `PLAN.md` and `CHECKLIST.md` should be the canonical campaign-control surface during execution.
167
168## Truth sources
169
170Use:
171
172- main experiment artifacts
173- baseline artifacts
174- `active_baseline_metric_contract_json` when available
175- recent decisions and milestone reports
176- code and configs used in the accepted main line
177- actual analysis outputs and logs
178- `bash_exec` session ids and managed shell logs for campaign runs
179
180Do not summarize a campaign from impressions alone.
181
182## Required durable outputs
183
184A campaign should usually leave behind:
185
186- a campaign identifier
187- a selected outline reference when the campaign is writing-facing
188- a refreshed `paper/paper_experiment_matrix.md`
189- a refreshed `paper/paper_experiment_matrix.json`
190- one directory per analysis run
191- any supplementary baseline reproduced for analysis under `baselines/local/<baseline_id>/` or attached under `baselines/imported/<baseline_id>/`
192- one quest-level supplementary baseline inventory at `artifacts/baselines/analysis_inventory.json`
193- one run artifact per analysis slice
194- one outline-bound todo manifest when the campaign is writing-facing
195- an aggregated campaign report
196- a decision about the next move
197
198In the current runtime, represent that with existing artifact actions only:
199
200- one `decision` artifact with `action='launch_analysis_campaign'`
201- one charter `report`
202- one `run` artifact per slice
203- optional `progress` artifacts during execution
204- one aggregated `report`
205- one closing `decision`
206
207## Workflow
208
209### 0. Launch the campaign durably
210
211Before launching any slice, record the campaign start through artifacts:
212
2131. write a `decision` artifact with:
214 - `action='launch_analysis_campaign'`
215 - `campaign_id`
216 - `parent_run_id` or `parent_idea_id`
217 - why the campaign is needed now
2182. write a charter `report` with the planned slice list
2193. update `plan.md` if the campaign materially changes the quest path
220
221Do not start a multi-slice campaign from chat-only intent.
222Do not start it from chat-only intent plus vague notes either: write `PLAN.md` and `CHECKLIST.md` first, using `references/campaign-plan-template.md` and `references/campaign-checklist-template.md` as the default structures.
223
224After the charter and launch decision are durably recorded, send one threaded `artifact.interact(kind='milestone', ...)` update naming:
225
226- why the campaign exists now
227- the claim-critical slices that will run first
228- the first thing the user should expect from the campaign
229- the first real checkpoint for the user
230- if the active surface is QQ, keep that campaign-launch milestone text-first unless a single summary image is already genuinely useful
231
232### 0.1 Bind the campaign to the selected outline when writing-facing
233
234If the campaign exists to support a paper or paper-like report:
235
236- do not proceed until one selected outline exists
237- if no selected outline exists yet, route to `write` or `decision` first so the outline can be created and selected durably
238- before deciding the slice list, create or refresh `paper/paper_experiment_matrix.md` when it is missing or stale
239- treat that matrix as the upstream paper experiment contract, not `todo_items` alone
240- use the matrix to decide:
241 - which rows are `main_required`
242 - which are `main_optional`
243 - which are appendix-only
244 - which are optional or should be dropped
245- do not start stable experiments-section drafting while currently feasible non-optional matrix rows remain unresolved
246- call `artifact.create_analysis_campaign(...)` with:
247 - `selected_outline_ref`
248 - `research_questions`
249 - `experimental_designs`
250 - `todo_items`
251- ensure each todo item names at least:
252 - `exp_id`
253 - `todo_id`
254 - `slice_id`
255 - `title`
256 - `research_question`
257 - `experimental_design`
258 - `tier`
259 - `paper_placement`
260 - `completion_condition`
261
262For writing-facing campaigns, every slice should also carry paper-contract identity, not just free-form text:
263
264- `section_id`
265- `item_id`
266- `claim_links`
267- `paper_role`
268
269Do not treat a completed analysis slice as paper-ready until those fields exist and the slice is mappable back into the selected outline or paper experiment matrix.
270Use `references/writing-facing-slice-examples.md` when the correct field values are not obvious.
271
272This keeps the analysis campaign aligned with the paper plan instead of becoming a free-floating batch of slices.
273
274### 1. Define the campaign charter
275
276State:
277
278- campaign id
279- parent run or parent idea
280- main claim under test
281- list of analysis questions
282- what will be held fixed
283- what may vary
284
285The charter should also include:
286
287- campaign type priority order
288- expected slice count
289- dependency structure between slices
290- the matrix path and current execution frontier
291- whether any slice requires isolated code changes or only reruns/config changes
292- the top-level success condition for ending the campaign
293- the top-level abandonment condition for stopping it early
294
295Prefer to keep this charter in `PLAN.md` first and mirror the execution frontier in `CHECKLIST.md`.
296
297For each analysis question, also state:
298
299- why it matters to the main claim
300- whether it exists mainly to support a core claim, validate a highlight, answer an efficiency or cost concern, or bound a limitation
301- what result would strengthen the claim
302- what result would weaken or complicate the claim
303- whether the run is:
304 - ablation
305 - robustness
306 - sensitivity
307 - error analysis
308 - efficiency
309 - environment variation
310
311If there are many possible slices, order them by decision value:
312
3131. most claim-critical ablation or contradiction check
3142. strongest robustness or sensitivity checks
3153. failure-mode explanation
3164. efficiency or secondary supporting analyses
317
318Do not spend half the campaign budget on secondary slices before the claim-critical ones run.
319When the parent line is still below `solid` evidence quality, use the campaign first to move it from `minimum` to `solid` before chasing broader polish.
320
321### 2. Split into isolated analysis runs
322
323Each analysis run should correspond to one need, such as:
324
325- remove one component
326- vary one hyperparameter family
327- run additional seeds
328- inspect one failure bucket
329- test one environment variation
330- measure one efficiency or cost dimension
331- validate one highlight hypothesis
332
333Avoid changing many factors at once unless the campaign is explicitly exploratory.
334
335For each slice, define at minimum:
336
337- research question
338- hypothesis or expected pattern
339- intervention
340- controls or fixed conditions
341- metric or observable
342- stop condition
343- evidence path expectations
344- `required_baselines` when the slice depends on an extra comparator that is not yet available in the quest
345
346Recommended extra per-slice fields:
347
348- `exp_id`
349- `slice_id`
350- `run_kind`
351- `slice_class`, such as `auxiliary`, `claim-carrying`, or `supporting`
352- `tier`, such as `main_required`, `main_optional`, `appendix`, or `optional`
353- `paper_placement`
354- `highlight_ids`
355- `required_baselines`, where each item records at least `baseline_id` plus the reason, benchmark, and split when known
356
357If a slice needs an extra comparator baseline:
358
359- reproduce it under `baselines/local/<baseline_id>/` unless it is attached under `baselines/imported/<baseline_id>/`
360- keep the usual durable baseline notes there, including `analysis_plan.md`, `setup.md`, `execution.md`, and `verification.md`
361- do not overwrite the canonical quest baseline gate just because an analysis slice needed a supplementary baseline
362- after the comparator is ready, record it back through `record_analysis_slice(..., comparison_baselines=[...])` with its `baseline_id`, path, benchmark/split, and metrics summary
363- `parent_run_id`
364- whether a code diff is required
365- whether an isolated branch/worktree is required
366- quantitative success criteria
367- quantitative abandonment criteria
368- contingency trigger for the next slice
369
370Recommended `run_kind` naming in the current runtime:
371
372- `analysis.ablation`
373- `analysis.robustness`
374- `analysis.sensitivity`
375- `analysis.error`
376- `analysis.efficiency`
377- `analysis.environment`
378
379Create the campaign with `artifact.create_analysis_campaign(...)` before starting any slice.
380Even one extra experiment should still be represented as a one-slice campaign so Git and Canvas show a real child node.
381Branch that campaign from the current workspace/result node rather than mutating the completed parent node in place.
382That tool should receive the full slice list, and each returned slice worktree becomes the required execution location for that slice.
383Only create the campaign after you have verified that the listed slices are actually executable with the current quest assets and runtime.
384When the campaign is writing-facing, the same call should also carry `selected_outline_ref`, `research_questions`, `experimental_designs`, and `todo_items`.
385If ids or refs are unclear, recover them first with `artifact.resolve_runtime_refs(...)`, `artifact.get_analysis_campaign(...)`, or `artifact.list_paper_outlines(...)` instead of guessing.
386Treat `campaign_id` as system-owned, and treat `slice_id` / `todo_id` as agent-authored semantic ids.
387Do not replace the normal campaign flow with repeated manual `artifact.prepare_branch(...)` calls.
388After each slice finishes, call `artifact.record_analysis_slice(...)` immediately so the result is mirrored back to the parent branch and the next slice can be activated.
389If a slice fails or becomes infeasible, still call `artifact.record_analysis_slice(...)` with an honest non-success status plus the real blocker and next recommendation; do not leave the campaign state ambiguous.
390After every completed, excluded, or blocked writing-facing slice:
391
392- reopen `paper/paper_experiment_matrix.md`
393- update the row status, feasibility, and result artifacts
394- update whether the row now belongs in main text, appendix, or omission
395- update the remaining execution frontier before choosing the next slice
396
397Do not keep launching writing-facing slices from stale memory when the matrix has changed.
398For slice recording, `deviations` and `evidence_paths` are optional context fields, not mandatory ceremony; include them only when they materially help explanation or auditability.
399Each `artifact.record_analysis_slice(...)` call should also include an `evaluation_summary` with exactly these six fields:
400
401- `takeaway`
402- `claim_update`
403- `baseline_relation`
404- `comparability`
405- `failure_mode`
406- `next_action`
407
408Use those six fields to keep each slice readable at a glance from Canvas, stage tabs, review, and rebuttal.
409The longer prose still matters, but the six-field summary is the stable routing summary.
410
411For writing-facing campaigns, prefer running `claim-carrying` slices before `supporting` slices unless an auxiliary check is required to make the main slice interpretable.
412
413For slices that run longer than a quick smoke check:
414
415- first run a bounded smoke test so the slice command, outputs, and metric path are validated cheaply
416- once the smoke test passes, launch the real slice with `bash_exec(mode='detach', ...)` and normally leave `timeout_seconds` unset for that long run
417- `bash_exec(mode='read', id=...)` returns the full rendered log when it is 2000 lines or fewer; for longer logs it returns the first 500 lines plus the last 1500 lines and a hint to inspect omitted sections with `start` and `tail`
418- if you need a middle section that was omitted from that default preview, use `bash_exec(mode='read', id=..., start=..., tail=...)`
419- monitor them with `bash_exec(mode='list')` and `bash_exec(mode='read', id=..., tail_limit=..., order='desc')`
420- after the first read, prefer `bash_exec(mode='read', id=..., after_seq=last_seen_seq, tail_limit=..., order='asc')` for incremental monitoring
421- if ids become unclear, recover them through `bash_exec(mode='history')`
422- launch long slices with a structured `comment` such as `{stage, goal, action, expected_signal, next_check}`
423- use `silent_seconds`, `progress_age_seconds`, `signal_age_seconds`, and `watchdog_overdue` from `bash_exec(mode='list'|'read', ...)` as the default stall checks
424- use an explicit wait-and-check cadence of about `60s`, `120s`, `300s`, `600s`, `1800s`, then every `1800s` while still running
425- if needed, use an explicit bounded wait such as `bash_exec(command='sleep 60', mode='await', timeout_seconds=70)` or `bash_exec(mode='await', id=..., timeout_seconds=...)` between checks
426- canonical sleep choice:
427 - if you only need wall-clock waiting between checks, use `bash_exec(command='sleep N', mode='await', timeout_seconds=N+buffer, ...)`
428 - keep a real buffer on that sleep timeout; do not set `timeout_seconds` exactly equal to `N`
429 - if you are waiting on an already running managed session, prefer `bash_exec(mode='await', id=..., timeout_seconds=...)` instead of starting a new sleep command
430- after the first meaningful signal and then at real checkpoints (e.g., completion, blocker, recovery, or a materially changed evidence frontier), send `artifact.interact(kind='progress', ...)` so the user sees the newest real state
431- after each completed sleep / await monitoring cycle for an active slice, inspect state first; only send another `artifact.interact(kind='progress', ...)` update if the user-visible state materially changed
432- include the estimated next reply time or next check time in those monitoring updates
433- stop them with `bash_exec(mode='kill', id=..., wait=true, timeout_seconds=...)` if the slice is invalid, wedged, or superseded; add `force=true` when immediate termination is required
434- when you control the slice code, prefer a throttled `tqdm` progress reporter and, when feasible, pair it with concise `__DS_PROGRESS__` lines carrying phase and ETA
435- do not mark a slice complete until the managed log and outputs both confirm completion
436
437### 3. Keep comparability
438
439Comparability rules:
440
441- keep the same evaluation contract unless the variation is the point
442- when `active_baseline_metric_contract_json` exists, keep slice comparisons aligned with it unless the slice explicitly records why it differs
443- state exactly what changed
444- state exactly what stayed fixed
445- keep naming and output paths clean so multiple runs can coexist
446
447For code-modifying slices, the default durable layout should stay interpretable:
448
449- working surface:
450 - `.ds/worktrees/<slice_id>/` when isolated worktrees are used
451- experiment surface:
452 - `experiments/analysis/<campaign_id>/<slice_id>/`
453- artifact surface:
454 - `artifacts/runs/<artifact_id>.json`
455 - `artifacts/reports/<artifact_id>.json`
456
457If the variation itself changes the evaluation setup, record that explicitly and do not present the run as a direct apples-to-apples comparison.
458
459### 4. Record each analysis slice
460
461Before a long slice starts, emit a `progress` artifact or `artifact.interact(kind='progress', ...)` update so the quest shows that the slice is active.
462
463For each run, record:
464
465- analysis question
466- intervention
467- metric or qualitative evidence
468- whether the result strengthens, weakens, or complicates the claim
469- paths to the evidence
470
471Preferred per-slice summary shape:
472
473- question
474- implementation change
475- main metric delta
476- interpretation
477- caveats
478
479Each completed slice should also leave a `run` artifact containing at least:
480
481- `campaign_id`
482- `slice_id`
483- `run_kind`
484- `parent_run_id`
485- `analysis_question`
486- `fixed_conditions`
487- `changed_factors`
488- `metrics_summary`
489- `metric_deltas`
490- `success_criteria`
491- `abandonment_criteria`
492- `verdict`
493- `reason`
494- `paths`
495
496If a slice fails before producing evidence, still record it as a failed or partial `run` artifact rather than silently skipping it.
497
498When a slice materially changes the recommended route or weakens the main claim, do not wait until the final synthesis to mention it.
499Send a threaded `artifact.interact(kind='milestone', ...)` update at that point with the new boundary or risk.
500
501### 5. Aggregate the campaign
502
503The campaign report should explain:
504
505- which findings are stable
506- which findings are fragile
507- what changed the interpretation of the main result
508- which open questions still remain
509
510Campaign reporting rules:
511
512- focus on the highest-impact findings first
513- results matter more than process narration
514- if using tables, show only the most decision-relevant rows
515- separate:
516 - stable support
517 - partial support
518 - contradiction
519 - unresolved ambiguity
520
521When there are many slices, summarize the top `3-5` most important ones first, then point to the full evidence paths.
522
523The aggregated report should also answer:
524
525- should the main claim be strengthened, weakened, narrowed, or abandoned?
526- which slice changed the interpretation most?
527- which slice is still worth rerunning, and why?
528- which planned slices were intentionally skipped because earlier results made them low value?
529
530When the aggregated campaign report is complete, send a richer threaded `artifact.interact(kind='milestone', ...)` update.
531Lead that milestone with a concise `1-2` sentence campaign outcome summary before expanding into slice-level detail.
532
533If QQ milestone media is enabled and the aggregated report materially changes the claim boundary, you may attach one campaign summary PNG to that closing milestone update.
534That update should explicitly classify the campaign outcome in the same language as the report:
535
536- stable support
537- partial support
538- contradiction
539- unresolved ambiguity
540
541### 6. Route the next step
542
543A campaign should end with an explicit next move:
544
545- continue the campaign
546- return to `experiment`
547- move to `write`
548- stop or reset the current line
549
550Record the post-campaign route as a `decision` artifact.
551When helpful, include a reflection block with:
552
553- `what_worked`
554- `what_failed`
555- `learned_constraints`
556
557and a `next_direction` block that states:
558
559- objective
560- key steps
561- success criteria
562- abandonment criteria
563
564This makes the next stage executable without guesswork.
565
566## Analysis-quality rules
567
568Good campaign behavior:
569
570- one clear question per run
571- one-factor-at-a-time changes when possible
572- clear comparison against the accepted reference line
573- visibility of null and negative findings
574- a logically ordered suite rather than a random batch
575
576Strong campaign ordering usually looks like:
577
5781. most claim-critical ablation or comparison
5792. strongest robustness or sensitivity checks
5803. failure-mode or error analysis
5814. efficiency or secondary analysis
582
583The exact order can vary, but the most claim-relevant evidence should appear first.
584
585Weak campaign behavior:
586
587- hidden scope expansion
588- many untracked simultaneous changes
589- campaign summary without per-run evidence
590- ignoring contradictory analysis results
591- reporting every minor slice with equal weight instead of prioritizing the important ones
592
593## Memory rules
594
595Stage-start requirement:
596
597- begin every analysis campaign pass with `memory.list_recent(scope='quest', limit=5)`
598- then run at least one analysis-relevant `memory.search(...)` before launching or resuming slices
599- if several campaigns, parent runs, or idea lines exist, narrow retrieval to the current `campaign_id`, `parent_run_id`, `idea_id`, or `branch` instead of mixing unrelated slice memory
600
601Write to memory only when the campaign yields reusable lessons, such as:
602
603- robust failure patterns
604- evaluation caveats
605- reproducible sensitivity findings
606
607Stage-end requirement:
608
609- if the campaign produced a durable cross-slice lesson, failure pattern, or comparability caveat, write at least one `memory.write(...)` before leaving the stage
610
611The campaign’s main record belongs in run artifacts and the aggregated report.
612When synthesizing the campaign, read the per-slice `evaluation_summary` fields first, then expand into longer evidence only where the short summaries are still ambiguous.
613
614## Artifact rules
615
616Typical artifact sequence:
617
618- decision artifact to launch the campaign
619- report artifact for the charter
620- progress artifacts during long campaigns
621- run artifacts per analysis slice
622- report artifact for the aggregated campaign summary
623- decision artifact for the next anchor
624
625## Failure and blocked handling
626
627Record blocked or failed campaign states explicitly, such as:
628
629- missing parent run
630- analysis question under-specified
631- campaign run failed before evidence was produced
632- metrics not comparable
633- campaign conclusion still ambiguous
634
635A blocked campaign should still name the next best action.
636
637## Exit criteria
638
639Exit the analysis-campaign stage once one of the following is durably true:
640
641- the campaign produced enough evidence for writing or decision-making
642- the campaign exposed a problem that requires returning to `experiment` or `idea`
643- the campaign is blocked and the blocker is durably recorded