OutlineDriven Skill doctor
Contract
| Field | Bound contract |
|---|---|
| Trigger | User wants agent setup graded from conversation history. |
| Authority | Reversible local: writes only to a scratch report directory under the current working tree; rollback is deleting that directory. No remote mutation. |
| Side effect | Creates one scratch report directory containing report.html and supporting assets. Never modifies real skill files, configuration, or conversation history. |
| Done | report.html exists in the scratch directory and renders a normalized 0-10 efficiency score, a normalized 0-10 code quality score (or N/A when the session has no code changes), per-turn findings with evidence citations (or no findings when every scored sub-score is at maximum), and ranked improvement suggestions (or none when every scored sub-score is at maximum). |
Inputs
- Conversation history: a session transcript from the current agent harness. Accept one of:
- A file path to a transcript log (JSON, JSONL, or plain-text log).
- A directory of transcript files; process the most recent file.
- If neither is supplied, attempt to locate the most recent session transcript in the default harness log directory.
- Output directory (optional): path for the scratch report directory. Default:
./skill-doctor-report-<timestamp>.
Procedure
Collect session data.
- Locate the conversation history from the supplied path, directory, or default harness log.
- If no transcript is found, write a minimal report.html stating "No conversation history found" and stop.
- Read the transcript file. If the file is empty or contains fewer than 2 turns, write a minimal report.html stating "Session too short to assess" and stop. Done when: the transcript is located and read, or a minimal report is written for the empty/short case.
Decode transcript into structured turns.
- Parse each message into a turn record with fields:
role(user/assistant/tool),content(text),tool_name(if applicable),tool_input(if applicable),tool_output(if applicable),timestamp(if available). - For JSON/JSONL transcripts, parse each line or object directly.
- For plain-text logs, identify turn boundaries by role markers (e.g., "User:", "Assistant:", "Tool:", or harness-specific prefixes). Extract tool calls from fenced code blocks or structured markers.
- Produce an ordered list of turn records. Done when: an ordered list of turn records is produced.
- Parse each message into a turn record with fields:
Score efficiency (0-10). Evaluate the agent's task-completion efficiency against these criteria:
- Task completion (0-3): Did the agent accomplish the stated goal? 3 = fully complete, 2 = mostly complete with minor gaps, 1 = partial progress, 0 = no progress.
- Unnecessary exploration (0-3, inverted): Did the agent read or explore files unrelated to the task? 0 = extensive off-topic exploration, 1 = moderate, 2 = minimal, 3 = only task-relevant files touched.
- Retry waste (0-2, inverted): How many failed attempts or redundant retries occurred? 2 = none, 1 = one retry, 0 = multiple retries.
- Turn economy (0-2, inverted): Was the turn count reasonable for the task complexity? 2 = concise, 1 = slightly verbose, 0 = significantly excessive turns.
- Sum sub-scores for the raw efficiency score (0-10). Done when: the efficiency score (0-10) is computed.
Score code quality (0-10). Evaluate the agent's code changes against these criteria:
- Correct file identification (0-3): Did the agent modify the right files and symbols? 3 = all correct, 2 = one wrong file, 1 = multiple wrong files, 0 = entirely wrong targets.
- Minimal diff (0-3): Were changes precise and minimal? 3 = surgical edits, 2 = minor unnecessary changes, 1 = moderate bloat, 0 = large unnecessary rewrites.
- Pattern adherence (0-2): Did changes follow existing codebase conventions? 2 = fully consistent, 1 = minor style deviations, 0 = ignores existing patterns.
- Regression avoidance (0-2): Did changes avoid introducing bugs, dead code, or broken imports? 2 = clean, 1 = minor issues, 0 = introduced regressions.
- Sum sub-scores for the raw code quality score (0-10).
- If the session contains no code changes, set code quality to N/A and state the reason in the score summary; do not emit this as a finding. Done when: the code quality score (0-10 or N/A) is computed.
Normalize scores.
- Efficiency normalized = raw efficiency score (already 0-10).
- Code quality normalized = raw code quality score (already 0-10), or N/A if no code changes. Done when: both scores are normalized.
Compile findings with evidence.
- For each sub-score that is not at maximum, produce a finding with:
- Category name (e.g., "Unnecessary exploration").
- Score and maximum.
- Evidence: cite specific turn numbers and content excerpts that justify the score.
- Suggestion: one concrete, actionable improvement.
- Rank suggestions by potential score impact (highest possible improvement first). Done when: all non-maximum sub-scores have findings with evidence and ranked suggestions.
- For each sub-score that is not at maximum, produce a finding with:
Generate report.html.
- Create the output directory.
- Write report.html as a self-contained HTML file with embedded CSS (no external dependencies, no network requests).
- Structure:
- Header: "Skill Doctor Report" with session identifier and timestamp.
- Score summary: two horizontal bar charts (efficiency, code quality) showing normalized scores out of 10. If code quality is N/A, show "N/A" instead of a bar and state the reason beside it.
- Findings section: each finding as a card with category, score, evidence block (cited turn excerpts in a styled blockquote), and suggestion.
- Suggestions summary: ranked list of all improvement suggestions.
- Use clean, readable styling: white background, dark text, clear section headings, adequate spacing. No external fonts, no JavaScript, no network dependencies. Done when: report.html is written with all sections present.
Verify report.
- Confirm report.html exists in the output directory.
- Confirm it contains the score summary, at least one finding, and at least one suggestion (unless the session was too short or had no history, or every scored sub-score is at maximum, in which case the minimal report satisfies the done predicate). Done when: report.html exists and contains the required sections.
Failure and recovery
| Failure class | Detection | Response |
|---|---|---|
| No conversation history | Transcript path does not exist or directory is empty | Write minimal report.html stating "No conversation history found". Skill terminates; no scores produced. |
| Session too short | Fewer than 2 turns after decoding | Write minimal report.html stating "Session too short to assess". Skill terminates; no scores produced. |
| Unparseable format | Decoder cannot identify turn boundaries | Write report.html with a parsing error note and any partial turns decoded. Mark affected scores as N/A. |
| Write failure | OS error creating output directory or writing report.html | Report the error. No partial artifacts left on disk; delete the output directory if partially created. |
Partial results: if decoding succeeds but scoring encounters an edge case (e.g., no code changes), produce the report with N/A for affected scores rather than failing.
Rollback: delete the entire scratch output directory to undo all effects.
Non-converged result: if the report cannot be generated at all, the skill returns a text description of the failure reason without creating any files.
Output
A self-contained report.html in the scratch output directory: normalized 0-10 efficiency score (always), normalized 0-10 code quality score or N/A, per-category findings with evidence-cited turn excerpts, and ranked improvement suggestions with potential score impact.