Analyze Chat Session

Analyze Chat Session

Analyze the current conversation's tool calls, responses, errors, and overall approach to produce a structured feedback report. The goal is continuous improvement of MCP tool usage patterns, prompt strategies, and ARC-1 server capabilities.

This skill is introspective — the LLM analyzes its own chat session (the conversation you're currently in, or a referenced one) and generates actionable feedback.

PRIVACY NOTICE — READ FIRST

Before using this skill's output outside the current system (e.g., pasting into a GitHub issue, sharing in a chat), the user MUST review it for sensitive data.

The analysis MUST NOT contain:

• SAP system URLs, hostnames, IP addresses, or ports
• Usernames, passwords, API keys, tokens, or credentials
• Client numbers, system IDs (SID), or environment identifiers
• Real business data (customer names, order numbers, financial values, employee data)
• File paths that reveal internal infrastructure (home directories, mount paths)
• Transport request numbers or package names that reveal internal naming conventions
• Any content from SAP table previews or SQL query results containing business data

The LLM must actively redact these from the output, replacing them with generic placeholders like <SAP_HOST>, <USERNAME>, <OBJECT_NAME>, <TABLE_DATA_REDACTED>, etc.

After generating the report, remind the user: "Please review this report for any remaining sensitive information before sharing it outside your organization."

Smart Defaults (apply silently, do NOT ask)

Proceed immediately with Smart Defaults when triggered. No questions needed.

Input

The user triggers this skill at any point during (or after) a conversation. No additional input is required — the LLM analyzes the current chat context. Proceed immediately with Smart Defaults.

Optionally, the user may specify:

• Focus area — e.g., "focus on the write operations" or "focus on the search failures"
• Output format — issue (GitHub issue-ready), internal (detailed, for internal review), or summary (brief, 1 paragraph)
• Severity filter — only report issues above a threshold (e.g., "only blockers")

Phase 1: Catalog the Session

Walk through the entire conversation and extract a structured catalog. For each turn, note:

1a. Identify the Goal

What was the user trying to accomplish? Categorize it:

1b. Catalog Tool Calls

For each MCP tool call in the session, record:

1c. Identify Patterns

Look for these anti-patterns:

Phase 2: Assess What Worked

2a. Successful Strategies

Identify and highlight things that worked well:

• Efficient tool selection — chose the right tool on the first try
• Good parameter usage — used format=structured, include=elements, method="*", etc.
• Effective error recovery — followed error hints, switched strategies after failures
• Context-first approach — used SAPContext/SAPSearch before diving into reads
• Batch operations — used batch_create or batch activate where appropriate
• Method surgery — used edit_method instead of full class rewrite

2b. What Partially Worked

Identify approaches that eventually succeeded but were inefficient:

• How many extra calls were needed?
• What was the corrective action?
• Could the first attempt have succeeded with different parameters?

2c. What Failed

Identify approaches that never worked:

• Was the information actually available through a different path?
• Was it a system limitation (BTP, permissions, missing feature)?
• Was it a tool limitation that should be reported?

Phase 3: Generate Improvement Suggestions

3a. For the LLM (Prompt/Approach Improvements)

Suggestions the user can feed back into system prompts or skill instructions:

• Strategy improvements — "Start with SAPSearch before SAPRead when object type is uncertain"
• Parameter optimizations — "Use format=structured for class reads to reduce call count"
• Sequence improvements — "Read method listing before attempting method surgery"
• Recovery patterns — "When SAPContext fails for FUNC, pass the group= parameter explicitly"

3b. For ARC-1 (Server/Tool Improvements)

Suggestions that should become GitHub issues for the ARC-1 project:

3c. For the User (Workflow Improvements)

Suggestions for the user's workflow:

• "Provide the object type for live usage lookups to avoid an extra resolution call"
• "Use the explain-abap-code skill first to understand objects before modifying them"
• "Provide object type when possible to avoid search round-trips"

3d. Delivery Triage (Quick Wins vs Planned Work)

For every recommendation, classify delivery scope explicitly:

Output two lists:

1. Quick wins to execute now (ordered by leverage)
2. Planned items (grouped into the minimum number of implementation plans)

This triage section is mandatory in issue and internal outputs.

Phase 4: Produce the Report

Format: issue (Default — GitHub Issue Ready)

Generate a report structured for creating a GitHub issue. Use this template:

## Session Analysis: <brief goal description>

### Goal
<1-2 sentences: what the user was trying to accomplish>

### Approach Summary
<3-5 bullet points: high-level steps taken>

### Tool Call Statistics
- Total calls: N
- Successful: N (N%)
- Failed: N (N%)
- Redundant/unnecessary: N (N%)
- Tools used: SAPRead (N), SAPSearch (N), SAPWrite (N), ...

### What Worked Well
- <bullet points of effective strategies>

### What Didn't Work
- <bullet points with error classification>
  - **Error**: <redacted error summary>
  - **Root cause**: <analysis>
  - **Better approach**: <suggestion>

### Suggested ARC-1 Improvements

#### <Improvement 1 Title>
**Category**: Missing operation | Poor error message | Missing parameter | ...
**Severity**: blocker | major | minor | enhancement
**Description**: <what should change and why>
**Reproduction**: <generic steps, NO sensitive data>

#### <Improvement 2 Title>
...

### Delivery Triage
- **Quick Wins (this chat):**
  - <item>
- **Needs Plan + Implementation Phase:**
  - <item>
  - <item>

### Prompt Strategy Recommendations
- <numbered list of approach improvements for future sessions>

---
*Generated by analyze-chat-session skill. **Review for sensitive data before sharing.***

Format: internal (Detailed Internal Review)

Same as issue but includes:

• Full tool call catalog table (with redacted parameters)
• Token usage estimates per call
• Sequence diagram of the call flow
• Detailed anti-pattern analysis with specific turn references

Format: summary (Brief)

A single paragraph summarizing: goal, outcome, top 2 things that worked, top 2 things that didn't, and the single most impactful improvement suggestion.

Phase 5: Offer Next Steps

After presenting the report, offer:

1. "Create GitHub issues?" — For each suggested ARC-1 improvement, offer to draft a GitHub issue body (still requires user review before posting)
2. "Refine a specific finding?" — Dive deeper into one particular issue
3. "Retry the failed approach?" — If the analysis reveals a better strategy, offer to try it now
4. "Save as skill feedback?" — If the analysis reveals patterns that should be encoded in a skill, offer to update the relevant .claude/commands/*.md file

Reference: ARC-1 Tool Capabilities Quick Reference

Use this reference to assess whether the LLM used the right tools. This is a concise mapping of common goals to optimal tool usage.

Read Operations (SAPRead)

Search (SAPSearch)

• searchType=object — by name/wildcard
• searchType=source_code — full-text (may be unavailable on some systems)

Write Sequence

1. Read first (understand current state)
2. SAPWrite action=create or action=update or action=edit_method
3. SAPActivate (single or batch with objects[])
4. SAPDiagnose action=syntax or action=unittest to verify

Context & Navigation

• SAPContext action=deps — dependency graph (preferred over manual reads)
• SAPContext action=usages — live reverse dependencies from SAP
• SAPNavigate action=definition — go-to-definition (needs URI + line/col + source)
• SAPNavigate action=references — where-used

Common Error Recovery Paths

Error Handling

Notes

When to Use This Skill

• After a complex debugging or development session
• When a new tool pattern was discovered through trial and error
• When multiple approaches failed and you want to document what was learned
• When preparing feedback for ARC-1 improvement
• When onboarding and wanting to capture learning patterns

What This Skill Does NOT Do

• No automated issue creation — it drafts issue content, but the user must review and create
• No system access — this skill only reads the chat history, it makes no MCP tool calls
• No cross-session analysis — it analyzes one conversation at a time
• No real-time monitoring — it's a post-hoc analysis, not a live dashboard