AI Fluency

AI Fluency Analysis

Analyze the user's past Claude Code conversations and produce an AI fluency report based on Anthropic's research.

Prerequisites

This skill requires cc-conversation-search to locate conversations. Check and install if needed:

if command -v cc-conversation-search &> /dev/null; then
    echo "Ready: $(cc-conversation-search --version)"
else
    if command -v uv &> /dev/null; then
        uv tool install cc-conversation-search
    else
        pip install --user cc-conversation-search
        export PATH="$HOME/.local/bin:$PATH"
    fi
    cc-conversation-search init --days 14
fi

Arguments

• No arguments: Analyze the last 2 weeks of conversations (default)
• A number (e.g., 7): Analyze conversations from the last N days
• A topic (e.g., "authentication"): Analyze only conversations matching that topic
• "all": Analyze all indexed conversations

Workflow

Step 1: Discover Conversations

Based on arguments, find conversations to analyze:

# Default: last 14 days
cc-conversation-search list --days 14 --json

# Or with topic filter
cc-conversation-search search "topic" --days 14 --json

Select a diverse sample of up to 8 conversations for analysis. Prefer conversations with at least 5 user messages. Aim for variety in:

• Conversation length (short and long)
• Task type (debugging, feature work, exploration, ops)
• Different projects if available

Step 2: Analyze Each Conversation

For each selected conversation, read the JSONL file and extract only messages where "role": "human". Ignore messages that are:

• Empty strings or whitespace only
• Pure tool results (automated system messages)
• Skill/command template expansions (long system prompts with frontmatter --- patterns or "Base directory for this skill" markers)

For each conversation, do TWO things:

1. Behavior detection (Layer 1): Check for the presence/absence of each of the 11 Observable Behaviors + Diligence signals
2. Dimension scoring (Layer 2): Score the 7 Practical Dimensions on a 1-10 scale

Step 3: Score and Generate HTML Report

After all conversation analyses are complete, aggregate the scores and generate an HTML report.

Use Task agents to analyze conversations in parallel (up to 4 at a time) for efficiency. Each agent should read one conversation file and return both the behavior checklist and dimension scores with evidence.

HTML Report Generation:

1. Read the template at .claude/skills/ai-fluency/report-template.html (relative to the project root)
2. Calculate all aggregated scores (behavior frequencies, dimension averages, population comparisons)
3. Replace all {{PLACEHOLDER}} markers with the actual data (see the HTML Report Placeholders section below)
4. Write the final HTML to ~/Desktop/ai-fluency-report.html
5. Open it with open ~/Desktop/ai-fluency-report.html
6. Also print a brief text summary to the conversation (overall score, level, top strength, top growth edge)

The 4D AI Fluency Framework

AI fluency is the practical skills, knowledge, insights, and values that help you use AI effectively, efficiently, ethically, and safely. The framework (developed by Professors Rick Dakan and Joseph Feller in collaboration with Anthropic) identifies four core competencies:

Most AI interactions involve small loops of Description and Discernment: describe what you need, evaluate what you get, refine your request, repeat. Delegation sets up the loop; Diligence governs it.

The Anthropic study measured 11 of 24 framework behaviors that are directly observable in conversation text. The remaining 13 (many Diligence behaviors) occur outside the chat. However, some Diligence signals ARE detectable in Claude Code conversations.

Layer 1: The 11 Observable Behaviors + Diligence Signals

These are the exact behaviors from Anthropic's study, organized by competency. For each conversation, mark whether the behavior is present or absent. A behavior is present if the user demonstrates it at least once in the conversation.

Description Behaviors (green -- task specification)

Delegation Behaviors (purple -- strategic big picture)

Delegation is about having a clear vision and strategically choosing how AI fits into your process. It's not just offloading tasks -- it's understanding your goal, recognizing what AI can and can't do well, and thoughtfully dividing the work.

Discernment Behaviors (pink -- critical evaluation)

Diligence Signals (the 4th D -- responsible use)

Diligence means taking ownership of AI-assisted work and being willing to stand behind it. Most Diligence behaviors occur outside the chat (e.g., reviewing before publishing, attributing AI involvement), but some signals ARE observable in Claude Code conversations. These don't have population baselines from the Anthropic study, so mark as present/absent without comparison.

Layer 2: The 7 Practical Dimensions (1-10 scale)

These dimensions provide nuanced scoring beyond binary behavior detection. They measure how WELL the user exhibits fluency behaviors, not just whether they appear.

Dimension 1: Specification (1-10)

How clearly the user defines tasks, constraints, and desired output.

Maps to behaviors: #2 Provides examples, #3 Specifies format, #5 Tone/style, #6 Defines audience, #7 Clarifies goal

Dimension 2: Iteration (1-10)

Whether the user refines through multi-turn dialogue vs. accepting first output.

Maps to behavior: #1 Iterates and refines

Key research finding: Iteration is the gateway behavior. Users who iterate show 2x more fluency behaviors across ALL other dimensions. Specifically:

• Clarifies goal: 54.5% vs 30.9% (+23.6pp)
• Provides examples: 44.4% vs 21.8% (+22.6pp)
• Identifies missing context: 22.8% vs 5.7% (+17.1pp)
• Specifies format: 32.4% vs 16.1% (+16.3pp)
• Questions reasoning: 17.9% vs 3.2% (+14.7pp)

Dimension 3: Discernment (1-10)

Whether the user critically evaluates AI output rather than accepting it.

Maps to behaviors: #9 Identifies missing context, #10 Questions reasoning, #11 Checks facts

Key research finding: These are the RAREST behaviors in the population. Only 8.7% of users check facts. This is the highest-leverage area for improvement for most users.

Dimension 4: Delegation (1-10)

The strategic big picture: understanding your goal, recognizing what AI can and can't do, and thoughtfully dividing the work. Delegation isn't just offloading tasks -- it's having a clear vision and strategically choosing how AI fits into your process.

Maps to behaviors: #4 Sets interaction mode, #7 Clarifies goal, #8 Consults AI on approach

Dimension 5: Task Decomposition (1-10)

Whether the user breaks complex work into appropriate sub-tasks.

Dimension 6: Tool Awareness (1-10)

Whether the user understands AI capabilities and limitations, and leverages the tool ecosystem effectively.

Dimension 7: Diligence (1-10)

Taking ownership of AI-assisted work. Using AI responsibly: protecting data, checking for bias, being transparent about AI involvement, and being willing to stand behind the final product.

Maps to signals: D1-D5

Note: Many Diligence behaviors happen outside the chat (e.g., how you present AI work to colleagues, whether you review before publishing). Score based on what's observable in the conversation, but note that this dimension may be underestimated.

The Artifact Paradox

CRITICAL DIAGNOSTIC: When AI generates artifacts (code, documents, structured output), a specific pattern emerges that you MUST check for:

Description and Delegation behaviors INCREASE with artifacts:

• Iterates and refines: 94.1% vs 84.5% (+9.7pp)
• Clarifies goal: 64% vs 49.3% (+14.7pp)
• Provides examples: 52.9% vs 39.5% (+13.4pp)
• Specifies format: 42.8% vs 28.3% (+14.5pp)

But ALL THREE Discernment behaviors DECREASE:

• Checks facts: 5.5% vs 9.2% (-3.7pp)
• Questions reasoning: 13% vs 16.2% (-3.1pp)
• Identifies missing context: 15.8% vs 21% (-5.2pp)

This means: When AI produces polished-looking output (especially code), users become MORE directive but LESS critical. They tell the AI what to build but stop questioning whether it's correct.

How to detect this in conversations:

1. Identify conversations where the AI generated code, PRs, documents, or other artifacts
2. Compare the user's discernment behaviors in those conversations vs. non-artifact conversations
3. Look specifically for: requesting PRs without reviewing diffs, accepting generated code without testing, not questioning architectural decisions when AI produces a working implementation

In the report, explicitly call out whether the user shows the artifact paradox and provide specific examples.

Scoring Guidelines

Per-Conversation Scoring

• Read ALL user messages in the conversation
• For Layer 1 (behaviors): binary present/absent for each of the 11 behaviors + 5 Diligence signals
• For Layer 2 (dimensions): score each of the 7 dimensions; score reflects the user's typical behavior, not their best single moment
• If a dimension is not applicable (e.g., no complex task to decompose), note "N/A" and exclude from averages

Cross-Conversation Scoring

• Behavior prevalence: Calculate what percentage of conversations each behavior appeared in
• Dimension averages: Average each dimension across all analyzed conversations, weighting longer conversations slightly more
• Consistency note: Flag if scores vary significantly by task type (some users are highly fluent in debugging but not in feature specification)

Overall Score

• Average of all 7 dimension averages
• Round to one decimal place

Fluency Levels

HTML Report Placeholders

The HTML template at .claude/skills/ai-fluency/report-template.html uses {{PLACEHOLDER}} markers. Replace each one:

Inline value placeholders

• {{HEADER_META}} — e.g. 8 conversations · 14 days · Feb 26, 2026
• {{HERO_SUBTITLE}} — e.g. Based on <span>8 conversations</span> over <span>14 days</span> &mdash; Anthropic 4D AI Fluency Index <em>(Dakan &amp; Feller, 2026)</em>
• {{OVERALL_SCORE}} — e.g. 5.7 (displayed in gauge)
• {{LEVEL_BADGE}} — e.g. Iterative <span class="arrow">&rarr;</span> approaching Evaluative (use the fluency level from scoring guidelines)
• {{TOTAL_CONVERSATIONS}} — e.g. 8
• {{OVERALL_SCORE_NUM}} — e.g. 5.7 (used in JS gauge animation)

Behavior cards placeholder ({{BEHAVIOR_CARDS}})

Generate 4 <div class="card"> blocks — one for each 4D competency (Description, Delegation, Discernment, Diligence). Each card contains behavior items:

<div class="card">
  <div class="card-title"><span class="dim-dot description"></span> <span class="dim-label description">Description</span> &mdash; clear communication</div>
  <div class="behavior-item">
    <div class="behavior-freq" style="color: var(--color-green);">6/8</div>
    <div class="behavior-text">Iterates and refines <span class="pct">(pop: 85.7%)</span></div>
    <span class="status-badge above">Above</span>
  </div>
  <!-- ... more behavior-items ... -->
</div>

Status badge classes: well-above, above, at, below, violation. Freq color: green if above population baseline, amber if within 10pp, red if below or 0.

Strengths placeholder ({{STRENGTHS_HTML}})

Generate 2-3 <div class="insight-card strength"> blocks with quotes in <blockquote> tags:

<div class="insight-card strength">
  <div class="insight-number strength">Strength 1</div>
  <div class="insight-title">Title with score</div>
  <div class="insight-body">
    <p>Description paragraph.</p>
    <blockquote>"user quote here" <span class="conv-ref">&mdash; Conv N</span></blockquote>
    <p>More analysis.</p>
  </div>
</div>

Growth edges placeholder ({{EDGES_HTML}})

Generate 2-3 <div class="insight-card edge"> blocks. Include <div class="suggestion-box"> for actionable advice:

<div class="insight-card edge">
  <div class="insight-number edge">Edge 1</div>
  <div class="insight-title">Title with score</div>
  <div class="insight-body">
    <p>Description.</p>
    <blockquote>"user quote" <span class="conv-ref">&mdash; Conv N</span></blockquote>
    <div class="suggestion-box"><strong>Suggestion:</strong> Actionable advice here.</div>
  </div>
</div>

Artifact paradox placeholder ({{PARADOX_HTML}})

Generate an <div class="insight-card paradox"> with a <div class="paradox-grid"> containing two paradox-case divs (one .bad, one .good) for comparison:

<div class="insight-card paradox">
  <div class="insight-number paradox">Verdict: [Present/Absent/Partial]</div>
  <div class="insight-title">Summary title</div>
  <div class="insight-body">
    <p>Analysis paragraph.</p>
    <div class="paradox-grid">
      <div class="paradox-case bad">
        <div class="paradox-case-title">Textbook Paradox &mdash; Conv N</div>
        <p>Description of the paradox case.</p>
        <div class="paradox-scores">
          <span class="paradox-score bad">Disc: 3</span>
          <span class="paradox-score bad">Iter: 1</span>
        </div>
      </div>
      <div class="paradox-case good">
        <div class="paradox-case-title">Counter-example &mdash; Conv N</div>
        <p>Description.</p>
        <div class="paradox-scores">
          <span class="paradox-score good">Disc: 8</span>
          <span class="paradox-score good">Iter: 8</span>
        </div>
      </div>
    </div>
    <p><strong>Pattern:</strong> Summary of when the paradox activates.</p>
  </div>
</div>

If no artifact paradox is detected, use a single .good case with an explanation.

Profile summary placeholder ({{PROFILE_SUMMARY_HTML}})

Generate 2-3 <p> tags summarizing the comparison to population norms. Use <strong> for emphasis.

JS data array placeholders

{{CONVERSATIONS_JS_ARRAY}} — JSON array of conversation objects:

[
  { num: 1, topic: 'Topic name', msgs: 6, spec: 2, iter: 2, disc: 4, deleg: 6, decomp: 2, tools: 5, dilig: 3, overall: 3.4 },
  // ... one per conversation
]

{{DIMENSIONS_JS_ARRAY}} — JSON array of dimension objects:

[
  { name: 'Specification', score: 4.4, assessment: 'Brief assessment text' },
  // ... 7 dimensions + Overall
]

{{COMPARISONS_JS_ARRAY}} — JSON array of population comparison objects:

[
  { label: 'Iteration rate', pop: 85.7, user: 75.0, delta: -10.7 },
  // ... 11 behaviors
]

Methodology and footer placeholders

• {{METHODOLOGY_TEXT}} — Methodology paragraph explaining the framework, sample size, scoring approach, and baseline source
• {{FOOTER_DATE_LINE}} — e.g. Generated February 26, 2026 · Framework: Anthropic 4D AI Fluency Index v1.0

Report Content Guidelines

For each report section, follow these content guidelines:

Section 4: Signature Strengths

Top 2-3 patterns with specific quotes from conversations as evidence.

Section 5: Growth Edges

Top 2-3 areas for improvement with specific quotes and actionable suggestions.

Section 6: The Artifact Paradox Check

Explicitly state whether the user shows the artifact paradox pattern. Compare their discernment behaviors in artifact-producing conversations vs. non-artifact conversations. Provide specific examples.

Section 7: Comparison to Anthropic's Research

Highlight behaviors where the user significantly exceeds or falls below the baseline (>10pp delta).

Analyzing User Messages from JSONL

Conversation files are JSONL (one JSON object per line). To extract user messages:

1. Each line is a JSON object with fields including type, message (which has role and content)
2. User messages have "role": "human" inside the message object
3. The content field may be a string or an array of content blocks
4. Filter out:
   • Empty or whitespace-only content
   • Messages that are purely tool_result blocks (automated responses to tool calls)
   • Very long messages that are clearly system-generated skill/command expansions (check for frontmatter --- patterns or "Base directory for this skill" markers)
5. Count and analyze the remaining substantive human messages

Use Task agents to analyze conversations in parallel (up to 4 at a time) for efficiency. Each agent should read one conversation file and return both the behavior checklist and dimension scores with evidence.

Reference

This skill is based on:

• Anthropic's AI Fluency Index (Jan 2026): https://www.anthropic.com/research/AI-fluency-index
• 4D AI Fluency Framework by Professor Rick Dakan (Ringling College of Art and Design) and Professor Joseph Feller, developed in collaboration with Anthropic
• Analysis of 9,830 Claude.ai conversations from January 2026
• The framework identifies 24 total behaviors for safe and effective human-AI collaboration; 11 are directly observable in conversation text (the remaining 13 occur outside the chat interface)

The 4 Ds (from Rick Dakan's lecture):

• Delegation: The big picture. What are you trying to accomplish? What work should you handle yourself? Where might AI be helpful? "Delegation isn't just about offloading tasks. It's about having a clear vision and strategically choosing how AI fits into your process."
• Description: Clear communication. Goes beyond just writing prompts -- it's about detailed, context-rich conversations that establish what you're hoping to achieve, format of output, how you want AI to approach the task, context it needs, and tone/style of interaction.
• Discernment: Thoughtfully evaluating what AI gives you. Are the facts accurate? Does the reasoning make sense? Does this output actually help you move forward? "Most of our interactions with AI involve small loops of description and discernment."
• Diligence: Responsible AI interactions. Ensuring fairness, verifying accuracy, protecting sensitive data, being transparent about AI involvement, and taking ownership. "Diligence means taking ownership of your AI-assisted work and being willing to stand behind final products created using AI."

Key findings from the study:

• 85.7% of conversations show iteration (the gateway behavior)
• Iterators exhibit 2.67 additional fluency behaviors (~2x the non-iterative rate)
• Only 8.7% of users check facts; only 15.8% question reasoning
• The artifact paradox: when AI generates code/documents, users become more directive but less critical (all 3 discernment behaviors decrease)
• Only 30% of users set interaction mode (direct how the AI should work with them)
• Only 10.1% consult the AI on approach before execution
• These competencies "aren't tied to specific AI tools or techniques that might become outdated -- they're fundamental skills that will help you adapt and grow alongside this rapidly evolving technology"