curiositech Cognitive Task Analysis for Agent Systems
Expert knowledge elicitation methodology focused on capturing procedural knowledge, identifying automation gaps, and matching elicitation methods to knowledge architectures.
DECISION POINTS
Primary Branch: Diagnosing Agent Performance Gaps
Is expert performance significantly better than agent performance on this task?
├─ YES: Automation gap likely
│ ├─ Agent has correct facts but wrong execution → Declarative/procedural mismatch
│ │ └─ Action: Use observational methods + process tracing
│ └─ Agent lacks pattern recognition skills → Missing classification procedures
│ └─ Action: Use think-aloud with expert cases + critical decision method
└─ NO: Knowledge completeness issue
├─ Agent fails on edge cases → Incomplete coverage
│ └─ Action: Expand expert pool + use multiple elicitation methods
└─ Agent inconsistent on similar cases → Representation format mismatch
└─ Action: Audit method-to-format coupling + redesign knowledge base
Secondary Branch: Method Selection
What type of knowledge does this task primarily require?
├─ Declarative ("knowing that" - facts, concepts, relationships)
│ └─ Use: Concept mapping, structured interviews, card sorting
├─ Procedural-Classification ("knowing which" - pattern recognition)
│ └─ Use: Think-aloud with cases, critical decision method, paired comparisons
└─ Procedural-Execution ("knowing how" - step sequences)
├─ Expert can explain steps → Use: Process tracing, task decomposition
└─ Expert cannot explain → Use: Observational shadowing + protocol analysis
Tertiary Branch: Knowledge Base Quality Assessment
How was this knowledge captured?
├─ Single method used → High risk of knowledge gaps
│ ├─ Interview/self-report only → Missing automated procedures
│ └─ Observation only → Missing conceptual frameworks
├─ Multiple methods used → Check method-knowledge type alignment
│ └─ Methods match knowledge types → Proceed to validation
└─ Unknown/undocumented → Assume incomplete capture
└─ Action: Re-elicit with method triangulation
FAILURE MODES
1. Automation Gap Blindness
Detection: Agent performs poorly on tasks experts handle effortlessly, despite having "complete" knowledge base built from expert interviews Root Cause: Automated expert knowledge is inaccessible to self-report; interview methods only capture conscious, declarative layer Fix: Add observational methods (shadowing, process tracing) and think-aloud protocols during actual task performance
2. Representation Bias
Detection: Knowledge base structure suspiciously mirrors intended output format (e.g., all knowledge fits cleanly into IF-THEN rules) Root Cause: Output format drove method selection instead of knowledge type driving method selection Fix: Re-elicit using methods matched to knowledge architecture, then adapt representation format to captured knowledge
3. Single-Method Tunnel Vision
Detection: All domain knowledge captured using one elicitation technique; gaps appear in specific contexts Root Cause: Belief that one method can capture all knowledge types; ignores differential access hypothesis Fix: Map knowledge types required by task, select complementary methods for each type, triangulate results
4. Expert Reconstruction Fallacy
Detection: Expert explanations are overly logical/linear; agent follows explanations but fails on expert-level cases Root Cause: Experts construct plausible post-hoc narratives rather than reporting actual cognitive process Fix: Use concurrent protocols (think-aloud during task) rather than retrospective explanations
5. Typological Skill Proliferation
Detection: Skill library grows continuously without consolidation; routing failures increase with library size Root Cause: Skills organized by surface features rather than knowledge architecture; lacks theoretical organizing principle Fix: Classify skills by knowledge type produced/consumed; consolidate using Hempel's reduction criterion
WORKED EXAMPLES
Example 1: Medical Diagnosis Agent Underperformance
Scenario: Radiologist expert can spot lung nodules in chest X-rays with 95% accuracy in 3 seconds. AI agent trained on same images achieves 78% accuracy with 30-second processing.
Expert Interview Attempt:
- Q: "How do you spot lung nodules?"
- A: "I look for density differences, irregular shapes, and size patterns. You check the upper lobes first, then lower lobes..."
Initial Implementation: Rule-based system checking density thresholds, shape parameters, systematic scanning pattern. Performance: 65% accuracy.
CTA Method Application:
- Knowledge Type Analysis: Task requires procedural-classification (pattern recognition), not declarative facts
- Method Selection: Think-aloud during live cases + eye-tracking observation
- Key Finding: Expert gaze patterns showed 200ms fixations on suspicious regions before conscious recognition; automated pattern matching occurred before declarative knowledge activated
Outcome: Redesigned agent with pattern recognition model trained on expert gaze data + semantic features. Performance: 91% accuracy.
What Novice Missed: Assumed expert's verbal explanation captured actual recognition process What Expert Caught: Distinguished between post-hoc explanation and real-time cognitive process
Example 2: Financial Trading Decision System
Scenario: Expert trader makes profitable split-second decisions on market volatility. Initial agent built from trading strategy documents performs poorly.
Document Analysis Approach: Extracted rules from strategy documents: "When VIX > 25 and S&P drops > 2%, buy volatility protection..."
Performance: 23% of trades profitable (random chance ≈ 30%)
CTA Method Application:
- Failure Mode Diagnosis: Automation gap - documents contain conscious strategies, not automated pattern recognition
- Method Selection: Concurrent protocol analysis during live trading + retrospective critical decision method
- Knowledge Architecture: Found three layers:
- Declarative: Market theories and formal strategies (captured in documents)
- Procedural-classification: Pattern recognition of market "feel" and momentum shifts
- Procedural-execution: Timing and sizing decisions based on risk appetite
Key Discovery: Expert's profitable decisions came from recognizing subtle momentum patterns in real-time price action, not from following documented strategies.
Redesign: Hybrid system combining formal strategies (for position direction) with pattern recognition model (for timing and sizing).
Performance: 67% profitable trades.
Example 3: Customer Service Routing System
Scenario: Expert human router assigns customer inquiries to specialist teams with 94% first-contact resolution. Automated system achieves 71%.
Initial Approach: Keyword matching based on inquiry categories expert provided in interview.
CTA Application:
- Method: Concurrent think-aloud + case comparison method
- Discovery: Expert used multiple information sources simultaneously:
- Surface content (captured by keywords)
- Customer frustration level (detected in language patterns)
- Interaction history (pattern of previous contacts)
- Team capacity and expertise overlap
Critical Insight: Expert performed multi-dimensional classification, not single-category assignment. Keyword approach captured only one dimension.
Implementation: Multi-factor routing algorithm weighing content classification, sentiment analysis, customer history clustering, and real-time team capacity.
Result: 89% first-contact resolution.
QUALITY GATES
- Knowledge elicitation used at least two different methods matched to knowledge types required
- Expert performance benchmarks established and agent performance gaps quantified
- Automated vs. conscious expert knowledge explicitly distinguished in knowledge base
- Method selection documented with rationale for each knowledge type targeted
- Representation format chosen after knowledge capture, not before
- Edge cases and failure modes tested against expert judgment samples
- Knowledge base includes both declarative concepts and procedural execution steps
- Expert reconstruction vs. actual cognitive process distinction validated through observation
- System performance approaches expert performance on time-pressured, routine decisions
- Agent can explain its reasoning at appropriate level of detail for each knowledge type
NOT-FOR BOUNDARIES
Do NOT use CTA for:
- Pure factual knowledge where experts are reliable self-reporters
- Tasks where statistical/ML approaches already match expert performance
- Domains where expert knowledge is primarily declarative and well-documented
- Simple rule-following tasks without complex pattern recognition
Delegate instead:
- For factual knowledge extraction → Use structured interviews or documentation analysis
- For statistical pattern recognition → Use machine learning with sufficient training data
- For workflow optimization → Use process mapping and lean methodologies
- For knowledge organization → Use ontology engineering approaches
This skill is specifically for capturing expert cognitive processes that are:
- Partially or fully automated (fast, unconscious)
- Involving complex pattern recognition
- Requiring procedural knowledge that experts cannot fully articulate
- Creating performance gaps between human experts and automated systems