Ln 514 Test Log Analyzer

Paths: File paths (shared/, references/, ../ln-*) are relative to skills repo root. If not found at CWD, locate this SKILL.md directory and go up one level for repo root.

Test Log Analyzer

Two-layer analysis of application logs. Python script handles collection and quantitative analysis; AI handles classification, quality assessment, and fix recommendations.

Inputs

No required inputs. Runs in current project directory, auto-detects log sources.

Optional args — caller instructions (natural language): time window, expected errors, test context. Example: "review logs for last 30min, auth 401 errors expected from negative tests".

Purpose & Scope

• Analyze application logs (after test runs, during development, or on demand)
• Classify errors into 4 categories: Real Bug, Test Artifact, Expected Behavior, Operational Warning
• Assess log quality: noisiness, completeness, level correctness, format, structured logging
• Map stack traces to source files; provide fix recommendations
• Report findings for quality verdict (only Real Bugs block)
• No status changes or task creation — report only

When to Use

• Analyze application logs in any project (default: last 1h)
• After test runs to classify errors and assess log quality
• Can be invoked with context instructions: Skill(skill: "ln-514-test-log-analyzer", args: "review last 30min, 401 errors expected")

Workflow

Phase 0: Parse Instructions

If args provided — extract: time window (default: 1h), expected errors list, test context. If no args — use defaults (last 1h, no expected errors).

Phase 1: Log Source Detection and Script Execution

MANDATORY READ: Load docs/project/infrastructure.md, docs/project/runbook.md

1. Check if scripts/analyze_test_logs.py exists in target project. If missing, copy from references/analyze_test_logs.py.
2. Detect log source mode (auto-detection priority: docker → file → loki):

3. Run script: python scripts/analyze_test_logs.py --mode {detected} [options]
4. If no log sources found → return NO_LOG_SOURCES status, skip to Phase 5.

Phase 2: 4-Category Error Classification

Classify each error group from script JSON output:

Test artifact detection heuristics:

• Test name contains: invalid, error, fail, reject, unauthorized, forbidden, not_found, bad_request, timeout
• Test asserts non-2xx status codes (4xx, 5xx)
• Test uses pytest.raises, expect(...).rejects, assertThrows, should.throw
• Errors correlate with test execution timestamps from regression test output
• Patterns matching tests/manual/ scripts

Error taxonomy per references/error_taxonomy.md (9 categories: CRASH, TIMEOUT, AUTH, DB, NETWORK, VALIDATION, CONFIG, RESOURCE, DEPRECATION).

Phase 3: Log Quality Assessment

MANDATORY READ: Load references/error_taxonomy.md (per-level criteria table + level correctness reference)

Step 1: Detect configured log level. Check in order:

1. LOG_LEVEL / LOGLEVEL env var (.env, docker-compose.yml, infrastructure.md)
2. Framework config: Python logging.conf / Django LOGGING / Node LOG_LEVEL
3. Default: assume INFO if not detected

Configured level determines WHICH levels appear in logs, but each level has its own noise threshold regardless.

Step 2: Assess 6 quality dimensions:

Traceability gap detection — scan source code for operations without INFO-level logging:

Log Format Quality (10-criterion checklist per references/log_analysis_output_format.md):

Score: passed criteria / 10.

Phase 4: Stack Trace Mapping + Fix Recommendations

For each Real Bug:

1. Extract stack trace frames; identify origin frame (first frame in project code, not in node_modules/site-packages)
2. Map to source file:line
3. Generate fix recommendation: what to change, where, effort estimate (S/M/L)

Prioritize using Sentry-inspired dimensions:

• High-volume (occurrence count), Post-test regression (new errors), High-impact path (auth/payment/DB), Correlated traces (trace_id across services)

Phase 5: Generate Report

MANDATORY READ: Load references/log_analysis_output_format.md

Output report to chat with header ## Test Log Analysis. Include:

• Signals table (Real Bugs count, Test Artifacts filtered, Log Noise status, Log Format score, Log Quality score)
• Real Bugs table (priority, category, error, source, fix recommendation)
• Filtered table (category, count, examples)
• Log Quality Issues table (dimension, service, issue, recommendation)
• Noise Report table (count, ratio, service, level, template, action)
• Machine-readable block <!-- LOG-ANALYSIS-DATA ... --> for programmatic consumption

Phase 6: Meta-Analysis

MANDATORY READ: Load shared/references/meta_analysis_protocol.md

Skill type: execution-worker. Run after all phases complete.

Verdict Contribution

Quality coordinator normalization matrix component:

Log quality/format issues are INFORMATIONAL — do not affect quality verdict. Only Real Bugs block.

Critical Rules

• No status changes or task creation; report only.
• Test Artifacts and Expected Behavior are ALWAYS filtered — never count as bugs.
• Log quality issues are advisory — inform, don't block.
• Script must handle gracefully: no Docker, no log files, no Loki → NO_LOG_SOURCES.
• Language preservation in comments (EN/RU).

Definition of Done

• Script deployed to target project scripts/ (or already exists).
• Log source detected and script executed (or NO_LOG_SOURCES returned).
• Errors classified into 4 categories; Real Bugs identified.
• Log quality assessed (6 dimensions + 10-criterion format checklist).
• Stack traces mapped to source files for Real Bugs.
• Report output to chat with signals table + machine-readable block.

Reference Files

• Error taxonomy: references/error_taxonomy.md
• Output format: references/log_analysis_output_format.md
• Analysis script: references/analyze_test_logs.py

Version: 1.0.0 Last Updated: 2026-03-13