intel Linux Kernel OOPS Analysis (x86)
This skill provides a structured approach to analyzing Linux kernel crash reports. It is divided into Basic Primitives for data extraction and Analysis Flows for diagnostic reasoning.
Capabilities
| Category | Description | Reference |
|---|---|---|
| Primitives | Low-level data extraction (registers, backtrace, taints) | primitives.md |
| Analysis Flows | Multi-agent pipeline orchestration (fetcher → collector → analyst → patcher/fact-checker) and crash-type-specific steps | flows.md |
| Agents | Sub-agent definitions: fetcher (Step 1), collector (Step 2), analyst (Step 3), patcher (Step 4), fact-checker (Step 5), and patch reviewer (Step 6) | agents/ |
| Fundamentals | Key Elements table, UNAME/distro, modules, taint, MSGID | fundamentals.md |
| Backtrace | Backtrace table construction, formats, IRQ/Task context | backtrace.md |
| Mapping | Resolve backtrace offsets to source lines via gdb/addr2line | mapping.md |
| Reporting | Source code formatting rules, line numbers, summarization | reporting.md |
| Distro: Debian | Download, extract, and locate debug files for Debian | debian.md |
| Distro: Debian (pre-fetch) | Fetcher-agent easy path: pool index lookup, -dbg package download |
fetch-debian.md |
| Distro: Fedora | Download, extract, and locate debug files for Fedora | fedora.md |
| Distro: Fedora (pre-fetch) | Fetcher-agent easy path: Koji RPM download, CKI git checkout | fetch-fedora.md |
| Distro: Ubuntu | Download, extract, and locate debug files for Ubuntu | ubuntu.md |
| Distro: Ubuntu (pre-fetch) | Fetcher-agent easy path: Launchpad git checkout, ddeb download | fetch-ubuntu.md |
| Bug tracker fetch | Fetch an OOPS from Launchpad, kernel Bugzilla, Debian BTS, or Red Hat Bugzilla | bugtracker.md |
| Image transcription | OCR a screenshot or decode a drm_panic QR code to extract OOPS text | image.md |
| Patch email | Prepare an LKML-ready patch email with send script when a new fix is created | patch.md |
| Analysis reply email | Generate a plain-text LKML reply (analysis-email.txt) and send script when a MSGID is available and analysis confidence is high |
decode-email.md |
| Locks held | Parse the "N locks held" lockdep block; enforce strict scope limits to prevent unbounded code exploration | lockdep.md |
| Security assessment | CVE lookup and structured severity classification (Paging Request, BUG, Panic only) | security.md |
| semcode MCP | Fast kernel code search, call graph traversal, lore email search via semcode MCP server | semcode.md |
| Examples | Sample OOPS reports for reference and testing | assets/examples/ |
Usage Guidelines
- Classification: Start by identifying the crash type (e.g., Paging Request, BUG, WARNING).
- Extraction: Use the Basic Primitives to pull structured data (Backtrace, Registers, Taint state).
- Flow Execution: Follow the specific Analysis Flow for the identified crash type to reach a root cause hypothesis.
Basic Primitives
Detailed instructions for data extraction can be found in references/primitives.md.
- Classify the crash type: Determine if it's an Oops, Panic, BUG, or WARNING.
- General fundamentals extraction: Pull kernel version, process, module list, taint flags, and hardware info.
- Backtrace extraction: Clean and format the call trace.
- Code byte line extraction: Interpret the
Code:line around the instruction pointer. - CPU Registers: Extract GPRs and special registers (CR2, etc.).
- Linux distribution specific tasks: Download and unpack distro kernel packages for offline analysis (Debian, Fedora, Ubuntu).
- Locating the vmlinux file: Heuristics to find the debug symbols.
- Locating a kernel module: Find the
.kofile for a loaded module. - Mapping backtrace lines to code: Using
addr2lineorgdb. - Reporting a source code function: Format abbreviated or full source listings with crash/call-site markers.
- Generating HTML output: Convert a Markdown report to HTML using
pandoc. - General fundamentals extraction: Demand-load fundamentals.md — Key Elements table, UNAME/distro detection, git-describe versions, crashing process, module list, taint flags, hardware name, email MSGID.
- Backtrace extraction: Demand-load backtrace.md — unified Backtrace table, RIP as first entry, Format 1/2, IRQ/Task context markers, module data, build ID hashes, "?" marker handling.
- Mapping backtrace lines to code: Demand-load mapping.md — vmlinux vs module ELF selection, addr2line/gdb resolution, return-address correction, inlined function row splitting.
- Reporting a source code function: Demand-load reporting.md — mandatory online URL, line-number prefixes, ~NNN approximation, inlined functions/macros, struct inclusion, ≤20/>20 line summarization.
- Transcribing an OOPS from a screenshot or image: Demand-load image.md — OCR via
marker_single(marker-pdf package) with vision model fallback; QR-code path for drm_panic screens; recordsIMAGE_SHA256andIMAGE_FILEin Key Elements. - Structured archive: Save reports to a source-keyed
reports/directory (e.g.reports/launchpad/<id>/) with Markdown, HTML, and optional patch files. Prompts before overwriting existing reports. - Security assessment: Demand-load security.md — CVE lookup via lore.kernel.org, and structured 4-step severity classification. Run after Deep Analysis for Paging Request, BUG/BUG_ON, and Panic crashes. Only adds a Security Note when confidence is high.
- Analysis reply email: Demand-load decode-email.md — plain-text LKML reply to the original report thread, with optional fix mention, decoded backtrace, and analysis. Only generated when a MSGID is available and analysis confidence is high.
- Locks held: Demand-load lockdep.md — parse the "N locks held" lockdep block into a table; enforce mandatory scope rules to prevent unbounded code exploration.
- semcode MCP navigation: Demand-load semcode.md — when the semcode MCP server is available, use it for fast function lookup, call graph traversal, regex/vector search over function bodies, commit history search, and lore.kernel.org email search. Check index freshness before querying; re-index
oops-workdir/linuxautonomously if stale.
Analysis Flows
Detailed diagnostic steps can be found in references/flows.md.
- Entry point: Initial triage and classification.
- "Unable to handle paging request": Memory access violation analysis.
- WARNING: Non-fatal but significant kernel issues.
- BUG / BUG_ON: Intentional kernel assertions.
- Panic: Fatal system state analysis.