ExploitGym-paper-to-security-skill

ExploitGym-paper-to-security-skill

Overview

This skill transforms the paper ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? into structured, reusable security artifacts. The paper introduces a benchmark for evaluating AI agents on real-world exploitation tasks across userspace, V8, and Linux kernel targets, with 898 instances and containerized evaluation environments.

The skill is designed to extract the paper's core ideas, identify reusable patterns, and produce documentation that is useful for security research, benchmarking, and skill authoring. It is intended for defensive analysis and research organization, not operational exploitation guidance.

Goals

• Summarize the paper in clear French Markdown.
• Extract benchmark structure, evaluation protocol, and key results.
• Identify reusable design patterns for similar benchmarks.
• Convert the paper into a skill-oriented outline.
• Produce defensive implications and research follow-ups.

Inputs

• pdf_text: Extracted text from the paper.
• markdown_notes: Optional notes or highlights.
• metadata: Optional structured metadata such as title, authors, or source.

Outputs

• summary.md: Concise executive summary.
• facts.json: Structured key facts and metrics.
• analysis.md: Technical analysis of methodology and results.
• defense.md: Defensive implications and security takeaways.
• skill_ideas.md: Reusable skill ideas derived from the paper.
• skill_stub.md: Draft skill skeleton for reuse.

Operating Principles

• Prefer faithful extraction over speculation.
• Preserve paper terminology when it is important.
• Separate facts from interpretation.
• Focus on high-level reasoning and benchmark design.
• Avoid procedural exploit instructions.
• Keep output consistent, structured, and easy to lint.

Processing Steps

1. Read the paper text and detect its major sections.
2. Extract title, authors, abstract, domains, and evaluation protocol.
3. Capture benchmark size, task categories, mitigation settings, and reported results.
4. Summarize the paper's main claims and limitations.
5. Derive reusable patterns for a generic research-to-skill pipeline.
6. Write the outputs in clean Markdown and JSON.

Extraction Schema

facts.json

{
  "paper_title": "ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?",
  "paper_type": "benchmark paper",
  "domains": [
    "userspace",
    "V8",
    "Linux kernel"
  ],
  "instance_count": 898,
  "evaluation_focus": "AI agent exploitation capability",
  "core_metrics": [
    "success rate",
    "flag capture",
    "judge-confirmed intended exploit use",
    "cost",
    "wall-clock time"
  ],
  "top_results": [
    {
      "model": "Claude Mythos Preview + Claude Code",
      "successes": 157
    },
    {
      "model": "GPT-5.5 + Codex CLI",
      "successes": 120
    }
  ]
}

Markdown Rules

· Use ATX headings only. · Keep headings short and meaningful. · Use bullet lists for parallel facts. · Prefer one idea per paragraph. · Avoid tables unless they add clear value. · Keep code blocks fenced and minimal. · Do not insert raw URLs in prose unless necessary. · Keep output deterministic and easy to diff.

Skill Logic

Paper summarization

Extract:

· the benchmark objective, · the task setup, · the evaluation protocol, · the main findings, · the defensive implications.

Benchmark analysis

Identify:

· how tasks are constructed, · what information the agent receives, · how success is verified, · which mitigations are evaluated, · which results are most significant.

Reusable skill ideas

Convert the paper into candidate downstream skills such as:

· paper summarizer, · benchmark extractor, · defensive implication analyzer, · case study generator, · evaluation pipeline designer.

Safety Boundaries

This skill must remain focused on analysis, summarization, and research organization. It must not provide instructions that facilitate offensive exploitation or misuse of vulnerabilities.

Example Workflow

1. Ingest the paper text.
2. Produce summary.md.
3. Produce facts.json.
4. Produce analysis.md.
5. Produce defense.md.
6. Produce skill_ideas.md.
7. Produce skill_stub.md.

Expected Style

The generated material should be:

· concise, · precise, · technically informed, · readable by a software engineer, · suitable for inclusion in a GitHub repository.

Maintenance Notes

· Update the skill if the paper version changes. · Keep the extracted metrics synchronized with the source. · Prefer reproducible formatting over editorial embellishment. · Ensure all generated Markdown remains markdownlint-friendly.