Paper Labeler

Paper Labeler

Classify LLM-for-code research papers from conference proceedings into a two-level label taxonomy.

Pipeline Overview

Four steps, run in sequence:

1. extract_papers.py — Parse raw venue files into a uniform JSON format
   • For NDSS: also run fetch_ndss_abstracts.py to scrape abstracts from paper pages
2. label_papers.py — Filter by relevance keywords, then label with Claude API
3. merge_labeldata.py — Merge results into data/labeldata/labeldata.json
4. build_site.py — Rebuild the website from updated labeldata

Batch mode:

• process_folder.py — Scan a rawdata folder, skip venues already in data/venues.json, and run the full pipeline (extract → label → merge → rebuild) for new venues only.

Additionally:

• import_original.py — Import papers from data/rawdata/original.json that are missing from labeldata, mapping old-style labels to the current taxonomy via Claude API. Avoids duplicates.

All scripts are in .claude/skills/paper-labeler/scripts/.

Step-by-Step Workflow

Batch Mode: Process a Folder

Scan a rawdata folder, skip venues already in data/venues.json, and run the full pipeline for each new venue. This is the simplest way to process new data.

# Process all unprocessed venues under data/rawdata/
python .claude/skills/paper-labeler/scripts/process_folder.py

# Process a specific year folder
python .claude/skills/paper-labeler/scripts/process_folder.py data/rawdata/2025/

# Dry-run: show which venues would be processed
python .claude/skills/paper-labeler/scripts/process_folder.py --dry-run

# Filter only (no API calls, no merge)
python .claude/skills/paper-labeler/scripts/process_folder.py --filter-only

Options:

• --dry-run — show which venues are new vs already processed, no action
• --filter-only — keyword filter only, no Claude API calls or merge
• --model MODEL — Bedrock model ID (default: us.anthropic.claude-sonnet-4-6)
• --region REGION — AWS region (default: us-east-1)
• --delay SECONDS — delay between API calls (default: 0.5)
• --no-rebuild — skip website rebuild after processing

The script:

1. Reads data/venues.json to get already-processed venues
2. Scans the folder for .bib and .html files, groups them by canonical venue name
3. Skips venues already in venues.json
4. For each new venue: extract → (fetch NDSS abstracts if needed) → filter+label → merge
5. Updates venues.json after each successful venue
6. Rebuilds the website once at the end

Venue names are derived from filenames: EMNLP-findings2024.html and EMNLP-main2024.html both map to EMNLP2024, so multiple track files for the same venue are processed together.

Step 1: Extract Papers

Determine the input format and run the extractor.

# BibTeX files (SE/PL/Security venues — most common)
python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/ASE2025.bib > /tmp/extracted.json

# ACL Anthology HTML files (ACL, EMNLP, NAACL)
python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/ACL2025.html > /tmp/extracted.json

# NDSS HTML files (titles only — abstracts need individual page fetching)
python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/NDSS2025.html > /tmp/extracted.json

Format auto-detection works by file extension and content. Override with --format bib|acl|ndss.

URL handling:

• BibTeX: uses url field if present; otherwise auto-constructs from doi field (https://doi.org/{doi})
• ACL Anthology HTML: extracts URLs from <a href=...> in title links (e.g., https://aclanthology.org/2025.acl-long.1/)
• NDSS HTML: extracts URLs from paper links
• If a paper's key is a DOI (e.g., 10.5555/...), it can be used as a fallback URL via https://doi.org/{key}

For NDSS: the HTML only contains titles and links (no inline abstracts). After extraction, use fetch_ndss_abstracts.py to scrape abstracts from individual paper pages:

python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/NDSS2025.html > /tmp/ndss_extracted.json
python .claude/skills/paper-labeler/scripts/fetch_ndss_abstracts.py /tmp/ndss_extracted.json -o /tmp/extracted.json

Options for fetch_ndss_abstracts.py:

• --delay SECONDS — delay between HTTP requests (default: 1.0)
• --resume — skip papers already in the output file (for resuming after interruption)

Step 2: Filter and Label

Phase 1 — Keyword filter (no API needed):

python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/extracted.json --phase filter -o /tmp/filtered.json

This keeps papers whose title+abstract mention both LLM-related AND code-related terms. It is deliberately conservative (high recall) to avoid missing relevant papers.

Phase 2 — Claude labeling via AWS Bedrock (requires AWS credentials):

python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/filtered.json --phase label -o /tmp/labeled.json

Or run both phases at once:

python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/extracted.json --phase all -o /tmp/labeled.json

Options:

• --model MODEL — Bedrock model ID (default: us.anthropic.claude-sonnet-4-6)
• --region REGION — AWS region (default: us-east-1)
• --delay SECONDS — delay between API calls (default: 0.5)
• --dry-run — preview what would be labeled, no API calls

Step 3: Merge into labeldata

python .claude/skills/paper-labeler/scripts/merge_labeldata.py /tmp/labeled.json

Options:

• --dry-run — show merge stats without writing
• --no-overwrite — skip papers already in labeldata.json
• --labeldata PATH — write to a different file

Step 4: Rebuild Website

After merging new papers into labeldata.json, rebuild the website:

python .claude/skills/paper-labeler/scripts/build_site.py

This reads data/labeldata/labeldata.json and generates web/index.html — a single-page app with category sidebar (grouped into three super-groups: Agent for SE, Agent Design and Analysis, Evaluation), search, year/venue filters, and expandable abstracts.

Label Taxonomy

The complete two-level taxonomy is hardcoded below. A paper can belong to multiple categories.

1. Code Generation

• Program Synthesis — generating code from natural language descriptions, formal specifications, or input-output examples; includes NL2Code, text-to-SQL, text-to-code, sketch-based synthesis, inductive synthesis, competition programming, code infilling
• Code Completion — token-level, line-level, or block-level prediction in editing context; next-token prediction, fill-in-the-middle, IDE autocompletion, inline suggestion, code recommendation
• Program Repair — automated bug fixing, patch generation, vulnerability remediation, error correction, program healing, fault repair, security patch, hotfix generation
• Code Translation — cross-language migration, transpilation, language porting, API migration, framework migration, legacy modernization, parallel code translation
• Decompilation — recovering high-level source code from binary, bytecode, IR, or WebAssembly; lifting, binary-to-source, disassembly-to-C, neural decompilation
• Refactoring — code transformation, extract method, rename, move method, code cleanup, style improvement, dead code removal, design pattern application, technical debt reduction

2. Static Analysis

• Bug Detection — non-security defects, code smells, anti-patterns, defect prediction, error-prone pattern detection, anomaly detection in code, coding rule violation
• Program Verification — formal proof generation, loop invariant synthesis, model checking, theorem proving, Hoare logic, SMT-based verification, bounded model checking, safety property verification, correctness proof, program equivalence
• Specification Inference — precondition inference, postcondition inference, contract generation, API specification mining, behavioral specification, protocol inference, function summary generation
• Pointer Analysis — points-to analysis, alias analysis, heap modeling, shape analysis, pointer dereference safety, null pointer detection
• Call Graph Analysis — call graph construction, function call resolution, virtual dispatch resolution, dynamic dispatch analysis, indirect call analysis, call graph pruning, call graph completeness
• Type Inference — type prediction, type annotation generation, gradual typing, type migration, TypeScript/Python type inference, generic type instantiation, type error detection
• Data-flow Analysis — reaching definitions, def-use chains, live variable analysis, constant propagation, value-flow analysis, information flow analysis, data dependency analysis
• Taint Analysis — taint propagation, source-sink analysis, injection vulnerability detection, untrusted input tracking, sanitization checking, taint policy enforcement
• Symbolic Execution — path exploration, constraint solving, concolic execution, symbolic path analysis, path feasibility checking, automatic test input generation via symbolic methods
• Abstract Interpretation — numerical abstract domains, interval analysis, polyhedral analysis, widening/narrowing, semantic abstraction, fixpoint computation, over-approximation
• Code Summarization — function-level summary, module-level summary, code documentation generation, code explanation, natural language description of code, code narration, code captioning
• Code Search — semantic code search, natural language code query, code retrieval, code recommendation, API discovery, example-based code search, cross-lingual code search
• Clone Detection — code clone detection, near-miss clone, semantic clone, Type-1/2/3/4 clone, code plagiarism, code similarity measurement, code duplication

3. Dynamic Analysis

• Test Case Generation — unit test generation, integration test generation, system test generation, regression test generation, parameterized test, property-based test, boundary value test, test input generation
• Test Oracle — assertion generation, expected output inference, metamorphic relation, test oracle derivation, differential oracle, invariant-based oracle, specification-based oracle
• Fuzzing — coverage-guided fuzzing, grammar-based fuzzing, mutation-based fuzzing, protocol fuzzing, API fuzzing, kernel fuzzing, IoT fuzzing, seed generation, harness generation, directed fuzzing
• Debugging — root cause analysis, automated debugging, fault localization, spectrum-based fault localization, delta debugging, statistical debugging, interactive debugging, failure explanation
• Mutation Testing — mutant generation, equivalent mutant detection, mutation score, mutation operator, higher-order mutant, mutation-based test assessment
• Bug Reproduction — crash reproduction, bug report reproduction, failure-inducing input, crash deduplication, core dump analysis, reproducing field failures
• Domain-Specific Testing — DBMS testing, compiler testing, network protocol testing, kernel testing, GUI testing, mobile app testing, web application testing, embedded system testing, smart contract testing
• PoC and Exploit Generation — proof-of-concept generation, exploit synthesis, vulnerability triggering, attack payload construction, CVE reproduction, exploit chain construction

4. Model Safety and Security

• Adversarial Attack — adversarial examples against code models, evasion attacks, adversarial perturbation, identifier renaming attack, dead code insertion, semantic-preserving perturbation, adversarial robustness evaluation
• Backdoor Detection — trojan detection in code models, backdoor trigger identification, poisoned model detection, neural cleanse for code models, training data poisoning defense, supply chain attack on models, model integrity verification
• Memorization — training data extraction, membership inference attack, data contamination detection, benchmark contamination, verbatim memorization, privacy leakage, copyright infringement, data provenance
• Secure Code Generation — generating vulnerability-free code, security-aware code generation, CWE-free generation, constrained decoding for security, RLHF for safe code, security policy enforcement during generation
• Watermarking — code watermark embedding, watermark detection, AI-generated code identification, code authorship attribution, code provenance verification, model fingerprinting, intellectual property protection
• Jailbreaking — bypassing model safety alignment, automated red teaming, safety guardrail evasion, harmful code generation, misuse potential evaluation, safety benchmark, attack taxonomy for code LLMs

5. Agent Safety and Security

• Prompt Injection — direct prompt injection, indirect prompt injection, context manipulation, instruction override, data exfiltration via prompt, cross-plugin attack, prompt leaking, goal hijacking
• Agent Defense — runtime trace analysis, agent sandboxing, guardrail enforcement, output filtering, safety monitoring, anomaly detection in agent behavior, agent alignment, safe tool invocation
• Access Control — permission management, least privilege enforcement, capability restriction, resource access policy, file system access control, network access control, API scope limitation, privilege escalation prevention, credential handling

6. Agent Design

• Planning — task decomposition, hierarchical planning, multi-step reasoning, goal-directed planning, plan generation, plan refinement, backtracking, subgoal identification, workflow orchestration, chain-of-thought, tree-of-thought, graph-of-thought
• Memory Management — context window management, long-term memory, episodic memory, working memory, conversation history compression, retrieval-augmented generation, knowledge base grounding, repo-level context retrieval, codebase indexing
• Tool Use — function calling, API invocation, compiler interaction, debugger integration, test execution, shell command execution, code interpreter, web browsing, database query, external tool orchestration, MCP
• Multi-Agent — multi-agent collaboration, pair programming, role-based agent teams, agent communication protocol, task delegation, agent negotiation, debate-based verification, ensemble of agents, agent specialization

7. Code Model

• Model Training — pre-training objectives (MLM, CLM, denoising), code-specific pre-training, continual pre-training, parameter-efficient fine-tuning (LoRA, adapter, prefix tuning), instruction tuning, RLHF, DPO, data curation, synthetic data generation, curriculum learning, code tokenization
• Binary and IR Model — binary code representation learning, LLVM IR modeling, assembly language model, WebAssembly model, bytecode model, cross-architecture binary analysis, binary similarity, binary function embedding

8. Other SE Tasks

• Code Review — automated code review, review comment generation, review score prediction, code change assessment, reviewer recommendation, review quality evaluation, pull request analysis
• Doc/Comment/Commit Message Generation — docstring generation, inline comment generation, commit message generation, changelog generation, API documentation, README generation, release note generation, Javadoc
• Log Analysis — log parsing, log anomaly detection, logging statement generation, log template extraction, log-based failure diagnosis, log level recommendation, structured logging

9. Evaluation

• Benchmark — dataset construction, evaluation framework, leaderboard, test suite, code generation benchmark, analysis benchmark, security benchmark, multi-task benchmark, cross-lingual benchmark, execution-based evaluation, human evaluation protocol
• Empirical Study — large-scale evaluation, comparative study, user study, developer survey, ablation study, replication study, industrial case study, mixed-methods study, A/B testing of LLM tools
• Survey — literature review, systematic literature review, systematic mapping study, meta-analysis, research roadmap, taxonomy study, state-of-the-art review, tutorial overview

Label Format

Each paper's labels field is a flat list containing both top-level and sub-level labels. Example: ["Static Analysis", "Bug Detection", "Evaluation", "Benchmark"]

Relevance Criteria

A paper is relevant if LLMs or AI agents are a central part of the paper's own contribution — either:

1. Using LLMs/agents to solve software engineering tasks (code generation, analysis, testing, repair, etc.), OR
2. Studying properties of code LLMs/agents themselves (safety, security, benchmarking, training)

NOT relevant:

• Papers that only mention LLMs/agents in background/related work but don't use or study them
• Traditional ML (SVM, random forest), traditional DL (CNN, RNN, GNN) applied to SE — unless specifically targeting code LLMs (e.g., adversarial attacks on code models)
• General NLP/vision/speech unrelated to code
• Neural network verification, ML compilers, quantum computing, pure PL theory
• General SE papers that don't use LLMs/agents as a core component

Output Format

The output JSON matches data/labeldata/labeldata.json schema:

{
  "Paper Title": {
    "type": "INPROCEEDINGS",
    "key": "...",
    "author": "...",
    "booktitle": "...",
    "title": "Paper Title",
    "year": "2025",
    "abstract": "...",
    "labels": ["Code Generation", "Program Synthesis"],
    "venue": "ASE2025",
    "url": "...",
    ...
  }
}

Common Patterns

Batch-process all new venues in a folder:

# Dry-run first to see what's new
python .claude/skills/paper-labeler/scripts/process_folder.py data/rawdata/2025/ --dry-run

# Process all new venues (skips already-processed ones)
python .claude/skills/paper-labeler/scripts/process_folder.py data/rawdata/2025/

Process a new SE venue (bib file):

python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/ASE2025.bib > /tmp/extracted.json
python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/extracted.json --phase all -o /tmp/labeled_ASE2025.json
python .claude/skills/paper-labeler/scripts/merge_labeldata.py /tmp/labeled_ASE2025.json --dry-run
python .claude/skills/paper-labeler/scripts/merge_labeldata.py /tmp/labeled_ASE2025.json
python .claude/skills/paper-labeler/scripts/build_site.py

Process a new NLP venue (ACL Anthology HTML):

python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/ACL2025.html > /tmp/extracted.json
python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/extracted.json --phase all -o /tmp/labeled_ACL2025.json
python .claude/skills/paper-labeler/scripts/merge_labeldata.py /tmp/labeled_ACL2025.json
python .claude/skills/paper-labeler/scripts/build_site.py

Process an NDSS venue (HTML — requires abstract fetching):

python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/NDSS2025.html > /tmp/ndss_extracted.json
python .claude/skills/paper-labeler/scripts/fetch_ndss_abstracts.py /tmp/ndss_extracted.json -o /tmp/extracted.json
python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/extracted.json --phase all -o /tmp/labeled_NDSS2025.json
python .claude/skills/paper-labeler/scripts/merge_labeldata.py /tmp/labeled_NDSS2025.json
python .claude/skills/paper-labeler/scripts/build_site.py

Preview without API calls:

python .claude/skills/paper-labeler/scripts/extract_papers.py data/rawdata/2025/CCS2025.bib > /tmp/extracted.json
python .claude/skills/paper-labeler/scripts/label_papers.py /tmp/extracted.json --phase all --dry-run

Note: Phase 2 uses AWS Bedrock (boto3). Ensure AWS credentials are configured (~/.aws/credentials, environment variables, or IAM role). Abstract fetching (fetch_ndss_abstracts.py) requires requests (pip install requests).