# Llmkit

> Reference for the llmkit LLM extraction framework — cache design, Pydantic validation, audit workflow, and per-project setup. Use when writing code that does LLM-based structured extraction, setting up new extraction tasks, or working with cached LLM outputs.

- Skill: `hsigstad/llmkit` (Agent Skill)
- Install (CLI): `npx skillmds@latest add hsigstad/llmkit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/hsigstad/llmkit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: hsigstad (https://skillmd.com/u/hsigstad)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/hsigstad/llmkit

---


# llmkit — LLM Extraction Framework

Shared package at `~/research/packages/llmkit/` (`pip install -e`). Provides deterministic, auditable LLM extraction with Pydantic validation and file-backed caching. Designed for academic research where referees need to understand exactly what happened.

## Core API

```python
from llmkit import LLMCache, ExtractionSchema, extract, audit_sample
from llmkit.cache import text_hash, content_hash
from llmkit.extract import ExtractionResult
```

### `LLMCache(directory: Path)`

File-backed cache. Each entry is a JSON file with three sections: `_cache_meta`, `input_text`, `extraction`.

```python
cache = LLMCache(Path("cache_dir"))
key = cache.key(doc_id, text_hash, model)                              # composite key (back-compat)
key = cache.key(doc_id, text_hash, model, schema_name="my_task")       # schema-aware key (recommended for new tasks)
hit = cache.get(key)                       # by composite key
hit = cache.get_by_doc(doc_id)             # legacy fallback (doc_id.json)
cache.put(key, extraction, doc_id=..., text_hash=..., ...)
entries = cache.iter_entries()
```

`schema_name` is **opt-in** so existing caches (procure, EJ) stay valid. When you pass it, `schema_name` is mixed into the key — preventing collisions when two extraction tasks run against the same `(doc_id, text)` on the same model. New tasks should pass it; established tasks can migrate by renaming cache files (the new key is computable from each file's `_cache_meta`).

### `ExtractionSchema(BaseModel)`

Base class for Pydantic schemas. Subclass and set `schema_name`/`schema_version` as `ClassVar[str]`:

```python
from typing import ClassVar
from llmkit import ExtractionSchema

class MySchema(ExtractionSchema):
    schema_name: ClassVar[str] = "my_task"
    schema_version: ClassVar[str] = "v1"
    field_a: str = ""
```

### `extract(...) -> ExtractionResult`

Main entry point. Checks cache → calls LLM → validates → caches.

```python
result = extract(
    doc_id="ABC", text="...",
    system_prompt="...", user_prompt="...",
    schema=MySchema, model="gpt-4o-mini",
    prompt_file="my_prompt.txt",
    cache=cache, client=openai_client,
    reextract=False,            # True to redo stale entries
    use_structured_outputs=True, # recommended — server-side schema enforcement
    schema_in_cache_key=True,   # recommended for new tasks (see above)
)
result.valid        # bool — did Pydantic validation pass
result.parsed       # MySchema instance (or None if invalid)
result.raw          # dict — raw LLM JSON
result.cached       # bool — loaded from cache
result.stale        # bool — cached but prompt changed since
result.usage        # dict — token counts
```

### `audit_sample(results, n=50, ...)`

Stratified sampling for human review. See `llmkit/audit.py`.

### `build_report(cache, *, prices=None)` / `render_markdown(report)`

Rolls a cache directory up into the reporting items a journal data editor
needs for a paper using LLM-generated variables, per the checklist in
Coqueret, Llull, Oswald, Pérignon, Scheuch & Vilhuber (2026), *Randomness in
large language models*. Groups entries by (model, schema, prompt_hash,
source_commit) and emits, per configuration: a model card (name, served
version, dates queried), the **verbatim system prompt + one example user
message**, the generation parameters actually used, and a **cost statement**
(total input/output tokens, estimated USD, elapsed wall-clock). The rendered
Markdown is framed around the two reproducibility checks (their §5.2):
free/exact *reproduction from deposited outputs* vs. paid *regeneration from
the model* (a fresh draw, not a byte copy).

```bash
python -m llmkit.report <cache_dir> -o REPLICATION_LLM.md
llmkit-report <cache_dir> --price gpt-4o-mini=0.15/0.60   # override $/1M in/out
```

Token counts always come from the cache's per-call `usage`; the USD estimate
uses a small built-in OpenAI price table (`DEFAULT_PRICES`, clearly caveated —
verify before quoting) or `--price` overrides. Models with no price show `n/a`
and are excluded from the USD total but still contribute token counts. The
cache holds **one draw per unique input**, so the totals describe a
single-draw regeneration. Where `temperature == 0`, the report adds the
paper's caveat that this is *not* deterministic. Legacy entries (no
`_cache_meta`) are counted but carry no tokens. See `llmkit/report.py`.

## Cache design

### Cache key = `hash(doc_id, text_hash, model[, schema_name])`

- **Prompt changes do NOT invalidate cache** — intentional, to avoid re-extraction during iterative development
- **Text changes DO invalidate** — different text = different key = cache miss
- **Model changes DO invalidate** — different model = different key
- **schema_name is opt-in** — pass it for new tasks (prevents cross-task collisions on the same document); omit it to stay compatible with caches written before the schema-aware option existed

### Cache metadata (audit trail, not part of key)

Each cached JSON file contains:

```json
{
  "_cache_meta": {
    "doc_id": "2Q6JZRW4NZ243MR",
    "text_hash": "f3a1b2c4d5e6f7a8",
    "prompt_hash": "9791af5d5643f4dc",
    "model": "gpt-4o-mini",
    "model_version": "gpt-4o-mini-2024-07-18",
    "temperature": 0,
    "max_tokens": 4000,
    "schema_name": "irregularity_extraction",
    "schema_version": "v1",
    "source_commit": "ccfb959",
    "validation_status": "valid",
    "finish_reason": "stop",
    "timestamp": "2026-03-23T10:01:06+00:00",
    "usage": {"prompt_tokens": 7838, "completion_tokens": 1737},
    "api_params": {"response_format": "json_object", "top_p": 1}
  },
  "messages": [
    {"role": "system", "content": "You are an expert..."},
    {"role": "user", "content": "Extract all irregularities..."}
  ],
  "extraction": { ... LLM output ... }
}
```

Three top-level sections: `_cache_meta` (audit trail), `messages` (exact API input), `extraction` (LLM output). No redundancy — document text lives inside the user message.

### Staleness and `--reextract`

```python
entry.is_stale(current_prompt_hash="...")  # True if prompt changed
```

- **Day-to-day**: run normally, cache accumulates, prompt edits don't trigger re-extraction
- **Before submission**: pass `--reextract` to bring all entries in line with final prompt/data
- Legacy entries (no metadata) are always considered stale

### Backward compatibility

Old-style caches (`{doc_id}.json` with raw extraction, no metadata wrapper) are readable via `get_by_doc()`. The `_load()` method detects both formats.

## LLM call modes

`extract()` supports two response modes. Both write the same cache shape; the `_cache_meta.api_params.response_format` field records which was used.

### `use_structured_outputs=True` (recommended)

Calls `client.chat.completions.parse(response_format=schema)`. OpenAI enforces the Pydantic schema **server-side** via constrained decoding. The model literally cannot produce out-of-schema field names or types.

- No JSON-keyword requirement on the prompt.
- No need to spell out nested field names in the system prompt — the schema does it.
- Requires a Structured-Outputs-capable model (gpt-4o, gpt-4o-mini, gpt-4.1, o-series).
- `_cache_meta.api_params.response_format == "structured_outputs"`; `schema_name` is recorded.

### `use_structured_outputs=False` (legacy / default for back-compat)

Calls `client.chat.completions.create(response_format={"type": "json_object"})`.

- Guarantees the output parses as JSON, but **does not enforce the schema** — field names are at the model's discretion. Always spell the schema out in the system prompt under this mode.
- Requires the literal word "json" to appear somewhere in the messages.
- Default `False` so existing caches/scripts don't change behavior; new tasks should set `True`.

## `__bool__` and the `c = cache or DEFAULT` trap

`LLMCache` is **always truthy** (`__bool__` returns `True`). Before this was set explicitly, `LLMCache.__len__` made an empty cache *falsy*, so the common idiom `c = passed_cache or DEFAULT_CACHE` silently substituted the default whenever a caller passed a brand-new empty cache. If you maintain wrapper code that predates this fix, prefer `c = DEFAULT_CACHE if cache is None else cache` regardless — defensive against any future override that might want to make an LLMCache falsy.

## Per-project setup

Each project defines its own schemas and prompts; llmkit provides the machinery.

### Directory structure

```
project/source/llm/
    __init__.py
    schemas.py          # Pydantic models (inherit ExtractionSchema)
    irregularity.py     # Wrapper wiring llmkit to project config
    prompts/
        irregularity_system.txt
        irregularity_user.txt
```

### Prompt versioning

Prompts are plain text files tracked by git. **No version suffixes in filenames** — git history IS the version control. The prompt's content hash is stored in cache metadata, and `source_commit` records which repo state produced the extraction.

To reconstruct the exact prompt for a cached entry:
```bash
git show <source_commit>:source/llm/prompts/irregularity_system.txt
```

### Project wrapper pattern

The project wrapper handles legacy cache fallback and project-specific config:

```python
# source/llm/my_task.py
from llmkit import LLMCache, extract
from llmkit.cache import content_hash, text_hash
from source.llm.schemas import MySchema

CACHE = LLMCache(DATA_DIR / "my_cache")

def extract_my_task(*, doc_id, text, client, reextract=False):
    # 1. Try new composite key
    # 2. Try legacy get_by_doc() fallback
    # 3. Call LLM via extract()
    ...
```

See `procure/source/llm/irregularity.py` for the full pattern including legacy cache handling.

## Existing implementation: procure

The `procure` project has a working extraction pipeline:

- **Schema**: `source/llm/schemas.py` → `IrregularityExtraction` (document-level + per-irregularity fields)
- **Prompts**: `source/llm/prompts/irregularity_system.txt`, `irregularity_user.txt`
- **Wrapper**: `source/llm/irregularity.py` → `extract_irregularities()`
- **Script**: `source/analysis/llm_extract_irregularities.py`
- **Cache**: `DATA_DIR/tce_sp/consulta/llm_cache/` (~6,141 legacy entries + new-format entries)
- **Downstream**: `source/analysis/build_consulta_llm.py` reads `llm_extract_irregularities.json`

CLI:
```bash
python3 -m source.analysis.llm_extract_irregularities            # use cache
python3 -m source.analysis.llm_extract_irregularities --reextract # redo stale
python3 -m source.analysis.llm_extract_irregularities --force     # redo all
python3 -m source.analysis.llm_extract_irregularities --dry-run   # preview
```

## When this skill triggers automatically

Consult this skill (even without explicit `/llmkit` invocation) whenever you're about to:
- Set up LLM-based structured extraction in any project
- Work with cached LLM outputs or cache metadata
- Write Pydantic schemas for LLM validation
- Design prompts for extraction tasks
- Implement audit/review workflows for LLM outputs
- Debug cache staleness or re-extraction issues

## Gotchas

- **Cache key excludes prompt** — by design. Don't add prompt to the key.
- **`model` in key vs `model_version` in metadata** — `model` is what you requested (e.g. "gpt-4o-mini"), `model_version` is what the API actually used (e.g. "gpt-4o-mini-2024-07-18"). Both are saved.
- **Legacy cache `_usage`** — old entries store `_usage` inside the extraction dict. The wrapper strips it out before validation. Don't rely on `_usage` being in `extraction` for new entries.
- **`ExtractionSchema` uses `ClassVar`** — not underscore-prefixed attrs (Pydantic v2 treats `_`-prefixed as private attrs, which aren't JSON-serializable).
- **`messages` stores the full API input** — each file is a self-contained audit record. Expect cache files to be large for long documents. Access document text via `entry.messages[1]["content"]`.
- **`diarios` moved** — the package is now at `~/research/packages/diarios/` (was `~/research/diarios/`). Imports unchanged.
- **Cache-key changes need a rename, not a re-extract** — switching a task to `schema_in_cache_key=True` invalidates existing cache lookups. Both the old and new key are deterministic functions of the file's `_cache_meta` (`doc_id`, `text_hash`, `model`, optional `schema_name`), so a one-shot script can rename every cache file from the old key to the new one without re-calling the LLM. Do that, not `--reextract`, when migrating an established task.