# Lexoid Python > Parse and convert documents (PDFs, images, web pages, DOCX/XLSX/PPTX, audio) inside a Python program using the `lexoid` library. Use when the user is writing Python code that needs to extract markdown from documents, run schema-constrained extraction, convert files to LaTeX, get bounding boxes, recursively crawl URLs, or integrate document parsing into a larger pipeline. Triggers include `from lexoid` imports, "use lexoid in Python", "parse PDFs programmatically", "extract structured data with a Pydantic/dataclass schema", or any request to embed parsing into a Python app/notebook. Use when this capability is needed. - Skill: `tomevault-io/lexoid-python` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/lexoid-python` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/lexoid-python/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/lexoid-python --- # Lexoid Python API Lexoid's Python API is the right choice whenever the user is writing Python — notebooks, services, batch pipelines, or anything that needs the parsed result as a Python dict/list. For shell one-offs, use the `lexoid-cli` skill instead. ## When to use this skill - The user is writing Python and wants to parse PDFs, images, URLs, DOCX/XLSX/PPTX, audio, or text-format files. - The user needs per-page segments, token usage, parser metadata, or bounding boxes in code. - The user wants schema-based structured extraction (`dict`, `dataclass`, or Pydantic `BaseModel`). - The user is integrating parsing into a larger app (Streamlit, FastAPI, RAG pipeline, etc.). ## Setup checks Before writing code, confirm: 1. `lexoid` is installed (`pip install lexoid`). 2. Required API key env vars are set for the chosen provider (see below). 3. For Linux DOCX → PDF conversion, LibreOffice (`lowriter`) is on PATH. 4. For `api_provider="ollama"`: an `ollama serve` process is running at `OLLAMA_BASE_URL` (default `http://localhost:11434`) and the target model has been pulled with `ollama pull `. 5. For `api_provider="local"` (SmolDocling/granite-docling, PaddleOCR-VL): **no server needed** — these models run in-process via `transformers` / PaddleOCR. The first call downloads weights from Hugging Face, so the host needs network access (or pre-cached weights) and enough disk/RAM/GPU for the chosen model. API keys by provider: | Provider | Env var | |--------------|-------------------------------| | `gemini` | `GOOGLE_API_KEY` | | `openai` | `OPENAI_API_KEY` | | `anthropic` | `ANTHROPIC_API_KEY` | | `mistral` | `MISTRAL_API_KEY` | | `huggingface`| `HUGGINGFACEHUB_API_TOKEN` | | `together` | `TOGETHER_API_KEY` | | `openrouter` | `OPENROUTER_API_KEY` | | `fireworks` | `FIREWORKS_API_KEY` | | `ollama` | none (uses `OLLAMA_BASE_URL`) | | `local` | none | ### Loading API keys from `.env` Lexoid reads API keys from the process environment; it does **not** load a `.env` file on its own. When keys live in a project `.env`, load them before calling any LLM-based API (`LLM_PARSE`, `parse_with_schema`, `parse_to_latex`, or `AUTO` when it routes to an LLM). `python-dotenv` ships as a Lexoid dependency, so it is already available: ```python from dotenv import load_dotenv # Loads .env from the current dir (or a parent) into os.environ if the file # exists; a no-op that returns False when no .env is found. Existing env vars # are not overwritten unless override=True is passed. load_dotenv() from lexoid.api import parse result = parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o") ``` Call `load_dotenv()` once at program/notebook startup, before the first `parse(...)` call. If the keys are already exported in the environment, this step is unnecessary (and harmless). ## Public API Four entry points in `lexoid.api`: - `parse(path, parser_type="AUTO", pages_per_split=4, max_processes=4, **kwargs)` — main function. Returns a dict. - `parse_with_schema(path, schema, api=None, model="gpt-4o-mini", **kwargs)` — structured JSON extraction. Returns a Python list whose shape depends on the mode and the model's output (see "Schema return shape" below). - `parse_to_latex(path, api=None, model="gpt-4o-mini", **kwargs)` — returns a LaTeX string. - `parse_chunk(path, parser_type, **kwargs)` — low-level single-chunk parser; users rarely need this. `ParserType` enum: `LLM_PARSE`, `STATIC_PARSE`, `AUTO`. ## `parse()` return shape ```python { "raw": str, # full markdown "segments": [ # one dict per page / section; may be empty # bboxes is included only when return_bboxes=True {"metadata": {"page": int}, "content": str, "bboxes": [(text, [x0, top, x1, bottom]), ...]}, ... ], "title": str, "url": str, # input URL or "" if input was a local file "parent_title": str, # parent doc title when recursive; "" otherwise "recursive_docs": [...], # empty unless depth > 1 # --- optional keys below --- "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int}, # zeros under STATIC_PARSE-only; ABSENT on the HTML/recursive-URL path # (URL input that isn't a file-typed URL and as_pdf is not set) "parsers_used": [str, ...], # ABSENT on the HTML/recursive-URL path (same condition) "token_cost": {...}, # only when api_cost_mapping is supplied "pdf_path": str, # only when as_pdf=True; file is removed unless save_dir is also set } ``` For URL inputs that resolve to HTML (no `.pdf`/image extension, and `as_pdf=False`), `parse()` short-circuits to `recursive_read_html()`, which returns only `raw`, `segments`, `title`, `url`, `parent_title`, and `recursive_docs`. Code that always reads `result["token_usage"]` or `result["parsers_used"]` will `KeyError` on that path — use `.get(...)` or guard with `if "token_usage" in result`. ## Common recipes ### Basic parsing ```python from lexoid.api import parse result = parse("document.pdf") markdown = result["raw"] for seg in result["segments"]: print(seg["metadata"]["page"], seg["content"][:80]) ``` ### Choose a parser explicitly ```python # Native-text PDFs — fastest, no API key parse("document.pdf", parser_type="STATIC_PARSE", framework="pdfplumber") # Scanned PDFs / images — local OCR, no API key parse("scanned.pdf", parser_type="STATIC_PARSE", framework="paddleocr") # LLM parsing parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o") parse("document.pdf", parser_type="LLM_PARSE", model="gemini-2.5-pro") parse("document.pdf", parser_type="LLM_PARSE", model="claude-3-5-sonnet-20241022") ``` ### AUTO routing with a priority ```python # Speed (default): static if no images, LLM otherwise parse("doc.pdf", parser_type="AUTO", router_priority="speed") # Accuracy: prefers LLM, except PDFs with hidden hyperlinks parse("doc.pdf", parser_type="AUTO", router_priority="accuracy") # Cost: tries PaddleOCR first; LLM fallback if extracted text is too short parse("doc.pdf", parser_type="AUTO", router_priority="cost", character_threshold=100) # ML-based LLM auto-selection (uses lexoid/core/llm_selector.py) parse("doc.pdf", parser_type="AUTO", autoselect_llm=True) ``` ### Local inference (no API key) ```python # Ollama — Lexoid forces max_processes=1 for Ollama parse("doc.pdf", parser_type="LLM_PARSE", api_provider="ollama", model="gemma4:latest", max_processes=1) # SmolDocling / granite-docling parse("doc.pdf", parser_type="LLM_PARSE", api_provider="local", model="ds4sd/SmolDocling-256M-preview") # PaddleOCR-VL parse("doc.pdf", parser_type="LLM_PARSE", api_provider="local", model="PaddlePaddle/PaddleOCR-VL") ``` ### Schema-based structured extraction #### Schema return shape `parse_with_schema` returns a Python `list`. The exact shape depends on the mode and on what JSON the model emits: - **Default (per-page) mode** — one entry per page. Each entry is the JSON the model returned for that page: a single `dict` for single-record schemas, or a `list` of `dict`s for multi-record schemas (e.g., a page of table rows). Index as `result[page_index]` for the page's value, and `result[page_index][record_index]` when each page has multiple records. - **`fill_single_schema=True`** — a single-element list whose element is the JSON for the whole document (typically one `dict`). If you need to support both single- and multi-record schemas in the same code path, normalize the per-page entries yourself (e.g., wrap a `dict` in `[dict]`). ```python from lexoid.api import parse_with_schema from pydantic import BaseModel class Invoice(BaseModel): invoice_number: str total: float # Per-page extraction. `pages` is a list with one entry per page; # each entry is whatever JSON the model returned for that page. pages = parse_with_schema("invoice.pdf", schema=Invoice, model="gpt-4o-mini") # e.g., pages[0] -> {"invoice_number": "...", "total": ...} # pages[0][0] -> first record if the model returned a list per page # Single instance for the whole document — returns a one-element list. [full] = parse_with_schema("contract.pdf", schema=Invoice, model="gpt-4o", fill_single_schema=True) # Dict schema with example data + alternate keys (improves match) pages = parse_with_schema( "invoice.pdf", schema={"invoice_number": "string", "total": "number"}, example_schema={"invoice_number": "INV-001", "total": 199.95}, alternate_keys={"invoice_number": ["Invoice #", "Invoice No."]}, ) # Dataclass schemas also work from dataclasses import dataclass @dataclass class Receipt: merchant: str amount: float parse_with_schema("receipt.pdf", schema=Receipt) ``` ### LaTeX conversion ```python from lexoid.api import parse_to_latex latex_source = parse_to_latex("paper.pdf", model="gpt-4o") ``` ### URLs and recursive crawling ```python # Single page parse("https://example.com") # Crawl 2 levels deep parse("https://example.com", depth=2) # Render webpage → PDF first, then parse and keep the intermediate PDF result = parse( "https://example.com", as_pdf=True, save_dir="output/", save_filename="example.pdf", ) intermediate = result["pdf_path"] ``` ### Bounding boxes ```python result = parse("doc.pdf", return_bboxes=True, bbox_framework="auto") for seg in result["segments"]: for text, bbox in seg.get("bboxes", []): # bbox = [x0, top, x1, bottom], normalized [0, 1] ... ``` ### Audio Audio inputs require a Gemini model (the only provider with audio support). ```python result = parse("interview.mp3", model="gemini-2.5-flash") print(result["raw"]) ``` ### Token cost tracking ```python result = parse( "doc.pdf", model="gpt-4o", api_cost_mapping="tests/api_cost_mapping.json", ) print(result["token_cost"]) # {"input": ..., "output": ..., "input-image": ..., "total": ...} ``` ## Key kwargs reference | kwarg | Purpose | |-------------------------|-------------------------------------------------------------------------| | `model` | LLM model name (default from `DEFAULT_LLM`, falls back to `gemini-2.5-flash`). | | `api_provider` | Override inferred provider. | | `framework` | `pdfplumber` / `pdfminer` / `paddleocr` for STATIC_PARSE. | | `temperature` | LLM sampling temperature (default `0.0`). | | `max_tokens` | LLM output token limit (default `1024`, `4096` for Ollama). | | `pages_per_split` | Pages per parallel chunk. | | `max_processes` | Parallel workers (forced to 1 when `parser_type="LLM_PARSE"` and `api_provider="ollama"`). | | `page_nums` | Specific 1-indexed pages to parse (PDFs only). | | `depth` | Recursive URL parsing depth. | | `as_pdf` | Convert input to PDF before parsing. | | `save_dir` | Where to keep the intermediate PDF if `as_pdf=True`. | | `return_bboxes` | Attach bounding boxes per segment. | | `bbox_framework` | `auto` / `pdfplumber` / `paddleocr`. | | `router_priority` | `speed` / `accuracy` / `cost` for AUTO mode. | | `character_threshold` | Min char count for STATIC accept under `cost` priority. | | `autoselect_llm` | ML-based LLM choice in AUTO mode. | | `retry_on_fail` | Fall back to alternate parser on error (default `True`). | | `max_image_dimension` | Max px to which images / page renders are downscaled. | | `api_cost_mapping` | Dict or JSON path with per-model cost — enables `token_cost` in output. | | `system_prompt` / `user_prompt` | Override the default LLM prompts. | | `verbose` | Verbose logging during LLM parsing. | ## Things to verify before reporting success - The result dict has non-empty `raw`. Empty `raw` with an `error` key means a recoverable failure occurred and Lexoid returned a stub. - For LLM_PARSE, `token_usage["total"]` is non-zero — zero suggests the API call silently failed. - For multi-page PDFs, `len(result["segments"])` matches the expected page count (or `len(page_nums)` if used). - For `parse_with_schema`, each per-page entry may be a `dict` or a `list` of `dict`s depending on the schema and the model. Check the type before indexing, and verify that the keys actually match the schema — the LLM can drift; pass `example_schema` to anchor it. ## See also - API reference: `docs/api.rst`. - CLI equivalent: `lexoid-cli` skill. - Example notebooks: `examples/example_notebook.ipynb`, `examples/example_notebook_colab.ipynb`. --- > Source: [oidlabs-com/Lexoid](https://github.com/oidlabs-com/Lexoid) — distributed by [TomeVault](https://tomevault.io).