all publishers

xberg-io

@xberg-io source repo

112 published skills · page 1 of 2

  1. ▌
    Liter LLM Streaming Responses · xberg-io bundle
    Use when streaming tokens incrementally from an LLM via liter-llm over SSE or async iterators. Covers chat_stream, delta handling, and null-content chunks.
    9.6k repo stars
  2. ▌
    Liter LLM Using The MCP Server · xberg-io bundle
    Use when calling LLM APIs through the liter-llm MCP server's 22 tools, and to decide when MCP beats the CLI or SDK. Covers the tool surface, the auto-installing launcher, and authentication.
    9.6k repo stars
  3. ▌
    Liter LLM Embeddings And Search · xberg-io bundle
    Use when generating embeddings, calling the 12 web-search providers, or running OCR over documents with the 4 OCR providers through liter-llm. Covers embed, search, and ocr methods plus reranking.
    9.6k repo stars
  4. ▌
    Liter LLM 3 · xberg-io bundle
    Universal LLM API client for 174 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.
    9.6k repo stars
  5. ▌
    Liter LLM Calling Llms · xberg-io bundle
    Use when sending chat completions through liter-llm and routing to a specific provider via the `provider/model` prefix. Covers the chat call shape, provider routing, model_hint, message roles, and error categories.
    9.6k repo stars
  6. ▌
    Liter LLM Tool Calling · xberg-io bundle
    Use when defining functions/tools for an LLM to call through liter-llm, or requesting structured JSON outputs. Covers tool schemas, tool_calls handling, and response formats.
    9.6k repo stars
  7. ▌
    Liter LLM Running The Proxy · xberg-io bundle
    Use when running the `liter-llm api` OpenAI-compatible gateway — virtual keys, per-key rate limits, budgets, cost tracking, and model routing. Covers the TOML config and the 22 REST endpoints.
    9.6k repo stars
  8. ▌
    Extracting Tables 5 · xberg-io
    Use when extracting tabular data from HTML. Covers GFM Markdown tables, the structured tables array (grid cells plus pre-rendered markdown), and HTML line breaks in table cells.
    9.6k repo stars
  9. ▌
    Using The MCP Server 5 · xberg-io
    Use when converting HTML to Markdown or extracting metadata and tables through the html-to-markdown MCP server's tools, rather than shelling out to the CLI. Covers the tool surface, the auto-installing launcher, and when MCP beats the CLI or SDK.
    9.6k repo stars
  10. ▌
    CLI 2 · xberg-io
    Use crawlberg cli reference documentation.
    9.6k repo stars
  11. ▌
    Crawlberg 3 · xberg-io
    Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
    9.6k repo stars
  12. ▌
    Mapping Urls 3 · xberg-io
    Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map URL` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
    9.6k repo stars
  13. ▌
    Serving The API 3 · xberg-io
    Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.
    9.6k repo stars
  14. ▌
    Headless Fallback 3 · xberg-io
    Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
    9.6k repo stars
  15. ▌
    Automating The Browser 3 · xberg-io
    Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.
    9.6k repo stars
  16. ▌
    Crate Structure 2 · xberg-io
    Map of the html-to-markdown workspace — the seven `crates/` (core, cli, ffi, py, node, php, wasm) with the unsafe_code forbid/override rule, the out-of-workspace language packages under `packages/`, and the primary `convert()` API with its ConversionResult shape, feature flags, and dual parser. Load when navigating crates/packages, deciding where code belongs, or working with the core conversion API.
    9.6k repo stars
  17. ▌
    Converting HTML 4 · xberg-io
    Use when converting HTML to Markdown, Djot, or plain text. Covers output formats, heading and code-block styles, lists, escaping, wrapping, and HTML preprocessing.
    9.6k repo stars
  18. ▌
    HTML To Markdown 4 · xberg-io bundle
    Convert HTML to Markdown, Djot, or plain text with structured extraction. Use when writing code that calls html-to-markdown APIs in Rust, Python, TypeScript, Go, Ruby, PHP, Java, C#, Elixir, R, C, or WASM. Covers installation, conversion, configuration, metadata extraction, tables, document structure, inline images, URL fetching, and CLI usage.
    9.6k repo stars
  19. ▌
    Extracting Metadata 4 · xberg-io
    Use when extracting metadata from HTML — title, description, language, Open Graph, JSON-LD / Microdata / RDFa, headers, links, and images. Covers the --json output shape and the --extract-metadata flag.
    9.6k repo stars
  20. ▌
    Fetching And Converting Urls 2 · xberg-io
    Use when fetching a live URL and converting it to Markdown. Covers --url, custom user agents, preprocessing for noisy pages, and the --json ConversionResult shape.
    9.6k repo stars
  21. ▌
    API 2 · xberg-io
    Use tree-sitter-language-pack api reference documentation.
    9.6k repo stars
  22. ▌
    MCP 2 · xberg-io
    Use tree-sitter-language-pack mcp reference documentation.
    9.6k repo stars
  23. ▌
    Parsing Source 2 · xberg-io
    Use when the user wants a tree-sitter syntax tree for a source file — an s-expression dump or JSON tree. Covers `ts-pack parse`, language auto-detection vs `--language`, stdin input, and reading `has_errors`.
    9.6k repo stars
  24. ▌
    Managing Parsers 4 · xberg-io
    Use when the user needs to manage the tree-sitter parser cache — prefetch parsers for offline or CI runs, list what is downloaded, inspect a language, find the cache directory, or clean it. Covers `ts-pack download`, `list`, `info`, `cache-dir`, `clean`, and `init`.
    9.6k repo stars
  25. ▌
    Chunking For Llms 2 · xberg-io
    Use when the user wants to split source code into chunks for an LLM context window without breaking syntax mid-construct. Covers `ts-pack process --chunk-size`, why syntax-aware splits beat fixed-byte splits, picking a size, and the chunk JSON shape.
    9.6k repo stars
  26. ▌
    Detecting Languages 4 · xberg-io
    Use when the user wants to know which programming language a file or snippet is. Covers implicit detection in `ts-pack parse`/`process`, confirming support with `ts-pack list`/`info`, and the SDK detection functions for path, extension, and raw content.
    9.6k repo stars
  27. ▌
    Extracting Code Structure 4 · xberg-io
    Use when the user wants structured code metadata from a source file — functions, classes, imports, exports, symbols, docstrings, comments, or syntax diagnostics. Covers `ts-pack process` feature flags, the JSON result shape, and the default feature set.
    9.6k repo stars
  28. ▌
    Tree Sitter Language Pack 4 · xberg-io
    Parse and extract code intelligence from 371 programming languages using tree-sitter grammars. Use when writing code that parses source, extracts structure/imports/exports/symbols/docstrings/comments, detects a language, runs syntax diagnostics, or produces syntax-aware chunks for LLMs — in Rust, Python, Node.js/TypeScript, or the ts-pack CLI. Covers installation, the CLI surface, the SDK surface, and parser-cache management.
    9.6k repo stars
  29. ▌
    Xberg 2 · xberg-io bundle
    Extract text, tables, metadata, and images from 98+ document formats (PDF, Office, images, HTML, email, archives, academic) using Xberg. Use when writing code that calls Xberg APIs in Python, Node.js/TypeScript, Rust, or CLI. Covers installation, extraction (sync/async), configuration (OCR, chunking, output format), batch processing, error handling, and plugins.
    9.6k repo stars
  30. ▌
    Chunking 2 · xberg-io
    Use when splitting extracted text into chunks for LLM context windows or RAG ingestion. Covers chunk size, overlap, markdown/yaml/semantic chunkers, tokenizer-based sizing, and the standalone `chunk` command.
    9.6k repo stars
  31. ▌
    Liter LLM 2 · xberg-io
    Universal LLM API client for 143 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.
    9.6k repo stars
  32. ▌
    Batch Extraction 2 · xberg-io
    Use when extracting from many files at once with shared config, bounded parallelism, per-file overrides, and error recovery. Covers the `batch` command, `--file-configs`, `--max-concurrent`, and output layout.
    9.6k repo stars
  33. ▌
    Picking A Format 2 · xberg-io
    Use when choosing an output format for extracted documents — text, markdown, djot, html, or JSON. Maps consumer (LLM, parser, archive) to the right `--format` / `--content-format` pair.
    9.6k repo stars
  34. ▌
    Extracting Tables 3 · xberg-io
    Use when extracting tabular data from PDFs, spreadsheets, or images. Covers layout-aware table detection, table model selection, output formats (markdown / JSON cells), and known limits.
    9.6k repo stars
  35. ▌
    Crawling A Site 3 · xberg-io
    Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
    9.6k repo stars
  36. ▌
    Extracting Keywords 2 · xberg-io
    Use when extracting keywords (YAKE/RAKE) from documents — and, secondarily, when detecting document language or generating embeddings for RAG and search. Covers the keyword config (and its feature gating), `--detect-language`, and the standalone `embed` command with real flags.
    9.6k repo stars
  37. ▌
    Extracting With Ocr 2 · xberg-io
    Use when extracting text from scanned PDFs, photographed pages, or images that have no embedded text layer. Covers OCR backends, language packs, force-OCR, and performance tuning.
    9.6k repo stars
  38. ▌
    Running The Proxy 2 · xberg-io
    Use when running the `liter-llm api` OpenAI-compatible gateway — virtual keys, per-key rate limits, budgets, cost tracking, and model routing. Covers the TOML config and the 22 REST endpoints.
    9.6k repo stars
  39. ▌
    Streaming Responses 2 · xberg-io
    Use when streaming tokens incrementally from an LLM via liter-llm over SSE or async iterators. Covers chat_stream, delta handling, and null-content chunks.
    9.6k repo stars
  40. ▌
    Using The MCP Server 3 · xberg-io
    Use when calling LLM APIs through the liter-llm MCP server's 22 tools, and to decide when MCP beats the CLI or SDK. Covers the tool surface, the auto-installing launcher, and authentication.
    9.6k repo stars
  41. ▌
    Converting HTML 3 · xberg-io
    Use when converting HTML to Markdown, Djot, or plain text. Covers output formats, heading and code-block styles, lists, escaping, wrapping, and HTML preprocessing.
    9.6k repo stars
  42. ▌
    HTML To Markdown 3 · xberg-io bundle
    Convert HTML to Markdown, Djot, or plain text with structured extraction. Use when writing code that calls html-to-markdown APIs in Rust, Python, TypeScript, Go, Ruby, PHP, Java, C#, Elixir, R, C, or WASM. Covers installation, conversion, configuration, metadata extraction, tables, document structure, inline images, URL fetching, and CLI usage.
    9.6k repo stars
  43. ▌
    Extracting Tables 4 · xberg-io
    Use when extracting tabular data from HTML. Covers GFM Markdown tables, the structured tables array (grid cells plus pre-rendered markdown), and HTML line breaks in table cells.
    9.6k repo stars
  44. ▌
    Scraping HTML To Markdown 3 · xberg-io
    Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape URL`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.
    9.6k repo stars
  45. ▌
    Extracting Metadata 3 · xberg-io
    Use when extracting metadata from HTML — title, description, language, Open Graph, JSON-LD / Microdata / RDFa, headers, links, and images. Covers the --json output shape and the --extract-metadata flag.
    9.6k repo stars
  46. ▌
    Managing Parsers 3 · xberg-io
    Use when the user needs to manage the tree-sitter parser cache — prefetch parsers for offline or CI runs, list what is downloaded, inspect a language, find the cache directory, or clean it. Covers `ts-pack download`, `list`, `info`, `cache-dir`, `clean`, and `init`.
    9.6k repo stars
  47. ▌
    Detecting Languages 3 · xberg-io
    Use when the user wants to know which programming language a file or snippet is. Covers implicit detection in `ts-pack parse`/`process`, confirming support with `ts-pack list`/`info`, and the SDK detection functions for path, extension, and raw content.
    9.6k repo stars
  48. ▌
    Using The MCP Server 4 · xberg-io
    Use when parsing source, extracting code structure, or detecting a language through the tree-sitter-language-pack MCP server's tools, rather than shelling out to the ts-pack CLI. Covers the tool surface, the auto-installing launcher, and when MCP beats the CLI or SDK.
    9.6k repo stars
  49. ▌
    Extracting Code Structure 3 · xberg-io
    Use when the user wants structured code metadata from a source file — functions, classes, imports, exports, symbols, docstrings, comments, or syntax diagnostics. Covers `ts-pack process` feature flags, the JSON result shape, and the default feature set.
    9.6k repo stars
  50. ▌
    Tree Sitter Language Pack 3 · xberg-io
    Parse and extract code intelligence from 306 programming languages using tree-sitter grammars. Use when writing code that parses source, extracts structure/imports/exports/symbols/docstrings/comments, detects a language, runs syntax diagnostics, or produces syntax-aware chunks for LLMs — in Rust, Python, Node.js/TypeScript, or the ts-pack CLI. Covers installation, the CLI surface, the SDK surface, and parser-cache management.
    9.6k repo stars
  51. ▌
    Managing Parsers 2 · xberg-io
    Use when the user needs to manage the tree-sitter parser cache — prefetch parsers for offline or CI runs, list what is downloaded, inspect a language, find the cache directory, or clean it. Covers `ts-pack download`, `list`, `info`, `cache-dir`, `clean`, and `init`.
    9.6k repo stars
  52. ▌
    Detecting Languages 2 · xberg-io
    Use when the user wants to know which programming language a file or snippet is. Covers implicit detection in `ts-pack parse`/`process`, confirming support with `ts-pack list`/`info`, and the SDK detection functions for path, extension, and raw content.
    9.6k repo stars
  53. ▌
    Extracting Code Structure 2 · xberg-io
    Use when the user wants structured code metadata from a source file — functions, classes, imports, exports, symbols, docstrings, comments, or syntax diagnostics. Covers `ts-pack process` feature flags, the JSON result shape, and the default feature set.
    9.6k repo stars
  54. ▌
    Tree Sitter Language Pack 2 · xberg-io
    Parse and extract code intelligence from 371 programming languages using tree-sitter grammars. Use when writing code that parses source, extracts structure/imports/exports/symbols/docstrings/comments, detects a language, runs syntax diagnostics, or produces syntax-aware chunks for LLMs — in Rust, Python, Node.js/TypeScript, or the ts-pack CLI. Covers installation, the CLI surface, the SDK surface, and parser-cache management.
    9.6k repo stars
  55. ▌
    Liter LLM · xberg-io
    Universal LLM API client for 143 providers with native bindings for 14 languages. Use when writing code that calls LLM APIs via liter-llm in Python, TypeScript, Rust, Go, Java, C#, Ruby, PHP, Elixir, WASM, or C, when running the OpenAI-compatible proxy, or when calling LLMs through the MCP server. Covers chat, streaming, tool calling, embeddings, image generation, speech, transcription, moderation, web search, OCR, reranking, provider routing, middleware, and configuration.
    9.6k repo stars
  56. ▌
    Running The Proxy · xberg-io
    Use when running the `liter-llm api` OpenAI-compatible gateway — virtual keys, per-key rate limits, budgets, cost tracking, and model routing. Covers the TOML config and the 22 REST endpoints.
    9.6k repo stars
  57. ▌
    Streaming Responses · xberg-io
    Use when streaming tokens incrementally from an LLM via liter-llm over SSE or async iterators. Covers chat_stream, delta handling, and null-content chunks.
    9.6k repo stars
  58. ▌
    Extracting Tables 2 · xberg-io
    Use when extracting tabular data from HTML. Covers GFM Markdown tables, the structured tables array (grid cells plus pre-rendered markdown), and HTML line breaks in table cells.
    9.6k repo stars
  59. ▌
    Using The MCP Server 2 · xberg-io
    Use when parsing source, extracting code structure, or detecting a language through the tree-sitter-language-pack MCP server's tools, rather than shelling out to the ts-pack CLI. Covers the tool surface, the auto-installing launcher, and when MCP beats the CLI or SDK.
    9.6k repo stars
  60. ▌
    Converting HTML 2 · xberg-io
    Use when converting HTML to Markdown, Djot, or plain text. Covers output formats, heading and code-block styles, lists, escaping, wrapping, and HTML preprocessing.
    9.6k repo stars
  61. ▌
    HTML To Markdown 2 · xberg-io bundle
    Convert HTML to Markdown, Djot, or plain text with structured extraction. Use when writing code that calls html-to-markdown APIs in Rust, Python, TypeScript, Go, Ruby, PHP, Java, C#, Elixir, R, C, or WASM. Covers installation, conversion, configuration, metadata extraction, tables, document structure, inline images, URL fetching, and CLI usage.
    9.6k repo stars
  62. ▌
    Extracting Metadata 2 · xberg-io
    Use when extracting metadata from HTML — title, description, language, Open Graph, JSON-LD / Microdata / RDFa, headers, links, and images. Covers the --json output shape and the --extract-metadata flag.
    9.6k repo stars
  63. ▌
    Crawlberg 2 · xberg-io
    Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
    9.6k repo stars
  64. ▌
    Mapping Urls 2 · xberg-io
    Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map URL` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
    9.6k repo stars
  65. ▌
    Crawling A Site 2 · xberg-io
    Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
    9.6k repo stars
  66. ▌
    Serving The API 2 · xberg-io
    Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.
    9.6k repo stars
  67. ▌
    Headless Fallback 2 · xberg-io
    Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
    9.6k repo stars
  68. ▌
    Automating The Browser 2 · xberg-io
    Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.
    9.6k repo stars
  69. ▌
    Scraping HTML To Markdown 2 · xberg-io
    Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape URL`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.
    9.6k repo stars
  70. ▌
    Formula Maintenance · xberg-io
    Homebrew tap formula maintenance for xberg-io CLIs — source-build vs binary-only formula types, the version-bump checklist (url/version/sha256/bottles/root_url), and building bottles with scripts/build-bottles.sh. Load when editing a .rb formula, cutting a release bump, or updating bottle hashes.
    9.6k repo stars
  71. ▌
    Converting HTML · xberg-io
    Use when converting HTML to Markdown, Djot, or plain text. Covers output formats, heading and code-block styles, lists, escaping, wrapping, and HTML preprocessing.
    9.6k repo stars
  72. ▌
    HTML To Markdown · xberg-io bundle
    Convert HTML to Markdown, Djot, or plain text with structured extraction. Use when writing code that calls html-to-markdown APIs in Rust, Python, TypeScript, Go, Ruby, PHP, Java, C#, Elixir, R, C, or WASM. Covers installation, conversion, configuration, metadata extraction, tables, document structure, inline images, URL fetching, and CLI usage.
    9.6k repo stars
  73. ▌
    Extracting Metadata · xberg-io
    Use when extracting metadata from HTML — title, description, language, Open Graph, JSON-LD / Microdata / RDFa, headers, links, and images. Covers the --json output shape and the --extract-metadata flag.
    9.6k repo stars
  74. ▌
    Fetching And Converting Urls · xberg-io
    Use when fetching a live URL and converting it to Markdown. Covers --url, custom user agents, preprocessing for noisy pages, and the --json ConversionResult shape.
    9.6k repo stars
  75. ▌
    Parsing Source · xberg-io
    Use when the user wants a tree-sitter syntax tree for a source file — an s-expression dump or JSON tree. Covers `ts-pack parse`, language auto-detection vs `--language`, stdin input, and reading `has_errors`.
    9.6k repo stars
  76. ▌
    Managing Parsers · xberg-io
    Use when the user needs to manage the tree-sitter parser cache — prefetch parsers for offline or CI runs, list what is downloaded, inspect a language, find the cache directory, or clean it. Covers `ts-pack download`, `list`, `info`, `cache-dir`, `clean`, and `init`.
    9.6k repo stars
  77. ▌
    Chunking For Llms · xberg-io
    Use when the user wants to split source code into chunks for an LLM context window without breaking syntax mid-construct. Covers `ts-pack process --chunk-size`, why syntax-aware splits beat fixed-byte splits, picking a size, and the chunk JSON shape.
    9.6k repo stars
  78. ▌
    Detecting Languages · xberg-io
    Use when the user wants to know which programming language a file or snippet is. Covers implicit detection in `ts-pack parse`/`process`, confirming support with `ts-pack list`/`info`, and the SDK detection functions for path, extension, and raw content.
    9.6k repo stars
  79. ▌
    Using The MCP Server · xberg-io
    Use when parsing source, extracting code structure, or detecting a language through the tree-sitter-language-pack MCP server's tools, rather than shelling out to the ts-pack CLI. Covers the tool surface, the auto-installing launcher, and when MCP beats the CLI or SDK.
    9.6k repo stars
  80. ▌
    Extracting Code Structure · xberg-io
    Use when the user wants structured code metadata from a source file — functions, classes, imports, exports, symbols, docstrings, comments, or syntax diagnostics. Covers `ts-pack process` feature flags, the JSON result shape, and the default feature set.
    9.6k repo stars
  81. ▌
    Tree Sitter Language Pack · xberg-io
    Parse and extract code intelligence from 371 programming languages using tree-sitter grammars. Use when writing code that parses source, extracts structure/imports/exports/symbols/docstrings/comments, detects a language, runs syntax diagnostics, or produces syntax-aware chunks for LLMs — in Rust, Python, Node.js/TypeScript, or the ts-pack CLI. Covers installation, the CLI surface, the SDK surface, and parser-cache management.
    9.6k repo stars
  82. ▌
    API · xberg-io
    Use crawlberg api reference documentation.
    9.6k repo stars
  83. ▌
    CLI · xberg-io
    Use crawlberg cli reference documentation.
    9.6k repo stars
  84. ▌
    MCP · xberg-io
    Use crawlberg mcp reference documentation.
    9.6k repo stars
  85. ▌
    Crawlberg · xberg-io
    Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
    9.6k repo stars
  86. ▌
    Dev Cycle · xberg-io
    Crawlberg iteration loops codified as Taskfile tasks — alef install/generate/format/bump, core and binding builds, e2e generate/build/test cycles, cleanup tiers, and the mock-server / stale-.so / precompiled-NIF / generated-e2e gotchas. Load when running or debugging crawlberg build, alef regeneration, or e2e workflows, or when unsure which task to run.
    9.6k repo stars
  87. ▌
    Mapping Urls · xberg-io
    Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map URL` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
    9.6k repo stars
  88. ▌
    Crawling A Site · xberg-io
    Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
    9.6k repo stars
  89. ▌
    Serving The API · xberg-io
    Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server — for example wiring crawlberg into other apps over REST. Covers `crawlberg serve`, the Firecrawl-v1-compatible endpoints, `--host`/`--port`, and when to prefer it.
    9.6k repo stars
  90. ▌
    Headless Fallback · xberg-io
    Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
    9.6k repo stars
  91. ▌
    Automating The Browser · xberg-io
    Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.
    9.6k repo stars
  92. ▌
    Scraping HTML To Markdown · xberg-io
    Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape URL`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.
    9.6k repo stars
  93. ▌
    Xberg · xberg-io bundle
    Extract text, tables, metadata, and images from 107 document formats (PDF, Office, images, HTML, email, archives, academic) using Xberg. Use when writing code that calls Xberg APIs in Python, Node.js/TypeScript, Rust, or CLI. Covers installation, extraction (sync/async), configuration (OCR, chunking, output format), batch processing, error handling, and plugins.
    9.6k repo stars
  94. ▌
    Alef Generated Bindings · xberg-io
    Alef-managed generated bindings in packages/* and binding crates — the regeneration workflow (task alef:generate / alef:verify), the alef.toml section layout, the core-side edits that break a regen, and the FFI bridge's JSON marshalling requirement. Load before editing anything under packages/* or a binding crate, before adding a trait method or extractor, or when regenerating or verifying Alef output.
    9.6k repo stars
  95. ▌
    Mime Detection Routing · xberg-io
    MIME type detection and extractor routing in core/mime.rs — the FORMATS registry that EXT_TO_MIME and SUPPORTED_MIME_TYPES are derived from, the path-based and bytes-based detection functions, priority-based registry selection, wildcard MIME families, and the real procedure for adding a format. Load when adding a format, wiring an extractor to a MIME type, or debugging why a file routes to the wrong (or no) extractor.
    9.6k repo stars
  96. ▌
    Chunking Embeddings · xberg-io
    Chunking, embeddings, and RAG pipeline integration
    9.6k repo stars
  97. ▌
    Feature Flag Policy · xberg-io
    Cargo feature flags for crates/xberg — ORT-incompatible targets (WASM, Android x86_64 emulator), type-only and tract inference companion features, WASM/Android-safe variants, PDF backend, mutually-exclusive ORT variants, platform-conditional deps, aggregate feature sets, and build profiles. Load when adding, wiring, or debugging a Cargo feature, or when reasoning about what compiles on WASM/Android/Windows/macOS-intel targets.
    9.6k repo stars
  98. ▌
    Polyrepo Boundaries · xberg-io
    Decide which xberg-io repository owns a cross-repository fix or API, and coordinate compatible changes across Xberg, Alef, enterprise, crawler, LLM, and OCR repositories. Load when work spans sibling repos; do not use for a self-contained Xberg edit.
    9.6k repo stars
  99. ▌
    Benchmark Workflow · xberg-io
    Run, diagnose, or change Xberg extraction benchmarks, quality scoring, benchmark fixtures, artifact contracts, and independently sourced ground truth. Load for the Benchmarks workflow or benchmark-harness work, not ordinary unit tests.
    9.6k repo stars
  100. ▌
    Release Versioning · xberg-io
    How xberg versions are synced and released — Cargo.toml is the single source of truth, `task version:sync` propagates it to alef-managed binding manifests AND the integrations under integrations/, which are versioned and published in lockstep with core (including -rc.N). Load before bumping a version, editing the version-sync task, or touching an integration's version/xberg dependency.
    9.6k repo stars