Ares

Ares — LLM-Powered Web Scraper

Ares is a Rust library, CLI, and HTTP server that extracts structured data from websites using LLMs and JSON Schemas.

Repository: https://github.com/AndreaBozzo/Ares License: Apache-2.0 | Rust edition: 2024 | MSRV: 1.88+

Pipeline

URL → [ContentCache?] → Fetcher (HTML) → Cleaner (Markdown) → [ExtractionCache?] → Extractor (LLM + JSON Schema) → Validate → Hash → Compare → Store

Each stage is a trait, so every component can be swapped or mocked independently. Optional in-memory caches (moka) skip fetch/extraction when content or results are already cached.

After extraction the result is validated against the JSON Schema (validate_extracted_output); on mismatch the pipeline returns AppError::ExtractionValidationError and nothing is persisted (toggle with .with_validation(false)). A heuristic groundedness check (ungrounded_fields) then warns — without failing — when short atomic values look absent from the source (a hallucination signal schema validation can't catch). Valid JSON is not necessarily grounded truth.

Crate Map

Core Traits (ares-core::traits)

pub trait Fetcher: Send + Sync + Clone {
    fn fetch(&self, url: &str) -> impl Future<Output = Result<String, AppError>> + Send;
}

pub trait Cleaner: Send + Sync + Clone {
    fn clean(&self, html: &str) -> Result<String, AppError>;
}

pub trait Extractor: Send + Sync + Clone {
    fn extract(&self, content: &str, schema: &serde_json::Value)
        -> impl Future<Output = Result<serde_json::Value, AppError>> + Send;
}

pub trait ExtractorFactory: Send + Sync + Clone {
    type Extractor: Extractor;
    fn create(&self, model: &str, base_url: &str) -> Result<Self::Extractor, AppError>;
}

pub trait ExtractionStore: Send + Sync + Clone {
    fn save(&self, extraction: &NewExtraction) -> impl Future<Output = Result<Uuid, AppError>> + Send;
    fn get_latest(&self, url: &str, schema_name: &str) -> impl Future<Output = Result<Option<Extraction>, AppError>> + Send;
    fn get_history(&self, url: &str, schema_name: &str, limit: usize, offset: usize) -> impl Future<Output = Result<Vec<Extraction>, AppError>> + Send;
}

JobQueue trait: see ares-core::job_queue — persistent queue with atomic claiming (SELECT FOR UPDATE SKIP LOCKED).

pub trait LinkDiscoverer: Send + Sync + Clone {
    fn discover_links(&self, html: &str, base_url: &str) -> Result<Vec<String>, AppError>;
}

pub trait RobotsChecker: Send + Sync + Clone {
    // Returns `true` if the URL may be fetched. On fetch/parse errors it
    // defaults to allowing (graceful degradation) — hence plain `bool`, not Result.
    fn is_allowed(&self, url: &str) -> impl Future<Output = bool> + Send;
}

Key Types

Quick Start (Library Usage)

use ares_client::{ReqwestFetcher, HtmdCleaner, OpenAiExtractor};
use ares_core::{ScrapeService, NullStore};

let fetcher = ReqwestFetcher::new()?;
let cleaner = HtmdCleaner::new();
let extractor = OpenAiExtractor::with_base_url(&api_key, "gpt-4o-mini", "https://api.openai.com/v1")?;

let service = ScrapeService::<_, _, _, NullStore>::new(fetcher, cleaner, extractor, "gpt-4o-mini".into());

let schema = serde_json::json!({
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "author": {"type": "string"}
    },
    "required": ["title", "author"]
});

let result = service.scrape("https://example.com/blog", &schema, "blog").await?;
println!("{}", serde_json::to_string_pretty(&result.extracted_data)?);

With persistence, use ScrapeService::with_store(fetcher, cleaner, extractor, store, model).

Providers & Backends

The Extractor trait is the seam for inference backends. Three ship in-tree, selected at runtime via --provider / ARES_PROVIDER (CLI), the provider field of POST /v1/scrape (API), or Provider + ProviderExtractor/ProviderExtractorFactory dispatch enums (library):

New backends implement Extractor + ExtractorFactory; nothing else in the pipeline changes.

Reference Guides

Version Notes

• Current version: 0.4.0
• Until crates.io release, use git dependency: ares-core = { git = "https://github.com/AndreaBozzo/Ares" }
• Works with any OpenAI-compatible API (OpenAI, Gemini, local servers) out of the box; Anthropic and native local inference are feature-gated.
• Optional features: browser (headless Chrome), anthropic (native Claude), local-llm (native Candle CPU inference).
• New in 0.3.0: Provider abstraction with runtime selection (--provider/ARES_PROVIDER), native Anthropic backend, output validation (validate_extracted_output, returned as 422 over HTTP) and groundedness checks (ungrounded_fields), --max-content cap, additional schemas (public_tenders, tender_list, job_board).
• New in 0.4.0: Native local inference via Candle (local-llm feature, --provider local, ares model subcommand).
• Earlier (0.2.0): Web crawling, in-memory caching, output formats (json/jsonl/csv/table/jq), proxy rotation, User-Agent rotation, browser stealth mode, TLS backend selection.