Duckduckgo Search

DuckDuckGo Search via Direct HTTP (Rust)

Integrate DuckDuckGo search into a Rust application by calling HTTP endpoints directly, using primp for browser fingerprint simulation to bypass anti-bot measures.

Why direct HTTP with primp?

• No external CLI dependencies — Pure Rust, no Python or ddgs CLI required
• Browser fingerprint simulation — primp mimics real Chrome/Firefox/Safari TLS and HTTP/2 signatures
• Full control — You own the request logic, retry strategy, and error handling
• Performance — Single binary, no process spawning overhead

Architecture

┌─────────────┐     ┌─────────────────────┐     ┌──────────────────┐
|  User query |────▶|  primp::Client      |────▶|  Bing / DDG API  |
└─────────────┘     |  (with impersonate) |     └──────────────────┘
                    └─────────────────────┘              │
                              │                          ▼
                              ▼                   ┌──────────────┐
                        ┌──────────┐              |  HTML / JSON |
                        |  Parse   │◀─────────────└──────────────┘
                        |  results |
                        └──────────┘

Text search uses Bing HTML (GET https://www.bing.com/search). News / Videos / Images use DuckDuckGo JSON APIs (GET https://duckduckgo.com/*.js) with a vqd token. Content extraction uses r.jina.ai or DuckDuckGo extract.

Prerequisites

Add to Cargo.toml:

[dependencies]
primp = "1.2"
reqwest = { version = "0.12", features = ["json", "cookies"] }
scraper = "0.22"
regex = "1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
urlencoding = "2"
chrono = "0.4"

Core implementation

1. Create the HTTP client

use primp::ClientBuilder;
use regex::Regex;

struct DdgClient {
    client: primp::Client,
    vqd_regex: Regex,
}

impl DdgClient {
    fn new() -> Result<Self, String> {
        let client = ClientBuilder::new()
            .impersonate(primp::Impersonate::Chrome)
            .timeout(Duration::from_secs(30))
            .connect_timeout(Duration::from_secs(15))
            .build()
            .map_err(|e| e.to_string())?;

        let vqd_regex = Regex::new(r#"vqd=["']?([\d-]+)["']?"#).unwrap();
        Ok(DdgClient { client, vqd_regex })
    }
}

Impersonate::Chrome (or Random) sets the TLS JA3 fingerprint, ALPN, HTTP/2 settings, and headers to match a real Chrome browser.

2. Obtain vqd token (required for DDG JSON APIs)

async fn get_vqd(&self, query: &str) -> Result<String, String> {
    let url = format!("https://duckduckgo.com?q={}", urlencoding::encode(query));
    let resp = self.client.get(&url).send().await
        .map_err(|e| format!("vqd request failed: {e}"))?;

    let body = resp.bytes().await.map_err(|e| e.to_string())?;

    if let Some(m) = self.vqd_regex.find(&String::from_utf8_lossy(&body)) {
        let matched = m.as_str();
        if let Some(val) = matched.strip_prefix("vqd=\"") {
            return Ok(val.trim_end_matches('"').to_string());
        }
        if let Some(val) = matched.strip_prefix("vqd=") {
            return Ok(val.to_string());
        }
    }

    Err("Could not extract vqd token".to_string())
}

3. Text search via Bing

Bing is more accessible than DuckDuckGo HTML in many network environments.

async fn text_search(&self, query: &str, max_results: usize) -> Result<Vec<SearchResult>, String> {
    let cookie_cd = "m=en-us&u=en-us";
    let cookie_s = "mkt=en-us&ui=en-us";

    let resp = self.client
        .get("https://www.bing.com/search")
        .header("Cookie", format!("_EDGE_CD={}; _EDGE_S={}", cookie_cd, cookie_s))
        .query(&[("q", query), ("pq", query), ("cc", "en")])
        .send()
        .await
        .map_err(|e| format!("Bing search failed: {e}"))?;

    let html = resp.text().await.map_err(|e| e.to_string())?;
    let document = Html::parse_document(&html);
    let item_selector = Selector::parse("li.b_algo").unwrap();

    let mut results = Vec::new();
    for item in document.select(&item_selector) {
        let href = item.select(&Selector::parse("h2 a").unwrap())
            .next()
            .and_then(|a| a.value().attr("href"))
            .map(|s| s.to_string());

        let title = item.select(&Selector::parse("h2 a").unwrap())
            .next()
            .map(|a| a.text().collect::<String>().trim().to_string())
            .unwrap_or_default();

        let body = item.select(&Selector::parse("p").unwrap())
            .next()
            .map(|p| p.text().collect::<String>().trim().to_string())
            .unwrap_or_default();

        let href = match href {
            Some(h) if h.starts_with("https://www.bing.com/ck/a?") => {
                unwrap_bing_url(&h).unwrap_or(h)
            }
            Some(h) => h,
            None => continue,
        };

        results.push(SearchResult { title, href, body });
        if results.len() >= max_results { break; }
    }

    Ok(results)
}

Bing URL unwrapping:

Bing wraps URLs in https://www.bing.com/ck/a?u=a1.... Decode with Base64 URL-safe:

fn unwrap_bing_url(raw_url: &str) -> Option<String> {
    let parsed = url::Url::parse(raw_url).ok()?;
    let u = parsed.query_pairs().find(|(k, _)| k == "u")?.1;
    if u.len() <= 2 { return None; }
    let b64_part = &u[2..];
    let padding = "=".repeat((-(b64_part.len() as i32)) as usize % 4);
    let decoded = base64::Engine::decode(
        &base64::engine::general_purpose::URL_SAFE,
        format!("{}{}", b64_part, padding),
    ).ok()?;
    String::from_utf8(decoded).ok()
}

4. DuckDuckGo JSON APIs (news / videos / images)

All require the vqd token obtained in step 2.

News:

GET https://duckduckgo.com/news.js
Params: l, o=json, noamp=1, q, vqd, p, df, s

Videos:

GET https://duckduckgo.com/v.js
Params: l, o=json, q, vqd, p, f, s

Images:

GET https://duckduckgo.com/i.js
Params: o=json, q, l, vqd, p, ct=AT, f, s

Common pattern:

async fn news_search(&self, query: &str, max_results: usize) -> Result<Vec<NewsResult>, String> {
    let vqd = self.get_vqd(query).await?;
    let mut params: Vec<(&str, String)> = vec![
        ("l", "us-en".to_string()),
        ("o", "json".to_string()),
        ("noamp", "1".to_string()),
        ("q", query.to_string()),
        ("vqd", vqd),
        ("p", "1".to_string()),
    ];

    let mut results = Vec::new();
    for _ in 0..5 {
        let resp = self.client
            .get("https://duckduckgo.com/news.js")
            .header("Referer", "https://duckduckgo.com/")
            .query(&params)
            .send()
            .await?;

        let json: serde_json::Value = resp.json().await?;
        let items = json.get("results").and_then(|r| r.as_array()).cloned().unwrap_or_default();

        for item in items {
            results.push(NewsResult {
                date: normalize_date(item.get("date")),
                title: item.get("title").and_then(|v| v.as_str()).unwrap_or("").to_string(),
                body: item.get("excerpt").and_then(|v| v.as_str()).unwrap_or("").to_string(),
                url: item.get("url").and_then(|v| v.as_str()).unwrap_or("").to_string(),
                image: item.get("image").and_then(|v| v.as_str()).map(|s| s.to_string()),
                source: item.get("source").and_then(|v| v.as_str()).unwrap_or("").to_string(),
            });
            if results.len() >= max_results { return Ok(results); }
        }

        // Pagination
        if let Some(next) = json.get("next").and_then(|v| v.as_str()) {
            if let Some(s_val) = extract_query_param(next, "s") {
                params.retain(|(k, _)| *k != "s");
                params.push(("s", s_val));
            } else { break; }
        } else { break; }
    }

    Ok(results)
}

5. Web content extraction

Use r.jina.ai as a lightweight extraction service:

async fn extract(&self, url: &str) -> Result<ExtractResult, String> {
    let resp = self.client
        .get(format!("https://r.jina.ai/http://{}", url.trim_start_matches("https://").trim_start_matches("http://")))
        .send()
        .await
        .map_err(|e| format!("extract failed: {e}"))?;

    let text = resp.text().await.map_err(|e| e.to_string())?;
    Ok(ExtractResult { url: url.to_string(), content: text })
}

Network environment considerations

If DuckDuckGo is blocked:

• Text search still works via Bing
• Images/videos/news require vqd token (unavailable if duckduckgo.com is blocked)
• Consider SearXNG instances as additional fallback

Anti-bot summary

Without primp, plain reqwest will be blocked by Bing/DuckDuckGo anti-bot systems.

Error handling

References

• references/duckduckgo-http-api.md — Complete DuckDuckGo JSON/HTTP endpoint documentation (images, videos, news, suggestions, instant answers)
• references/bing-text-search.md — Bing HTML parsing, URL decoding, pagination, and cookie requirements