Refurbish Demos

Refurbish Demos

Upgrade demo agent knowledge from old AI-written summaries to properly scraped page content with real sourceUrls. Each knowledge item becomes a URL-type entry with the actual page URL, so the chatbot can cite specific pages in its answers.

Parameter: campaignId (CRM campaign ID) or all-sent (all demos where outreach was sent)

Announce:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Refurbish Demos — upgrading knowledge
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Environment

Read from .env in the working directory:

• NUXT_MCP_DEMO_TOKEN — Bearer token for InboxMate MCP (contains # character — use curl, not Python urllib)
• PSQUARED_CRM_TOKEN — Bearer token for Twenty CRM
• EMAIL_DRAFT_ONLY_BEARER — Bearer token for notification service (used in Option B: all-sent)

MCP endpoint: https://app.psquared.dev/api/mcp CRM endpoint: https://crm.psquared.dev/graphql Demo API: https://app.psquared.dev/api/demo/{demoId} Supabase project: fevtfywriufbqnvbgyrm (for direct DB queries when needed) Demo account ID: 8942a6e5-91cb-4c5d-8ef5-98cfe7945620

MCP Call Template

All MCP calls use this pattern (use curl, NOT Python urllib — the token contains #):

curl -s --max-time 120 -X POST https://app.psquared.dev/api/mcp \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $NUXT_MCP_DEMO_TOKEN" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"TOOL_NAME","arguments":{...}}}'

Available MCP Tools

PHASE 1 — Build Work List

Announce: [1/4] Building work list...

Option A: By campaign ID

Query CRM for opportunities in the campaign:

curl -s -X POST https://crm.psquared.dev/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $PSQUARED_CRM_TOKEN" \
  -d "{\"query\":\"{ opportunities(filter: { campaignId: { eq: \\\"CAMPAIGN_ID\\\" } }, first: 150) { edges { node { id name demoStatus demoUrl { primaryLinkUrl } company { id name domainName { primaryLinkUrl } } } } } }\"}"

Option B: All sent demos

Get sent email drafts from notification service, extract demo IDs, then look up agents:

curl -s "https://notifications.psquared.dev/drafts?status=SENT&draftType=outreach&pageSize=200" \
  -H "Authorization: Bearer $EMAIL_DRAFT_ONLY_BEARER"

Extract demoUrl from each draft's variables, parse the ?id= param to get demoId.

For each demo ID, fetch agent info:

curl -s https://app.psquared.dev/api/demo/DEMO_ID
# Returns: { agentId, companyName, companyDomain, ... }

Then get bucket ID:

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_agent","arguments":{"agentId":"AGENT_ID"}}}

Extract knowledgeBucketIds[0].

Filter out: demoStatus = SKIP_*, DISQUALIFIED. Skip demos without companyDomain.

Build: [ { companyName, companyDomain, agentId, bucketId, demoId } ]

Announce: Found N demos to refurbish.

PHASE 2 — Discover Real Pages Per Company

Announce: [2/4] Discovering pages to scrape...

For each company, discover which pages actually exist before scraping. Never guess URLs blindly — generic patterns like /kontakt/, /ueber-uns/, /leistungen/ fail on 60%+ of sites (e-commerce, single-page, non-standard CMS). This produces "Seite nicht gefunden" knowledge items that pollute RAG.

Step 2a: WebFetch the homepage

WebFetch https://www.{domain}/ (or https://{domain}/ if www fails). Extract:

• Navigation links (main nav, header menu)
• Footer links (about, contact, FAQ, legal)
• Any prominent section links

Step 2b: Select 6-8 best URLs

From the discovered links, pick URLs that give the chatbot useful knowledge:

Priority order:

1. Homepage (always)
2. About / Über uns / Unternehmen (who they are)
3. Services / Leistungen / Produkte (what they offer)
4. Contact / Kontakt / Standorte / Filialen (how to reach them)
5. FAQ / Service / Hilfe (common questions)
6. Team / Karriere (if relevant)
7. Impressum (legal, low priority but useful for address/contact)
8. Key category/product pages (for e-commerce sites)

Rules:

• Only include URLs on the same domain
• Skip: blog posts, privacy policy, AGB, login pages, PDF links, anchor-only links
• Skip junk URLs: /wp-json/, /xmlrpc.php, /feed/, /favicon.ico, apple-touch-icon, .webp, .ico, .css, .js
• For e-commerce sites: pick category overview pages, brand pages, store locator — NOT individual product pages
• For JS-rendered pages (store locators, maps): if Tavily returns empty, WebFetch manually
• Max 8 URLs total (keeps scrape time reasonable)

Step 2c: Split into batches of 2-3 URLs

Tavily advanced mode takes ~5-30s per URL. A single call with 5+ URLs hits server timeouts (120s). Always split into batches of max 3 URLs per MCP call.

Fallback: If WebFetch fails

If the homepage can't be fetched (timeout, blocking), fall back to these common patterns but expect failures and don't count on them:

https://www.{domain}/
https://www.{domain}/kontakt/
https://www.{domain}/ueber-uns/
https://www.{domain}/impressum/

PHASE 3 — Clear & Rebuild Knowledge

Announce: [3/4] Rebuilding knowledge for N demos...

Process one company at a time:

Step 3a: Wipe existing knowledge

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"cleanup_agent","arguments":{"agentId":"AGENT_ID"}}}

This clears the agent's buckets AND deletes all orphaned buckets in the demo account (from old demo creation flow). First call handles the global orphan cleanup.

Step 3b: Scrape via MCP in batches of 3

Send discovered URLs to scrape_and_build_knowledge in batches of max 3:

{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"scrape_and_build_knowledge","arguments":{"bucketId":"BUCKET_ID","urls":["URL1","URL2","URL3"]}}}

The MCP tool automatically:

• Rejects 404/"Seite nicht gefunden" pages (returns them in failed array with reason 404/not-found page detected)
• Sets sourceUrl on created items (enables chat citations)
• Supports content up to 50K characters

Check the response created and failed arrays after each batch.

Step 3c: Handle failures — WebFetch fallback

For each failed URL, check the failure reason:

WebFetch fallback for timeouts/scrape failures:

1. WebFetch the URL with a prompt to extract all content
2. Clean the content: remove nav, footer, cookie banners, base64 images
3. Keep: headings, lists, structure, ALL details (names, prices, hours, addresses)
4. Call add_to_bucket with cleaned content + sourceUrl:

{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"add_to_bucket","arguments":{"bucketId":"BUCKET_ID","title":"Page Title","content":"[full cleaned page text]","sourceUrl":"https://domain.de/page/"}}}

Step 3d: If ALL URLs fail (site blocks Tavily entirely)

Some sites block Tavily's scraper. If every URL times out or fails:

1. WebFetch ALL discovered URLs yourself — WebFetch uses a different scraper that often works when Tavily doesn't
2. For each page, extract detailed content (not summaries)
3. Add each via add_to_bucket with sourceUrl
4. You MUST NOT leave the bucket empty — an empty bucket means the chatbot can't answer anything

Content rules for manual WebFetch entries:

• Full page text, not an AI summary — keep the original wording
• Preserve headings, lists, and structure
• Remove nav, footer, cookie banners, boilerplate
• Keep ALL specifics: names, services, prices, hours, addresses, team members
• Max 50,000 characters per entry
• sourceUrl MUST be the actual page URL

Step 3e: Verify minimum quality

After all scraping + fallbacks, check:

• At least 3 items in the bucket (< 3 = chatbot is too thin)
• No items with "nicht gefunden" / "not found" / "404" in the title (shouldn't happen with MCP detection, but verify)
• No junk items (wp-json, xmlrpc, feed, favicon — delete if present)

Step 3f: Fix button icon if broken

Valid icons: messageCircle, messageSquare, sparkles, support, help, inboxmate, heart, zap, globe, wave, brain, lightbulb, compass, star, shield, robot, mascot

Any other value (e.g. shoppingBag, truck, home, car, music) renders as a broken circle in the widget. Fix to messageCircle:

{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"update_widget_style","arguments":{"agentId":"AGENT_ID","buttonIcon":"messageCircle"}}}

Step 3g: Republish agent

{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"publish_agent","arguments":{"agentId":"AGENT_ID"}}}

Log per company: ✓ {companyName}: {N} items created, {M} failed → republished

PHASE 4 — Report

Announce: [4/4] Refurbish complete.

Print summary table:

| Company | Domain | Items | Failed | Icon Fixed | Status |
|---------|--------|-------|--------|------------|--------|
| Schuh Marke GmbH | schuh-marke.de | 5 | 0 | yes | ✓ |
| Mainfilm | mainfilm.tv | 5 | 0 | no | ✓ |

Totals:

• Companies processed: N
• Knowledge items created: N
• Failed URLs: N (fell back to WebFetch: N, completely failed: N)
• Icons fixed: N
• Tavily credits used: ~N (2 credits per 5 URLs at advanced mode)

Batch Script

For large batches (100+ demos), use the Python batch script instead of processing one-by-one from Claude:

python3 -u scripts/refurbish-all.py < /tmp/worklist.json > /tmp/refurbish.log 2>&1

The script at claude-overlord-folder/scripts/refurbish-all.py:

• Discovers real URLs from homepage nav (not hardcoded patterns)
• Filters junk URLs (wp-json, xmlrpc, feed, favicon, etc.)
• Handles www. prefix correctly (no www.www. duplication)
• Splits URLs into batches of 3 to avoid server timeouts
• Uses curl (not urllib) for MCP calls — handles # in auth token
• Processes sequentially with 1s pause every 5 agents
• Input: JSON array of [{company_name, company_domain, agent_id, bucket_id}]

Limitation: The script does NOT do WebFetch fallback — it only uses MCP scrape_and_build_knowledge (Tavily). If Tavily fails for a site, the agent will be left with fewer items. After a batch run, check the log for agents with 0-2 items created and fix those manually with WebFetch.

Known Issues & Workarounds

Tavily advanced mode timeouts

Tavily extract_depth: 'advanced' takes up to 30s per URL. Batch 5 URLs in one MCP call = 150s, which exceeds server timeout. Always split into batches of 2-3 URLs.

Fallback API key

If primary Tavily key runs out, set NUXT_TAVILY_FALLBACK_API_KEY in agenthub .env. Auto-switches on 429/402 errors.

"Seite nicht gefunden" pages

If 404 pages end up in knowledge (from fallback URL guessing or site changes), they DO hurt RAG — the chatbot may cite non-existent pages or give confused answers. Phase 2 now discovers real URLs first to prevent this. If you see 404 items after a refurbish, the homepage WebFetch likely failed and fell back to guessed URLs — investigate and re-run with manual URL discovery.

Bucket name mismatches

Old demo creation flow sometimes linked wrong-named buckets to agents (e.g. "Autohaus Freier Wissensbasis" on a Hinzmann Elektrotechnik agent). The content is correct after refurbish — the bucket name is cosmetic. Can be fixed via SQL if needed.

Orphaned buckets

Old demo creation left ~160 orphaned "Wissensdatenbank" buckets not linked to any agent. cleanup_agent deletes these. For large batches, clean up via SQL first (faster):

DELETE FROM knowledge_bucket_chunks WHERE bucket_item_id IN (
  SELECT kbi.id FROM knowledge_bucket_items kbi
  JOIN knowledge_buckets kb ON kb.id = kbi.bucket_id
  WHERE kb.account_id = '8942a6e5-91cb-4c5d-8ef5-98cfe7945620'
  AND kb.id NOT IN (SELECT unnest(knowledge_bucket_ids) FROM agents WHERE account_id = '8942a6e5-91cb-4c5d-8ef5-98cfe7945620')
);
-- Then delete items, then buckets (same pattern)

Content length limit

Knowledge entries allow up to 50,000 characters (was 20K, bumped 2026-03-28). Tavily advanced mode returns full pages including nav/footer HTML — the extra headroom prevents truncation of actual content. Pages hitting the limit likely have nav pollution but the important content still fits.

JS-rendered content (store locators, maps, dynamic lists)

Some pages load data via JavaScript (store locators, interactive maps, product configurators). Tavily can't extract this content. If a page returns mostly nav/boilerplate with no real data, WebFetch it manually, extract the data from the page description or other sources, and add via add_to_bucket with sourceUrl.

Cross-contamination

Audit showed only 1 true case out of 152 sent demos (Dachdeckermeister Hoffmann had DBM Metallbau content). Refurbish fixes this by clearing and re-scraping based on the correct domain from the demo page.