abdul977 Website Cloner
A repeatable method to mirror a live website into a self-contained local copy that renders offline — verified page-by-page in a real browser until zero local assets are missing. Works for static sites, single-page apps (React/Vue/Angular/webpack builds), and multi-page sites.
Use it two ways:
- Faithful offline mirror — an exact, working copy of the site (for archiving, a reference, or a base to edit).
- Rebuild-better starting point — clone first to capture every asset and the exact design, then improve it (see
references/rebuild-better.md).
Read
references/legal-and-ethics.mdbefore cloning anything you don't own. Cloning a site you have no rights to can infringe copyright/trademark. For client work, get written authorization first — usetemplates/client-intake.mdandtemplates/authorization-to-clone.md.
When to use
Trigger on requests like: "clone this website", "copy this site", "mirror utiva.io offline", "replicate this competitor's landing page", "download the whole site so it works offline", "recreate this design but better", "my client wants a copy of X."
If the user wants a brand-new design (not based on an existing live site), this skill is the wrong tool — build from scratch instead.
The method (overview)
0. SCOPE → confirm target URL, rights, and what "clone" means for this job
1. PROBE → is the site reachable? static or SPA? which domain serves the build?
2. FETCH SHELL→ download the entry HTML
3. MAP ASSETS → find every CSS/JS/font/image the HTML references
4. DOWNLOAD → pull all direct assets, preserving /static/... paths
5. CHUNKS → for SPAs, parse the webpack/bundler chunk map, pull every chunk
6. MEDIA SWEEP→ extract every media ref from ALL js/css, download what's missing
7. SERVE → run a local server (SPA-fallback for client-routed sites)
8. VERIFY → load it in a browser; collect 404s; download stragglers; repeat until 0
9. ALL PAGES → extract the route table; crawl each static page; sweep its media
10. DELIVER → clean temp files, write handover doc, tell the user how to run it
Each step below is concrete. scripts/ contains ready-to-run helpers so you don't reinvent them.
Step 0 — Scope & rights
Before touching anything, pin down:
- Exact target URL (and whether they want the whole site or specific pages).
- Do they have the right to clone it? If it's not their own site, stop and get authorization (
templates/authorization-to-clone.md). If it's a client asking you to copy someone else's site, that's a red flag — seereferences/legal-and-ethics.md. - What "clone" means here: faithful offline mirror, or clone-to-rebuild-better? This changes the deliverable.
- Where to put it: default to a new folder like
<sitename>-live/in the working directory.
If cloning for a client, send them templates/client-intake.md first — it collects everything you'll need (URLs, credentials, brand assets, hosting, legal sign-off, definition of "better").
Step 1 — Probe the target
Find the working domain and detect the site type.
# Which domain(s) resolve? Try www + apex, .io/.com/.co variants.
for url in "https://www.SITE.com" "https://SITE.com"; do
code=$(curl -s -o /dev/null -w "%{http_code}" -L --max-time 15 "$url")
echo "$code $url"
done
Detect static vs SPA: fetch the HTML and look at <body>.
- SPA (React/Vue/Angular): body is basically
<div id="root"></div>or<div id="app"></div>with<script src="/static/js/....chunk.js">. The content is rendered by JS. → Do the full chunk + media sweep (steps 5–6). - Static / SSR: body already contains the real HTML content. → Simpler; steps 5 mostly N/A, but still sweep CSS/inline media.
curl -s -L "https://www.SITE.com/" -o probe.html
grep -oE '<div id="(root|app)">' probe.html # non-empty match ⇒ likely SPA
grep -oE 'static/js/[a-z0-9.]+\.js' probe.html | head
grep -oiE 'wp-content|wp-includes|generator" content="WordPress' probe.html | head # ⇒ WordPress
grep -oE 'framerusercontent\.com|framerstatic\.com' probe.html | head # ⇒ Framer
If multiple domains resolve, pick the one whose build hashes match what you need (compare main.<hash>.chunk.js across domains) — see scripts/probe.sh.
Expired TLS cert?
curlreturns000withcertificate has expiredon many small/older sites (very common on WordPress). It's not a dead site — add-kto every curl (and an unverified SSL context in Python). Ifcurl -skreturns 200 but plaincurlreturns 000, the cert is the only issue.WordPress detected? It's a multi-page/SSR mirror (no chunk map — skip step 5), but it has specific styling traps that only surface offline (absolute URLs,
.phppseudo-stylesheets, and Google Fonts). Readreferences/wordpress-sites.md— it's the most common case and the one most likely to look fine online yet break offline.Framer detected? Don't use the steps below — Framer has no chunk map (step 5 finds nothing) and its traps mostly yield a completely blank page, not a missing asset. Read
references/framer-sites.mdand runscripts/clone-framer.py, which handles all of it. Two things bite immediately: the CDN truncates uncompressed responses (always--compressed), andpython -m http.servercannot serve the site at all (wrong MIME for.mjs).
Step 2 — Fetch the entry HTML
OUT="SITE-live"; BASE="https://www.SITE.com"
rm -rf "$OUT" && mkdir -p "$OUT"
curl -s -L --max-time 30 "$BASE/" -o "$OUT/index.html"
Step 3 — Map the directly-referenced assets
Pull every root-relative src=/href= path out of the HTML.
cd "$OUT"
grep -oE '(src|href)="/[^"]+"' index.html \
| sed -E 's/^(src|href)="//; s/"$//' \
| grep -vE '^//' | sort -u > _assets.txt
cat _assets.txt # css, js, manifest.json, favicon, icons, etc.
Step 4 — Download direct assets (preserve paths)
while read -r path; do
[ -z "$path" ] && continue
mkdir -p ".$(dirname "$path")"
code=$(curl -s -L --max-time 30 -w "%{http_code}" "$BASE$path" -o ".$path")
echo "$code $path"
done < _assets.txt
All should be 200. A 404 here means the HTML references something the server doesn't have (rare) — note it and move on.
Step 5 — SPA chunk map (React/webpack/Vite builds)
SPAs lazy-load numbered chunks. The webpack runtime embeds a chunk-id → hash table in the HTML (or main bundle). Parse it and download every JS chunk + every non-empty CSS chunk. The empty-CSS hash is 31d6cfe0 (skip those).
Use scripts/extract-chunks.js (robust) — it finds the {0:"hash",1:"hash",...} blocks and emits the full chunk list. Then:
node ../scripts/extract-chunks.js index.html > _chunks.txt
# download all, skipping ones you already have
while read -r path; do
[ -z "$path" ] && continue; [ -s ".$path" ] && continue
curl -s -L --max-time 45 "$BASE$path" -o ".$path" &
done < _chunks.txt; wait
Downloading all chunks (even tiny ones) guarantees every route works, not just the homepage.
Vite/Rollup builds use assets/*.js with a manifest — see references/spa-builds.md for per-bundler variations.
Step 6 — Media sweep (the critical step)
Images/fonts are referenced two ways: in CSS via url(...), and inside the JS as split/concatenated strings the app builds at runtime. A regex over the downloaded JS + CSS catches them all.
grep -ohE 'static/media/[A-Za-z0-9_.-]+\.(svg|png|jpg|jpeg|webp|gif|ttf|woff2?|eot|mp4|ico)' \
static/js/*.js static/css/*.css | sort -u | sed 's#^#/#' > _allmedia.txt
echo "distinct media refs: $(wc -l < _allmedia.txt)"
# download whatever we don't already have, in parallel batches
count=0
while read -r path; do
[ -z "$path" ] && continue; [ -s ".$path" ] && continue
mkdir -p ".$(dirname "$path")"
curl -s -L --max-time 40 "$BASE$path" -o ".$path" &
count=$((count+1)); [ $((count%20)) -eq 0 ] && wait
done < _allmedia.txt; wait
scripts/media-sweep.sh wraps this with retry + 404-body detection. Long downloads time out foreground calls — run big sweeps with run_in_background: true, or batch them (the script does both).
Adjust the media path prefix (static/media/) to whatever the target uses (assets/, images/, _next/static/, a CDN path, etc.). See references/spa-builds.md.
Step 7 — Serve it locally
For a plain static site, python -m http.server is enough. For a client-routed SPA, deep links (/about-us) must return index.html, not 404 — use the SPA-fallback server:
python ../scripts/spa_server.py 8080 # from inside SITE-live/
scripts/spa_server.py serves real files when they exist and falls back to index.html for any extension-less path.
Step 8 — Verify in a browser, sweep stragglers, repeat
This is what makes the clone actually work. Load the served page in a real browser (Playwright MCP: browser_navigate then browser_console_messages / browser_network_requests) and collect failures.
- Load
http://127.0.0.1:8080/(bare/, not/index.html— SPA routers 404 on/index.html). - Get the local 404s:
browser_network_requestsfiltered to127.0.0.1.*404. - Those are assets the JS requested at runtime that your static regex missed. Download that exact list.
- Reload. Repeat until 0 local 404s.
Distinguish local 404s (must fix — download them) from external 404/errors (harmless — third-party APIs, Facebook/LinkedIn pixels, HubSpot, the site's own backend API). External calls can't work offline and are expected. See references/troubleshooting.md.
Step 9 — Capture every page (multi-page + SPA routes)
SPA: the routes live in the JS. Extract them:
grep -ohE 'path:"[^"]*"' static/js/*.js | sort -u
This yields the full route table. Classify:
- Static content pages (
/about-us,/pricing,/contact) → all share the one shell + assets; the media sweep in step 6 (over all chunks) already grabbed their images. Crawl a few in the browser (step 8) to confirm 0 local 404s. - Param routes (
/blog/:slug,/product/:id) →:slugneeds the live backend/CMS. Cannot be mirrored statically — the frame loads, individual content doesn't. Tell the user. - Auth/dashboard (
/login,/dashboard/*) → forms render; auth needs the backend.
Multi-page / static site: discover pages from the sitemap and internal links:
curl -s "$BASE/sitemap.xml" | grep -oE '<loc>[^<]+</loc>' | sed 's/<[^>]*>//g'
Download each page's HTML to path/index.html, then run the media sweep across all of them. See references/multipage-sites.md.
Step 9.5 — Make it truly self-contained (offline styling)
Being online hides broken styling. Two things routinely make a clone look perfect in your browser yet break when the client opens it offline or by double-clicking — most often on WordPress (see references/wordpress-sites.md), but the fonts issue is universal:
- Absolute / root-relative paths → the site only works behind a server; a
file://double-click resolves/wp-content/...to the drive root and all CSS disappears. Convert to page-relative:python ../scripts/relativize-paths.py "$OUT" # also renames .php pseudo-stylesheets → .css/.js - Google Fonts loaded from
fonts.googleapis.com(Poppins, Roboto, Montserrat…) live on Google's CDN, so an asset sweep never grabs them. The clone looks right while online and falls back to Times/Arial offline — the classic "the fonts/styling look wrong." Bundle them locally:python ../scripts/localize-google-fonts.py "$OUT" # downloads woff2 + repoints every page
Then run the true offline test — don't trust an online screenshot:
- Serve, load
/, and in the console confirm zero external font requests:await document.fonts.ready; performance.getEntriesByType('resource').filter(r=>/googleapis|gstatic/.test(r.name))→ want[]. - Double-click
index.html(realfile://) and confirm it's fully styled. - Diff full-page screenshots against the live site — but remember rotating sliders/carousels show different frames each shot (timing, not a bug); confirm the same asset set is present via a background-image /
<img>scan before calling anything "missing."
Step 10 — Clean up & deliver
rm -f _assets.txt _allmedia.txt _chunks.txt _need*.txt probe.html # temp files
du -sh . # report final size
Then write the project handover using templates/handover.md (what's included, what can't be mirrored and why, how to run it, how to deploy). Give the user the exact run command and the "use / not /index.html" caveat.
Quick reference — the whole thing, generically
SITE=example.com; BASE="https://www.$SITE"; OUT="${SITE%%.*}-live"
bash scripts/probe.sh "$SITE" # step 1
mkdir -p "$OUT" && cd "$OUT"
curl -s -L "$BASE/" -o index.html # step 2
bash ../scripts/download-direct.sh "$BASE" # steps 3-4
node ../scripts/extract-chunks.js index.html > _chunks.txt && bash ../scripts/download-list.sh "$BASE" _chunks.txt # step 5
bash ../scripts/media-sweep.sh "$BASE" # step 6
python ../scripts/relativize-paths.py . # step 9.5 (offline-safe paths + .php→.css)
python ../scripts/localize-google-fonts.py . # step 9.5 (bundle Google Fonts)
python ../scripts/spa_server.py 8080 # step 7
# → browser-verify (step 8), crawl routes (step 9), true offline test (9.5), deliver (10)
WordPress (real HTML per page, no chunk map): skip step 5; discover pages from /wp-sitemap.xml; use -k for the usual expired cert; then run the two step-9.5 scripts — they're what make the styling survive offline. Full playbook: references/wordpress-sites.md.
Framer (SSR'd HTML + ES modules, no chunk map): the steps above don't apply — one script does the whole job, then verify statically and in a browser. Full playbook: references/framer-sites.md.
python scripts/clone-framer.py SITE.com site-live # pages+assets+CMS+lazy chunks+rewrite
python scripts/verify-clone.py site-live # want 0 missing, 0 CDN refs
python scripts/framer-server.py 8080 site-live # NOT python -m http.server
Files in this skill
scripts/— probe, download, chunk-extract, media-sweep, and SPA server (all reusable).scripts/clone-framer.py— whole-job Framer cloner: sitemap pages, recursive assets, runtime CMS chunks, lazy backtick-import chunks, type-aware URL rewriting, telemetry/editor stripping.scripts/framer-server.py— server for ESM clones; sendstext/javascriptfor.mjsand for versionedName.js@0.0.57filenames, whichhttp.servergets wrong (blank page).scripts/verify-clone.py— static self-containment gate for any clone: every local ref resolves, nothing still cites a CDN. Non-zero exit on failure.scripts/relativize-paths.py— absolute/root-relative → page-relative so the clone works viafile://double-click and a server; also renames.phppseudo-stylesheets to.css/.js.scripts/localize-google-fonts.py— download Google Fonts (Poppins etc.) + repoint pages so typography is correct offline.references/legal-and-ethics.md— when cloning is/ isn't OK; client red flags.references/spa-builds.md— webpack vs Vite vs Next.js vs Angular asset layouts.references/multipage-sites.md— sitemap crawl for non-SPA sites.references/wordpress-sites.md— WordPress playbook (Elementor/Pagelayer/Divi): expired-cert-k, wp-sitemap discovery, absolute-URL &.php-asset traps, Google-Fonts localization, true offline test.references/troubleshooting.md— blank page, wrong route, timeouts, CORS, fonts, external 404s.references/rebuild-better.md— turning a clone into an improved rebuild.references/pricing.md— how to price a clone/rebuild job in Naira: rate card, 3 tiers, phased/batch delivery, payment terms.templates/client-intake.md— send this to a client to collect all requirements.templates/authorization-to-clone.md— written permission / sign-off form.templates/pricing-quote.md— client-facing proposal: 3 packages + phased option, ready to send.templates/handover.md— delivery doc for the finished clone.
Pricing a job for a client
When the user wants to quote a client, read references/pricing.md (the logic, anchored to the provider's own rate card) and send the client templates/pricing-quote.md (polished, 3 tiers + phased delivery). Always clarify which tier the client actually wants before quoting — "a site like [big site]" can mean anything from ₦350k (static look-alike) to ₦25M+ (full platform).