# Website Cloner

> Clone / mirror any live website so it renders fully offline, or as a starting point to rebuild it better. Use when the user wants to copy, clone, replicate, mirror, download, or recreate an existing website (a competitor site, a client's current site, a reference design, a template) OR wants to price/quote such a job for a client. Handles static sites, SPAs (React/Vue/Angular), Framer sites, and multi-page + WordPress sites (Elementor/Pagelayer/Divi); downloads HTML + all CSS/JS/images/fonts, localizes Google Fonts, makes paths offline-safe, wires up an offline server, and verifies every page renders with zero missing assets and correct styling offline. Also produces a client intake brief, a Naira pricing model (3 tiers + phased/batch delivery), a client-facing quote, and a project handover doc.

- Skill: `abdul977/website-cloner` (Agent Skill, multi-file: 24 files)
- Install (CLI): `npx skillmds@latest add abdul977/website-cloner`
- Raw SKILL.md: https://api.skillmd.com/api/skills/abdul977/website-cloner/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: abdul977 (https://skillmd.com/u/abdul977)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/abdul977/website-cloner

---


# Website Cloner

A repeatable method to mirror a live website into a self-contained local copy that renders offline — verified page-by-page in a real browser until zero local assets are missing. Works for static sites, single-page apps (React/Vue/Angular/webpack builds), and multi-page sites.

Use it two ways:
1. **Faithful offline mirror** — an exact, working copy of the site (for archiving, a reference, or a base to edit).
2. **Rebuild-better starting point** — clone first to capture every asset and the exact design, then improve it (see `references/rebuild-better.md`).

> **Read `references/legal-and-ethics.md` before cloning anything you don't own.** Cloning a site you have no rights to can infringe copyright/trademark. For client work, get written authorization first — use `templates/client-intake.md` and `templates/authorization-to-clone.md`.

---

## When to use

Trigger on requests like: "clone this website", "copy this site", "mirror utiva.io offline", "replicate this competitor's landing page", "download the whole site so it works offline", "recreate this design but better", "my client wants a copy of X."

If the user wants a *brand-new* design (not based on an existing live site), this skill is the wrong tool — build from scratch instead.

---

## The method (overview)

```
0. SCOPE      → confirm target URL, rights, and what "clone" means for this job
1. PROBE      → is the site reachable? static or SPA? which domain serves the build?
2. FETCH SHELL→ download the entry HTML
3. MAP ASSETS → find every CSS/JS/font/image the HTML references
4. DOWNLOAD   → pull all direct assets, preserving /static/... paths
5. CHUNKS     → for SPAs, parse the webpack/bundler chunk map, pull every chunk
6. MEDIA SWEEP→ extract every media ref from ALL js/css, download what's missing
7. SERVE      → run a local server (SPA-fallback for client-routed sites)
8. VERIFY     → load it in a browser; collect 404s; download stragglers; repeat until 0
9. ALL PAGES  → extract the route table; crawl each static page; sweep its media
10. DELIVER   → clean temp files, write handover doc, tell the user how to run it
```

Each step below is concrete. `scripts/` contains ready-to-run helpers so you don't reinvent them.

---

## Step 0 — Scope & rights

Before touching anything, pin down:
- **Exact target URL** (and whether they want the whole site or specific pages).
- **Do they have the right to clone it?** If it's not their own site, stop and get authorization (`templates/authorization-to-clone.md`). If it's a client asking you to copy *someone else's* site, that's a red flag — see `references/legal-and-ethics.md`.
- **What "clone" means here:** faithful offline mirror, or clone-to-rebuild-better? This changes the deliverable.
- **Where to put it:** default to a new folder like `<sitename>-live/` in the working directory.

If cloning for a client, send them `templates/client-intake.md` first — it collects everything you'll need (URLs, credentials, brand assets, hosting, legal sign-off, definition of "better").

---

## Step 1 — Probe the target

Find the working domain and detect the site type.

```bash
# Which domain(s) resolve? Try www + apex, .io/.com/.co variants.
for url in "https://www.SITE.com" "https://SITE.com"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -L --max-time 15 "$url")
  echo "$code  $url"
done
```

**Detect static vs SPA:** fetch the HTML and look at `<body>`.
- **SPA** (React/Vue/Angular): body is basically `<div id="root"></div>` or `<div id="app"></div>` with `<script src="/static/js/....chunk.js">`. The content is rendered by JS. → Do the full chunk + media sweep (steps 5–6).
- **Static / SSR**: body already contains the real HTML content. → Simpler; steps 5 mostly N/A, but still sweep CSS/inline media.

```bash
curl -s -L "https://www.SITE.com/" -o probe.html
grep -oE '<div id="(root|app)">' probe.html   # non-empty match ⇒ likely SPA
grep -oE 'static/js/[a-z0-9.]+\.js' probe.html | head
grep -oiE 'wp-content|wp-includes|generator" content="WordPress' probe.html | head  # ⇒ WordPress
grep -oE 'framerusercontent\.com|framerstatic\.com' probe.html | head               # ⇒ Framer
```

If multiple domains resolve, **pick the one whose build hashes match** what you need (compare `main.<hash>.chunk.js` across domains) — see `scripts/probe.sh`.

> **Expired TLS cert?** `curl` returns `000` with `certificate has expired` on many small/older sites (very common on WordPress). It's not a dead site — add **`-k`** to every curl (and an unverified SSL context in Python). If `curl -sk` returns 200 but plain `curl` returns 000, the cert is the only issue.
>
> **WordPress detected?** It's a multi-page/SSR mirror (no chunk map — skip step 5), but it has specific styling traps that only surface **offline** (absolute URLs, `.php` pseudo-stylesheets, and Google Fonts). **Read `references/wordpress-sites.md`** — it's the most common case and the one most likely to look fine online yet break offline.
>
> **Framer detected?** Don't use the steps below — Framer has no chunk map (step 5 finds nothing) and its traps mostly yield a **completely blank page**, not a missing asset. **Read `references/framer-sites.md`** and run `scripts/clone-framer.py`, which handles all of it. Two things bite immediately: the CDN truncates uncompressed responses (always `--compressed`), and `python -m http.server` cannot serve the site at all (wrong MIME for `.mjs`).

---

## Step 2 — Fetch the entry HTML

```bash
OUT="SITE-live"; BASE="https://www.SITE.com"
rm -rf "$OUT" && mkdir -p "$OUT"
curl -s -L --max-time 30 "$BASE/" -o "$OUT/index.html"
```

---

## Step 3 — Map the directly-referenced assets

Pull every root-relative `src=`/`href=` path out of the HTML.

```bash
cd "$OUT"
grep -oE '(src|href)="/[^"]+"' index.html \
  | sed -E 's/^(src|href)="//; s/"$//' \
  | grep -vE '^//' | sort -u > _assets.txt
cat _assets.txt   # css, js, manifest.json, favicon, icons, etc.
```

---

## Step 4 — Download direct assets (preserve paths)

```bash
while read -r path; do
  [ -z "$path" ] && continue
  mkdir -p ".$(dirname "$path")"
  code=$(curl -s -L --max-time 30 -w "%{http_code}" "$BASE$path" -o ".$path")
  echo "$code  $path"
done < _assets.txt
```

All should be `200`. A `404` here means the HTML references something the server doesn't have (rare) — note it and move on.

---

## Step 5 — SPA chunk map (React/webpack/Vite builds)

SPAs lazy-load numbered chunks. The webpack runtime embeds a **chunk-id → hash** table in the HTML (or main bundle). Parse it and download every JS chunk + every non-empty CSS chunk. The empty-CSS hash is `31d6cfe0` (skip those).

Use `scripts/extract-chunks.js` (robust) — it finds the `{0:"hash",1:"hash",...}` blocks and emits the full chunk list. Then:

```bash
node ../scripts/extract-chunks.js index.html > _chunks.txt
# download all, skipping ones you already have
while read -r path; do
  [ -z "$path" ] && continue; [ -s ".$path" ] && continue
  curl -s -L --max-time 45 "$BASE$path" -o ".$path" &
done < _chunks.txt; wait
```

> Downloading **all** chunks (even tiny ones) guarantees every route works, not just the homepage.

Vite/Rollup builds use `assets/*.js` with a manifest — see `references/spa-builds.md` for per-bundler variations.

---

## Step 6 — Media sweep (the critical step)

Images/fonts are referenced two ways: in CSS via `url(...)`, **and inside the JS** as split/concatenated strings the app builds at runtime. A regex over the *downloaded* JS + CSS catches them all.

```bash
grep -ohE 'static/media/[A-Za-z0-9_.-]+\.(svg|png|jpg|jpeg|webp|gif|ttf|woff2?|eot|mp4|ico)' \
  static/js/*.js static/css/*.css | sort -u | sed 's#^#/#' > _allmedia.txt
echo "distinct media refs: $(wc -l < _allmedia.txt)"

# download whatever we don't already have, in parallel batches
count=0
while read -r path; do
  [ -z "$path" ] && continue; [ -s ".$path" ] && continue
  mkdir -p ".$(dirname "$path")"
  curl -s -L --max-time 40 "$BASE$path" -o ".$path" &
  count=$((count+1)); [ $((count%20)) -eq 0 ] && wait
done < _allmedia.txt; wait
```

`scripts/media-sweep.sh` wraps this with retry + 404-body detection. **Long downloads time out foreground calls** — run big sweeps with `run_in_background: true`, or batch them (the script does both).

**Adjust the media path prefix** (`static/media/`) to whatever the target uses (`assets/`, `images/`, `_next/static/`, a CDN path, etc.). See `references/spa-builds.md`.

---

## Step 7 — Serve it locally

For a plain static site, `python -m http.server` is enough. For a **client-routed SPA**, deep links (`/about-us`) must return `index.html`, not 404 — use the SPA-fallback server:

```bash
python ../scripts/spa_server.py 8080   # from inside SITE-live/
```

`scripts/spa_server.py` serves real files when they exist and falls back to `index.html` for any extension-less path.

---

## Step 8 — Verify in a browser, sweep stragglers, repeat

This is what makes the clone actually work. **Load the served page in a real browser** (Playwright MCP: `browser_navigate` then `browser_console_messages` / `browser_network_requests`) and collect failures.

- Load `http://127.0.0.1:8080/` (bare `/`, **not** `/index.html` — SPA routers 404 on `/index.html`).
- Get the **local** 404s: `browser_network_requests` filtered to `127.0.0.1.*404`.
- Those are assets the JS requested at runtime that your static regex missed. Download that exact list.
- Reload. Repeat until **0 local 404s**.

Distinguish **local 404s** (must fix — download them) from **external 404/errors** (harmless — third-party APIs, Facebook/LinkedIn pixels, HubSpot, the site's own backend API). External calls can't work offline and are expected. See `references/troubleshooting.md`.

---

## Step 9 — Capture every page (multi-page + SPA routes)

**SPA:** the routes live in the JS. Extract them:
```bash
grep -ohE 'path:"[^"]*"' static/js/*.js | sort -u
```
This yields the full route table. Classify:
- **Static content pages** (`/about-us`, `/pricing`, `/contact`) → all share the one shell + assets; the media sweep in step 6 (over *all* chunks) already grabbed their images. Crawl a few in the browser (step 8) to confirm 0 local 404s.
- **Param routes** (`/blog/:slug`, `/product/:id`) → `:slug` needs the live backend/CMS. **Cannot be mirrored statically** — the frame loads, individual content doesn't. Tell the user.
- **Auth/dashboard** (`/login`, `/dashboard/*`) → forms render; auth needs the backend.

**Multi-page / static site:** discover pages from the sitemap and internal links:
```bash
curl -s "$BASE/sitemap.xml" | grep -oE '<loc>[^<]+</loc>' | sed 's/<[^>]*>//g'
```
Download each page's HTML to `path/index.html`, then run the media sweep across all of them. See `references/multipage-sites.md`.

---

## Step 9.5 — Make it truly self-contained (offline styling)

**Being online hides broken styling.** Two things routinely make a clone look perfect in your browser yet break when the client opens it offline or by double-clicking — most often on **WordPress** (see `references/wordpress-sites.md`), but the fonts issue is universal:

1. **Absolute / root-relative paths** → the site only works behind a server; a `file://` double-click resolves `/wp-content/...` to the drive root and **all CSS disappears**. Convert to page-relative:
   ```bash
   python ../scripts/relativize-paths.py "$OUT"    # also renames .php pseudo-stylesheets → .css/.js
   ```
2. **Google Fonts loaded from `fonts.googleapis.com`** (Poppins, Roboto, Montserrat…) live on Google's CDN, so an asset sweep never grabs them. The clone looks right *while online* and falls back to Times/Arial *offline* — the classic "the fonts/styling look wrong." Bundle them locally:
   ```bash
   python ../scripts/localize-google-fonts.py "$OUT"   # downloads woff2 + repoints every page
   ```

**Then run the true offline test** — don't trust an online screenshot:
- Serve, load `/`, and in the console confirm **zero external font requests**:
  `await document.fonts.ready; performance.getEntriesByType('resource').filter(r=>/googleapis|gstatic/.test(r.name))` → want `[]`.
- **Double-click `index.html`** (real `file://`) and confirm it's fully styled.
- Diff full-page screenshots against the live site — but remember rotating **sliders/carousels** show different frames each shot (timing, not a bug); confirm the same asset set is present via a background-image / `<img>` scan before calling anything "missing."

---

## Step 10 — Clean up & deliver

```bash
rm -f _assets.txt _allmedia.txt _chunks.txt _need*.txt probe.html   # temp files
du -sh .   # report final size
```

Then write the **project handover** using `templates/handover.md` (what's included, what can't be mirrored and why, how to run it, how to deploy). Give the user the exact run command and the "use `/` not `/index.html`" caveat.

---

## Quick reference — the whole thing, generically

```bash
SITE=example.com; BASE="https://www.$SITE"; OUT="${SITE%%.*}-live"
bash scripts/probe.sh "$SITE"                       # step 1
mkdir -p "$OUT" && cd "$OUT"
curl -s -L "$BASE/" -o index.html                   # step 2
bash ../scripts/download-direct.sh "$BASE"          # steps 3-4
node ../scripts/extract-chunks.js index.html > _chunks.txt && bash ../scripts/download-list.sh "$BASE" _chunks.txt   # step 5
bash ../scripts/media-sweep.sh "$BASE"              # step 6
python ../scripts/relativize-paths.py .             # step 9.5 (offline-safe paths + .php→.css)
python ../scripts/localize-google-fonts.py .        # step 9.5 (bundle Google Fonts)
python ../scripts/spa_server.py 8080                # step 7
# → browser-verify (step 8), crawl routes (step 9), true offline test (9.5), deliver (10)
```

**WordPress** (real HTML per page, no chunk map): skip step 5; discover pages from `/wp-sitemap.xml`; use `-k` for the usual expired cert; then run the two step-9.5 scripts — they're what make the styling survive offline. Full playbook: `references/wordpress-sites.md`.

**Framer** (SSR'd HTML + ES modules, no chunk map): the steps above don't apply — one script does the whole job, then verify statically and in a browser. Full playbook: `references/framer-sites.md`.
```bash
python scripts/clone-framer.py SITE.com site-live    # pages+assets+CMS+lazy chunks+rewrite
python scripts/verify-clone.py site-live             # want 0 missing, 0 CDN refs
python scripts/framer-server.py 8080 site-live       # NOT python -m http.server
```

## Files in this skill
- `scripts/` — probe, download, chunk-extract, media-sweep, and SPA server (all reusable).
- `scripts/clone-framer.py` — **whole-job Framer cloner**: sitemap pages, recursive assets, runtime CMS chunks, lazy backtick-import chunks, type-aware URL rewriting, telemetry/editor stripping.
- `scripts/framer-server.py` — server for ESM clones; sends `text/javascript` for `.mjs` **and** for versioned `Name.js@0.0.57` filenames, which `http.server` gets wrong (blank page).
- `scripts/verify-clone.py` — static self-containment gate for **any** clone: every local ref resolves, nothing still cites a CDN. Non-zero exit on failure.
- `scripts/relativize-paths.py` — absolute/root-relative → **page-relative** so the clone works via `file://` double-click *and* a server; also renames `.php` pseudo-stylesheets to `.css/.js`.
- `scripts/localize-google-fonts.py` — download Google Fonts (Poppins etc.) + repoint pages so typography is correct **offline**.
- `references/legal-and-ethics.md` — when cloning is/ isn't OK; client red flags.
- `references/spa-builds.md` — webpack vs Vite vs Next.js vs Angular asset layouts.
- `references/multipage-sites.md` — sitemap crawl for non-SPA sites.
- `references/wordpress-sites.md` — **WordPress playbook** (Elementor/Pagelayer/Divi): expired-cert `-k`, wp-sitemap discovery, absolute-URL & `.php`-asset traps, Google-Fonts localization, true offline test.
- `references/troubleshooting.md` — blank page, wrong route, timeouts, CORS, fonts, external 404s.
- `references/rebuild-better.md` — turning a clone into an improved rebuild.
- `references/pricing.md` — **how to price a clone/rebuild job in Naira**: rate card, 3 tiers, phased/batch delivery, payment terms.
- `templates/client-intake.md` — **send this to a client** to collect all requirements.
- `templates/authorization-to-clone.md` — written permission / sign-off form.
- `templates/pricing-quote.md` — **client-facing proposal**: 3 packages + phased option, ready to send.
- `templates/handover.md` — delivery doc for the finished clone.

## Pricing a job for a client
When the user wants to quote a client, read `references/pricing.md` (the logic, anchored to the provider's own rate card) and send the client `templates/pricing-quote.md` (polished, 3 tiers + phased delivery). Always clarify **which tier** the client actually wants before quoting — "a site like [big site]" can mean anything from ₦350k (static look-alike) to ₦25M+ (full platform).