Design category and listing architecture so that a site's indexable value does not depend on paginated pages, using demand-backed indexable subcategories as the alternative to deep pagination. Load this skill when the user asks whether to noindex, canonicalise, block or prune paginated URLs (`?page=`, `?p=`, `/page/2/`); when they mention pagination handling, page-2+ indexation, infinite scroll and crawlability, orphaned product or article pages, "products only reachable from page 7", crawl budget wasted on listing pages, thin or duplicate category pages, faceted navigation index bloat, or splitting a large category into subcategories. Also load for "how do we cut paginated URLs without losing products" and listing-page taxonomy redesign for large catalogues. Applies to e-commerce catalogues, editorial archives and directories.
This skill makes a large listing-based site independent of its paginated pages before any
pagination pruning happens, by designing demand-backed leaf subcategories that surface every
item on page 1 of at least one listing — turning a defensive index-hygiene job into new ranking
surface.
Core principle
Before pruning a class of URLs from the index, re-architect so the internal link graph no longer
depends on them. Pagination handling is the last step, not the first. If deep pages are the only
route to a chunk of the catalogue, any noindex, canonical, redirect or crawl restriction applied to them
degrades the discoverability of the items behind them — no matter how correct the directive looks
in isolation.
The agent must therefore refuse to hand over a pagination directive as a standalone
recommendation. It delivers a sequenced plan: audit → subcategory design → coverage verification
→ conditional pruning, rolled out per category.
Phase 1 — Pagination-independence audit
Establish which items are reachable only through pagination.
Step 0 — define the paginated set from the linked surface, not from a URL pattern
Before configuring any crawl or query, fetch page 1, page 2 and a deep page of one representative
listing per template, and record four things:
(a) The pagination URL pattern as actually emitted in hrefs — not the platform's documented
convention. Verify it with counting probes: a plausible parameter can return zero links while an
undocumented variant (an infinite-scroll form such as &p=) carries the whole set.
(b) Which page URLs the pagination block links to: a full page list, next-only, or next+last.
This decides crawl depth and what "reachable" means for anything deep in the sequence — see
What the crawl cannot tell you below.
(c) The canonical target of each paginated page. This is evidence in its own right — if every
paginated page already canonicalises to page 1, the site itself declares them non-canonical, which
is often the substantive finding the whole audit is chasing, available from one fetch of the head.
(d) Whether a page-1 parameter variant exists (?p=1, ?page=1) and whether anything links to
it. It commonly returns 200 with content identical to the bare listing URL while appearing nowhere
in the link graph.
Define the "paginated source" class from (a) and (b). A URL pattern describes what can be
constructed; the link graph describes what exists, and the two diverge in both directions — a
matching variant that is never linked would be mis-classified as a legitimate paginated source,
and the linked surface is usually far sparser than the URL space. Every analysis or pruning decision
that turns on classifying paginated sources must derive the class this way. The divergence is cheap
to measure now and expensive to discover after the numbers are quoted.
If the analysis runs through a crawler or link-analysis platform with a page-type segmentation
layer, check before building any query that paginated listing URLs are their own segment class.
Where the platform's API has no negation operator, the segmentation is the negation operator:
"all sources except pagination" can only be expressed as "sources in [every other class]", which
requires pagination to be a named class. Paginated URLs almost always match their parent listing's
pattern, so they collapse into it unless deliberately carved out by an earlier-ordered rule. This is
a hidden precondition of the whole audit — fixing the segmentation upstream is cheap; working around
a missing class inside the query is not.
Audit steps
Crawl the site with all pagination routes excluded: exclude pagination parameters and path
patterns, and disable execution of any infinite-scroll or "load more" mechanism, so the crawler
sees only what a link graph without pagination exposes. Where listings load items client-side,
confirm whether page 1 renders its items in the initial HTML at all — if it does not, that is a
separate finding that must be fixed before anything else in this workflow means anything.
Record the crawl's depth limit and whether it seeds from XML sitemaps. Both are part of
the finding and are quoted alongside it.
Build a full item inventory independently: XML sitemaps, a full unrestricted crawl, or a
database or feed export. Prefer at least two sources and reconcile them. Where the question is
what the crawl could not reach, at least one source must be assembled by a method other than
traversal — a sitemap, a product feed, a database export, server log files, Search Console, or
the CMS's own item count.
Diff the two. Items present in the inventory but absent from the pagination-free crawl are
pagination-orphaned — provided the crawl finished. Before reading any absence off a
crawl, read the live queued/waiting count from the crawler's progress state, not a listing's
rounded completion percentage (a few hundred queued URLs against 100k fetched rounds to
100%). An item in the crawl's queue was discovered via a link and simply not fetched: it is
not orphaned, and the queue concentrates exactly the late-found, lightly-linked items the
diff is hunting. Export the queued URLs and subtract them from the absent set, or state that
the crawl finished with an empty queue. Recipe:
references/measurement-recipes.md ("Queue guard before any absence diff").
Constrain both endpoints to live URLs before ranking anything. Filter link sources to
HTTP 200 — links from redirecting pages are not real links — and filter targets to HTTP 200 as
well. Removed items (404/410), redirects and noindexed targets belong out of the population, not
at the top of it. A "worst-linked first" ordering surfaces deleted inventory before it surfaces a
real discovery problem, and reads as urgent precisely because the numbers are extreme; delisted
items often remain linked from a sale category's deep pagination long after removal. Inspect the
top few rows of any such ranking manually before quoting the number — the extreme tail is where
population-definition artefacts concentrate. Filtering can invert the headline: "two items are
reachable only via pagination" becomes "no live item is", which redirects the analysis from
whether pagination is a sole access path to how much of an item's link equity depends on it.
Attribute each orphan to its category, and — on multi-market or multilingual sites — to its
market. Orphan rates commonly differ sharply per market because catalogue depth and taxonomy
depth differ.
Report per category: total items, items on page 1, orphan count, orphan share, page count —
as a breakdown of the item-level coverage set (see Measure coverage at item level below),
never as a ranking of listings by size.
What the crawl cannot tell you
A dataset gathered by traversal cannot evidence the absence of pages it was never able to reach,
and its internal consistency actively conceals the gap. Two reasoning errors follow, and both must
be closed before any pruning decision rests on crawl data.
The unreachable band is the middle, not the end. Use the link pattern from Step 0 (b) to
work out which pages a depth-limited crawl could actually reach. With next+last pagination, page 1
links to page 2 and to page N, so both ends of the sequence sit at shallow depth; depth then
accumulates inward from page 2 forwards and — where a previous-link or rel=prev exists — from the
last page backwards. Under a depth limit, the reachable set is the first several pages plus a band
near the end, and the unreachable region is the middle. High page numbers routinely record
shallow depths for exactly this reason. Sorting by page number and assuming depth rises
monotonically gets this precisely backwards.
Missing items may be missing from the crawl, not from the site. Without sitemap seeding, an item
linked only from an unreachable middle page is absent from the crawl entirely — not merely absent
from its link graph — so every crawl-derived metric agrees the item does not exist and the analysis
returns a confident, self-consistent, wrong answer. Seeding inverts the signature: with it, such
items appear in the crawl with zero inlinks; without it, they are missing altogether. The seeding
setting therefore determines which query finds them.
So never state "no item depends on pagination for discovery" from crawl data alone. Qualify the
conclusion with the depth limit and the seeding configuration, and confirm it by set difference
against the independent inventory from step 2. The comparison, not the crawl, is the instrument.
The crawl also has a vintage, and the two deliverables it supports age at different rates.
The structural conclusion this workflow exists to produce — whether any item depends on
pagination as its sole access path — is about the shape of the link graph and survives weeks
of drift. The orphan table's per-URL rows are claims about specific pages right now, and decay
continuously from the moment the crawl completes. The same export therefore supports one
deliverable and not the other, and the distinction is invisible unless the vintage is known;
nothing in the workflow forces the question, which is why it belongs in the pre-flight.
Resolve and record the crawl's completion date alongside every figure, and state the
orphan list's as-of date in the deliverable. A report labelled "Last Crawl" is not a date —
resolve the tool's actual crawl-date / crawl-name value.
Ship aggregate and structural conclusions on an older crawl if need be. Do not ship the
per-URL orphan or low-inlink list that way: live-verify a stratified sample first and report
the observed stale rate, or regenerate from a fresh crawl.
The failure shape: a verified orphan list whose crawl date was never established. A
stratified sample finds a sizeable share of the "orphans" linked today — from prose inside
their category's description block, index,follow, self-canonical, no nofollow, exactly the
links the report says do not exist — because the descriptions were written after the crawl
ran. Part of the list is already fixed at delivery, and nobody can say which part.
Test reachability at anchor level, not by substring
Any manual check of whether an item is linked from a listing page must match the anchor that
carries the link — a pattern including <a and href= with the item's URL — never a bare item-id
substring. Modern listing templates ship a JSON product payload (dataLayer, hydration state) that
mentions far more item ids than the grid actually links, so a substring test answers "does this id
appear in the HTML", which is not the question. Useful cross-check: count product anchors per page
and compare against the count of id substring matches — a large gap means the substring test is
reading the payload, not the grid. And treat an arithmetically impossible result — an item
"appearing" on more pages of a stable-sorted sequence than the sequence can contain — as a
measurement fault, not a site fault: re-examine the method before hypothesising about the site,
because impossibility is the cheapest error signal available and it is absent from every plausible
wrong answer. The corrected test can invert the diagnosis entirely — from "hidden behind deep
pagination" to "linked from nowhere in its own category", a different problem with a different fix.
Source classification and measurement traps
Every per-item inlink count rests on a definition of "source", and the definition's failure cases
apply to every item uniformly — they shift the whole distribution by one and never look like
outliers. Apply these rules before quoting any count:
Remove self-links first. Crawler inlink totals commonly include the page itself (breadcrumb
last node, share links, a language switcher pointing at the same URL), so every item carries a
phantom +1 and "exactly one non-paginated source" is, for most of its members, zero plus self.
Label near-universal structural sources separately — HTML sitemap page, sitewide menus,
footer "all products". They make every item look linked. Report reachability (any
same-domain, non-self, non-paginated source) and editorial linking (a category body,
related-item or content link) as two numbers.
Multi-domain crawl projects: separate same-domain from cross-domain sources. Language or
country switcher and alternate links from sister domains are links between separate sites and
do not count toward internal pagination-independence. They are also why a multi-domain tool can
show no item below one non-paginated source while a single-domain crawl, which cannot see them
at all, shows hundreds.
Both sides of any subtraction from one snapshot. Never subtract a live-fetched count from a
crawl-stored count on a page set containing rotating listings (sale, new-in, bestseller,
featured): everything that changed between the two time points becomes a finding, and the
false negatives are silent. Same crawl for both sides, or exclude the rotating listings from
both and say so.
Apply the crawl-depth bound as a free pre-filter. A pagination-only item has crawl depth ≥ 3
(home → page 1 → page N → item). Items at depth ≤ 2 are excluded without any inlink data, and a
method that flags a depth-≤2 item as pagination-only is broken — reject the method, not the item.
Check what makes the bound pass before reading it as a property of the architecture: if a
complete HTML sitemap (or similar all-items page) is linked at depth 1, every item passes
trivially and the bound proves reachability only. Then measure page-1 coverage before
concluding that listings carry discovery, and report the HTML sitemap as the sole
non-paginated source for the items off page 1.
Test listing stability before treating a pagination-only set as a fixed remediation list.
An item on two pages of one listing within a crawl (once the anchor-level method is confirmed
sound), or on different pages across crawls, means the sort drifts. Then "pagination-only" is a
snapshot of a process, and the remedy is structural — the page-1 ceiling, stable related-item
links, a complete HTML sitemap as backstop — not linking one snapshot's list, because a
different set drifts onto page 2+ next week.
Read an absence off a fetched page only after checking the response is complete. When
verifying that a parent listing links a subcategory, filter page or item, confirm a marker that
must be present (filter panel, product grid, footer) and a size in line with sibling fetches,
test the link class on several pairs, and cross-check against crawl depth and actual inlink
sources rather than the inlink count alone. A partial response contains no links to anything.
Read paginated URLs' analytics from a dimension that keeps the query string. GA4's
"Landing page" dimension strips the query string, so ?p=N landings fold into the base URL
and a zero for them is an instrument artefact, not evidence that paginated pages get no
entries. Use "Landing page + query string", or Search Console page data.
The cheap test that precedes any aggregate: pull the actual inlink sources of the single
weakest item the method produced and classify each by hand — self, paginated, cross-domain,
structural, editorial. One read of the right row exposes a phantom source in every row. And when
an early verification turns up something odd but apparently harmless, write down what it would
imply if it were systematic before deciding it is minor; an anomaly noticed and rationalised is
costlier than one never seen. On one mid-sized catalogue, exactly this inverted a "no item depends
on pagination" finding into a material orphan share. Source-class table and the observed
cases: references/measurement-recipes.md ("Classifying inlink sources") and
its sibling sections on same-snapshot consistency, listing stability and fetched-page absence.
Measure coverage at item level
The pass condition is "every item linked from page 1 of at least one listing", so measure it
directly, at item level, as Phase 1's primary metric: fetch page 1 of every listing at one
moment, extract only the product grid (not menu, footer or cross-sell blocks), union the grids,
and diff against the independent inventory from step 2. The residual is the coverage gap. Two
rules travel with it:
Validate the extractor against a count the page declares about itself — an inline size
variable, a "showing 1–36 of 412" element, a payload field. Extracted grid count must equal
min(declared size, items-per-page) on every listing; a mismatch means the extractor is reading
something other than the grid. The same check exposes zero-item hub pages that look like top
categories and provide no coverage.
Do not rank listings by size to find uncovered items. Listing-size tables double-count
when facets cross-cut: the largest "leaf" listings in a big catalogue are attribute facets
over the same items, many already on page 1 of a smaller listing, and cross-cutting facets are
leaves by URL. The size table misleads most in exactly the catalogues where the problem is
biggest.
Record, per uncovered item, the earliest page it appears on in any listing — that distribution
sizes the items-per-page lever in Phase 2. Recipe:
references/measurement-recipes.md ("Item-level page-1 coverage from validated grid
extraction").
The output of Phase 1 is the item-level coverage gap and its per-category, per-market breakdown,
with each count's source classes, snapshot and queue state recorded. It is the input to every
later decision and the baseline the rollout is measured against.
Worked example (constructed). A catalogue category holds 1,200 items at 30 items per listing
page: 40 pages, 30 items linked from page 1, 1,170 pagination-orphaned (97.5%). A directive applied
to pages 2–40 in this state removes the only internal path to 97.5% of that category. The same
category after Phase 2 holds eight leaf subcategories of 80–250 items each; at 30 per page none
are single-page, so the design is revised — either the per-page count rises or the subcategories
split further — until page-1 coverage is met.
Phase 2 — Design demand-backed leaf subcategories
Subcategories are the alternative to deep pagination. Design them inside the existing main
category structure, not as a parallel taxonomy.
Constraints every candidate subcategory must satisfy:
Demand-backed. Each subcategory maps to real search or user demand — query data, internal
site-search logs, filter-usage data, or a keyword source. A subcategory nobody searches for is
index bloat with extra steps. A keyword tool's volume commonly sits on one lead variant of a
concept: query at least the singular, the plural and the common qualified form and take the
max before calling a term demand-less, and report which variant carried it.
Above the floor. A minimum item count, typically around six to eight, below which the page
is too thin to justify indexation. Set the floor explicitly with the user and record it.
Below the ceiling. Small enough that page 1 holds all or nearly all of its items. This is the
binding constraint: derive the ceiling from the site's items-per-page setting, and size the
items-per-page lever before designing any subcategory — from the earliest-page distribution
of uncovered items (Phase 1), compute the share each candidate page size brings onto a page 1
and where the residual concentrates, so design targets the residue and the listings that hold
it. Price the lever on per-item markup and DOM plus server render time (a staging test at the
target size); below-the-fold card images are commonly lazy-loaded already, and the cost must
be measured as a function of item count, never generalised from the first cards, which are
the above-the-fold exceptions by construction.
Covering. Every item in the catalogue must be linked from page 1 of at least one
subcategory. Coverage is the test the whole design passes or fails on.
Curated. Where subcategories are generated from facet combinations, only the curated,
demand-backed combinations become indexable and internally linked. Everything else stays
non-indexable. This is index governance, and it is part of the deliverable.
From the coverage gap to a page list
"Items not on page 1" is not the scope of subcategory work. Coverage decides what a subcategory
could fix; demand decides where one is worth having; the two are different sets, and
conflating them builds zero-demand pages for the wrong reason. Work in this order:
Inventory existing landing-page mechanisms first. Clean-URL filter landing pages, "shop
by" blocks or merchandising pages the platform already serves are the vehicle for new
subcategories before any new mechanism is specified. A clean slug in markup does not mean a
landing page exists — check each against the crawl's listing set, and verify parent → child
links under the fetched-page rule from Phase 1.
Pull exact facet membership from the platform's filter endpoint where it exposes one
(facet counts say how big a value is, not which uncovered items it lifts), and validate it
against the panel's counts.
Run a greedy set cover over the uncovered items with candidates inside the floor–ceiling
band, and present the coverage-vs-pages curve: it shows the ceiling (usually well below 100%)
and the long tail of pages needed to reach it.
Gate the candidates on search demand, under the variant rule above.
Deliver two tracks: demand-led subcategories (the candidates that pass the gate — usually
a handful, and the fundable part) and a coverage backstop for the rest (the items-per-page
lever, stable related-item links, a complete HTML sitemap) so no zero-demand page is built to
cover an item.
Recipes: references/subcategory-sizing.md ("Size the items-per-page lever"), with sibling
sections on pricing the lever, exact facet membership and set cover, and the demand gate.
Also specify, per subcategory: URL, title and heading pattern, where it is linked from (parent
category body, faceted navigation, breadcrumb), and its intended indexation state.
Frame the upside explicitly when presenting: the new subcategories are a fresh ranking surface
addressing mid-tail demand the main category never ranked for. This is offence as well as defence,
and it is usually what makes the work fundable.
Phase 3 — Verify page-1 coverage
Do not trust the design document. Verify against the built site, per category, before pruning
anything.
Re-run the pagination-free crawl from Phase 1 against the environment with the new
subcategories live.
Recompute the orphan table — as a set difference against the independent inventory, under the
same depth limit and seeding configuration as Phase 1, both restated with the result. A zero
computed inside the crawl alone is not a pass. The pass condition is zero pagination-orphaned
items in the categories being signed off, or an explicitly accepted, enumerated residual.
Watch for spillover: a subcategory that sat just under the page-1 ceiling at design time and
has since grown past it. Its overflow items are orphaned again the moment pagination is pruned.
Spillover is the single most likely cause of a regression in this workflow.
Confirm the new subcategory links are in the initial HTML and are crawlable links, not
script-driven handlers — from complete responses, on several parent/child pairs, under the
fetched-page rule in Phase 1. Recompute the coverage gap on the same-moment page-1 grids, with
the queue guard and source classification restated.
Phase 4 — Conditional pruning, rolled out per category
Only categories that passed Phase 3 get pagination handling. This is a per-category rollout, not a
site-wide switch.
Precondition, before any link-removal recommendation — three reads of the template, in order,
then one of the rendered page:
Enumerate every paginated anchor the listing template emits, not "the pagination control".
One template commonly carries several: a "load more" anchor, a "load a lot more" variant that
jumps ahead by a multiplier, and a hidden (display:none) no-JS block whose only output is a
next-page link. Usually only the last is genuinely crawler-only — which is what people tend to
believe about the first. A removal ticket naming just the element under discussion leaves the
others live and the change silently fails.
For each anchor, record whether the crawl path and the user-facing control are the same
element. A progressively-enhanced "load more" button is commonly an anchor to the next
paginated URL — JS intercepts the click and appends results, while the href is simultaneously
the no-JS fallback and the crawl path — so "remove the links crawlers use, keep the control users
use" is one change, not two. A belief about why a link exists ("this is only here for crawlers")
is a claim about the template, and templates are cheap to inspect and frequently contradict it.
Where they are one element, read the click handler and record whether the JS path depends on
the href. Same element is not the same as inseparable. The common pattern —
preventDefault() followed by an AJAX append — never reads the href, so stripping it leaves
the control identical for every JS-enabled visitor and the whole cost falls on no-JS visitors
and crawlers; on a JS-dependent shop that is approximately nobody plus the path being removed on
purpose. Removal is then cheap. Look one layer further into the script for an existing seam: if
the next-page URL resolves as data-href || href (or similar), the change is renaming an
attribute and the JS needs no edit at all. Only where the handler does read the href, or no
handler is bound and the anchor is the navigation, is the cost real — link removal is then a
front-end change with UX consequences and must be scoped, estimated and risk-assessed as one,
not presented as an SEO-only tweak.
Load the listing in a real browser before any claim about what users see or can do.
Markup shows what the server sends, not what the user sees: a numbered block can be
display:none while a scroll script loads the next pages and rewrites the address bar. Read
the pagination block's computed display, scroll to the grid end, count the items, and watch
the address bar and the network log. For each scroll or AJAX load, record the requested URL
and whether it differs from the crawlable paginated URL (consequence: the AJAX carve-out
below). A UX statement is never made from static HTML alone. Recipe:
references/pruning-evidence.md ("Rendered-page check").
The coupling finding and the cost finding come from the same inspection. Stopping after the first
turns a good check into a bad estimate, and an overstated cost argues against a change the client
may have good reason to make — the more damaging error, because it looks like rigour.
Evidence for pruning — what the paginated URLs do in search today. Before choosing a lever,
check the paginated URLs' indexability (canonical, robots, hreflang) and pull, per market, the
ranking tool's keywords whose ranking URL matches the pagination parameter and Search Console
page × query for the paginated set. Paginated URLs ranking for head terms or item queries are the
cannibalisation case that justifies pruning. A paginated URL holding head-term traffic needs a
deliberately chosen redirect target and its own monitoring; a blanket "301 to page 1" rule must
not cover it by default. Where paginated URLs carry hreflang to sister domains' paginated URLs,
pruning ships on all of those domains together. Recipe: references/pruning-evidence.md
("Evidence for pruning").
Once items are genuinely pagination-independent, a self-referencing canonical on each paginated
page is usually sufficient: the pages remain crawlable, the items no longer depend on them, and
no signal is misdirected. Treat canonicalising deep pages to page 1 as a secondary lever, used
where the client specifically needs consolidation and accepts that it is a hint rather than a
directive.
Blocking pagination in robots.txt cuts the crawler's path through those pages entirely. Reserve
it for cases with a demonstrated crawl-budget problem, and only after coverage has been verified.
301-redirecting ?p=N to the base listing URL is the most aggressive lever of all — not a
gentler alternative to a robots block. The ladder orders by how much discoverability is removed,
not by how forceful the mechanism feels: a robots-blocked page is merely unfetched, whereas a
redirected one resolves to different content, so any item appearing only on page 2+ becomes
unreachable the instant it ships, with no grace period and no partial behaviour. The property
that makes the redirect attractive — a directive where a canonical is a hint — is the property
that makes it final. It arrives in conversation looking like a tidy-up after link removal, or as
the fix for the stranding risk of a robots block; it inherits none of that option's caution by
default and deserves more. The escalation case that justifies it: a canonical-to-page-1
demonstrably ignored on URLs with long indexation history, after coverage is verified. Include
the page-1 parameter variant (?p=1) in the redirect — a pure duplicate that is typically never
linked. Two operational requirements travel with it:
Sequencing inverts. The redirect goes in before the internal links are removed, not
after. Once every fetch resolves to the base URL, removing the links is tidiness rather than a
precondition, and the window in which URLs are unlinked-but-not-yet-consolidated never opens.
The AJAX carve-out is a requirement of the dev instruction, not a caveat. A
progressively-enhanced listing typically fetches its own paginated URL space for partial
responses — ?p=N&infScroll=1 or similar — so a redirect matching [?&]p=\d+ alone catches the
site's own requests and the listing silently serves page 1 forever. Enumerate every parameter
the listing's scripts append (one template has been observed using two different such
parameters, one per load-more control) and exempt each by name in the ticket. Where the
rendered-page check (read 4) shows a scroll load requesting the plain paginated URL with no
marker, there is nothing to exempt: the dev instruction first adds a marker parameter to the
script, and the redirect exempts that marker.
Before finalising the directive, check current search engine guidance on pagination handling
and rel next/prev, and verify it against the vendor's dated changelog or announcement rather
than the documentation page alone — documentation pages are edited silently, changelogs carry
dates. Record the date and source of what was checked in the deliverable.
Sequence the rollout: one or two categories first, monitor indexation and organic coverage of
their items for a full crawl-and-index cycle, then extend. Define in advance what a regression
looks like and what rolls it back.
Phase 5 — Governance
Hand over the rules that keep the architecture true after the project ends.
The page-1 ceiling is an ongoing constraint. Specify a recurring check that flags subcategories
which have grown past it, and the action when one does (split, or raise items-per-page).
New items must land in at least one demand-backed subcategory at creation. Specify who enforces
this and where.
The curated facet-combination allow-list needs an owner and a review cadence. Without one it
either ossifies or leaks.
Re-run the Phase 1 item-level coverage diff periodically as the standing health metric: the
uncovered count should stay at the accepted residual. On drifting listings this is the only
measure that holds — a fixed list of items does not.
Anti-patterns
Directive first. Recommending noindex, canonical-to-page-1, a robots.txt block or a redirect
on pagination before the orphan audit exists. This is the failure this skill exists to prevent.
Ranking the redirect below the robots block. A 301 to page 1 is proposed as the tidy-up
after link removal or as the safe fix for a robots block's stranding risk, and priced beside
"canonical, but it works". Order levers by what becomes unreachable the moment they ship: a
redirect removes more discoverability than any of them, and a redirect that does not exempt the
listing's own AJAX parameters by name breaks the listing.
Defining the paginated set by regex. Classifying sources from a URL pattern rather than from
the links the templates emit. It admits parameter variants nothing links to and misses linked
variants the pattern never anticipated, and both errors survive into every downstream number.
Asserting the separation between crawl path and control without reading the template — in
either direction. A belief about the purpose of a piece of markup is not evidence about the
markup. Assuming the load-more link exists "only for crawlers" and is removable for free is one
error, and it survives longest among the people who built the thing; assuming that because it is
one element the removal must be expensive is the other, and it looks like rigour. Enumerate the
anchors, read the handler, then price it.
Testing whether an item is linked by grepping for its id. An id substring hits the JSON
product payload, not the link; only an anchor-level match (<a + href=) answers "is this
item linked from here". An impossible result (one item on more pages than a stable sort
allows) is the instrument failing, not the site.
Shipping a per-URL orphan list without its vintage. An undated list implicitly claims to
describe the present. Structural conclusions survive an ageing crawl; per-URL remediation
lists do not, and the stale rate is measurable on a sample before delivery.
Reading a crawl's silence as evidence. Treating "not in the crawl data" as "not on the site"
without stating the depth limit and seeding configuration and checking a non-traversal source.
A depth-limited crawl of next+last pagination is blind to the middle of every long sequence.
Ranking by link deficiency without filtering dead targets. The top of an unfiltered
"worst-linked" list is mostly removed pages, so the finding is an artefact of the population
definition rather than a fact about the site.
Reading an absence off an unfinished crawl or a partial response. A listing's rounded 100%
with URLs still queued, or one truncated fetch of a parent page, both produce a clean,
plausible, wrong "not linked". Check the queue and the response before the diff.
Counting a self-link or a sister-domain switcher link as a non-paginated source. Both apply
to every item uniformly, so they never surface as outliers; "exactly one source" is then "zero
plus self". Classify the weakest item's sources by hand before trusting the aggregate.
Joining a live fetch against a crawl snapshot on a rotating listing. Every item that moved
into or out of the sale listing between the two time points becomes a finding. Both sides from
one snapshot, and the depth bound before any of it.
Ranking listings by size to find uncovered items. Cross-cutting facets double-count the
same items; measure item-level page-1 coverage directly with a validated grid extractor.
Pricing the items-per-page lever from the first cards, or treating "items not on page 1" as
the scope of subcategory work. The first cards are the above-the-fold exceptions — measure
as a function of item count. Demand decides where a subcategory is worth having; the uncovered
remainder gets a coverage backstop, not zero-demand pages.
Noindexing paginated pages that are still the only route to items. Beyond the coverage
problem, a noindexed page's onward links carry less weight over time — the orphaning is
progressive and easy to miss.
Subcategories invented from the catalogue tree alone. Splitting by whatever attribute the
product data happens to carry, with no demand check, produces pages nobody searches for and a
larger index to govern.
Auto-generating every facet combination. Turning the whole facet matrix into indexable pages
replaces a pagination problem with a combinatorial one. Curate.
Site-wide rollout on one category's evidence. Coverage is per category; so is the rollout.
Ignoring per-market variance. Signing off a market with a shallow catalogue and applying the
same directive to a deep one re-orphans thousands of items.
Treating design sign-off as verification. The design document is a plan. Only a re-crawl is
evidence.
Dropping the ranking-surface framing. Presented purely as index hygiene, the work reads as
cost with no upside and stalls.
Pre-flight verification
Before delivering, the agent runs this against its own output:
The paginated-source class was derived from the links the templates actually emit, with the
emitted pattern, the linked page set (full list / next-only / next+last), each paginated page's
canonical target and the status of any page-1 parameter variant recorded.
Where a segmentation layer drives the analysis, pagination is its own segment class.
A pagination-free crawl was actually run (or its absence is flagged as a blocking gap), not
assumed from sitemap data alone.
The crawl's depth limit and sitemap-seeding setting are stated, the unreachable band of the
sequence is reasoned about from the link pattern, and no "nothing depends on pagination"
conclusion rests on crawl data alone.
Both endpoints of any link ranking are constrained to live 200 URLs, and the top rows of any
"worst-linked" ordering were inspected before the number was quoted.
The item inventory came from at least two reconciled sources, at least one of them not
assembled by traversal.
The crawl's queued/waiting count was read from its live progress state before any absence
diff; the deliverable states that the queue was empty or how many queued URLs were
subtracted from the absent set.
Every per-item source count excludes self-links, labels structural sources separately
(reachability and editorial linking reported as two numbers), counts same-domain sources
only in a multi-domain project, takes both sides of any subtraction from one snapshot, and
applies the depth-≥3 bound; the weakest item's sources were classified by hand.
A depth bound passed via a depth-1 HTML sitemap is reported as reachability only, with the
sitemap named as sole non-paginated source off page 1; analytics for paginated URLs keep
the query string (never GA4's "Landing page").
Listing stability was tested before any pagination-only set was presented as a fixed
remediation list, and where listings drift the remedy is structural.
Item-level page-1 coverage was measured from same-moment page-1 grids with the extractor
validated against a declared per-listing count, and the per-category, per-market table is
a breakdown of that set, with counts and shares, not a size ranking of listings.
The crawl's completion date is resolved and recorded, and the orphan table carries an
as-of date. Any per-URL list shipped for item-by-item action was live-v
…(truncated)
1---2name: pagination-independence-architecture3description: "Design category and listing architecture so that a site's indexable value does not depend on paginated pages, using demand-backed indexable subcategories as the alternative to deep pagination. Load this skill when the user asks whether to noindex, canonicalise, block or prune paginated URLs (`?page=`, `?p=`, `/page/2/`); when they mention pagination handling, page-2+ indexation, infinite scroll and crawlability, orphaned product or article pages, \"products only reachable from page 7\", crawl budget wasted on listing pages, thin or duplicate category pages, faceted navigation index bloat, or splitting a large category into subcategories. Also load for \"how do we cut paginated URLs without losing products\" and listing-page taxonomy redesign for large catalogues. Applies to e-commerce catalogues, editorial archives and directories."4version: 1.0.05license: CC-BY-4.06metadata:7 author: Eoghan Henn and contributors8 source: https://skillmd.com/skills/rebelytics/pagination-independence-architecture9 upstream: https://github.com/rebelytics/pagination-independence-architecture10---1112# Pagination-Independence Architecture1314This skill makes a large listing-based site independent of its paginated pages before any15pagination pruning happens, by designing demand-backed leaf subcategories that surface every16item on page 1 of at least one listing — turning a defensive index-hygiene job into new ranking17surface.1819## Core principle2021Before pruning a class of URLs from the index, re-architect so the internal link graph no longer22depends on them. Pagination handling is the last step, not the first. If deep pages are the only23route to a chunk of the catalogue, any noindex, canonical, redirect or crawl restriction applied to them24degrades the discoverability of the items behind them — no matter how correct the directive looks25in isolation.2627The agent must therefore refuse to hand over a pagination directive as a standalone28recommendation. It delivers a sequenced plan: audit → subcategory design → coverage verification29→ conditional pruning, rolled out per category.3031## Phase 1 — Pagination-independence audit3233Establish which items are reachable *only* through pagination.3435### Step 0 — define the paginated set from the linked surface, not from a URL pattern3637Before configuring any crawl or query, fetch page 1, page 2 and a deep page of one representative38listing per template, and record four things:3940- **(a)** The pagination URL pattern *as actually emitted in hrefs* — not the platform's documented41 convention. Verify it with counting probes: a plausible parameter can return zero links while an42 undocumented variant (an infinite-scroll form such as `&p=`) carries the whole set.43- **(b)** Which page URLs the pagination block links to: a full page list, next-only, or next+last.44 This decides crawl depth and what "reachable" means for anything deep in the sequence — see45 *What the crawl cannot tell you* below.46- **(c)** The canonical target of each paginated page. This is evidence in its own right — if every47 paginated page already canonicalises to page 1, the site itself declares them non-canonical, which48 is often the substantive finding the whole audit is chasing, available from one fetch of the head.49- **(d)** Whether a page-1 parameter variant exists (`?p=1`, `?page=1`) and whether anything links to50 it. It commonly returns 200 with content identical to the bare listing URL while appearing nowhere51 in the link graph.5253Define the "paginated source" class from (a) and (b). A URL pattern describes what can be54constructed; the link graph describes what exists, and the two diverge in **both** directions — a55matching variant that is never linked would be mis-classified as a legitimate paginated source,56and the linked surface is usually far sparser than the URL space. Every analysis or pruning decision57that turns on classifying paginated sources must derive the class this way. The divergence is cheap58to measure now and expensive to discover after the numbers are quoted.5960If the analysis runs through a crawler or link-analysis platform with a page-type **segmentation**61layer, check before building any query that paginated listing URLs are their own segment class.62Where the platform's API has no negation operator, the segmentation *is* the negation operator:63"all sources except pagination" can only be expressed as "sources in [every other class]", which64requires pagination to be a named class. Paginated URLs almost always match their parent listing's65pattern, so they collapse into it unless deliberately carved out by an earlier-ordered rule. This is66a hidden precondition of the whole audit — fixing the segmentation upstream is cheap; working around67a missing class inside the query is not.6869### Audit steps70711. Crawl the site with all pagination routes excluded: exclude pagination parameters and path72 patterns, and disable execution of any infinite-scroll or "load more" mechanism, so the crawler73 sees only what a link graph without pagination exposes. Where listings load items client-side,74 confirm whether page 1 renders its items in the initial HTML at all — if it does not, that is a75 separate finding that must be fixed before anything else in this workflow means anything.76 Record the crawl's **depth limit** and whether it **seeds from XML sitemaps**. Both are part of77 the finding and are quoted alongside it.782. Build a full item inventory independently: XML sitemaps, a full unrestricted crawl, or a79 database or feed export. Prefer at least two sources and reconcile them. Where the question is80 what the crawl could not reach, at least one source must be assembled by a method other than81 traversal — a sitemap, a product feed, a database export, server log files, Search Console, or82 the CMS's own item count.833. Diff the two. Items present in the inventory but absent from the pagination-free crawl are84 **pagination-orphaned** — provided the crawl finished. Before reading any absence off a85 crawl, read the live queued/waiting count from the crawler's progress state, not a listing's86 rounded completion percentage (a few hundred queued URLs against 100k fetched rounds to87 100%). An item in the crawl's queue was discovered via a link and simply not fetched: it is88 not orphaned, and the queue concentrates exactly the late-found, lightly-linked items the89 diff is hunting. Export the queued URLs and subtract them from the absent set, or state that90 the crawl finished with an empty queue. Recipe:91 `references/measurement-recipes.md` ("Queue guard before any absence diff").924. **Constrain both endpoints to live URLs before ranking anything.** Filter link sources to93 HTTP 200 — links from redirecting pages are not real links — *and* filter targets to HTTP 200 as94 well. Removed items (404/410), redirects and noindexed targets belong out of the population, not95 at the top of it. A "worst-linked first" ordering surfaces deleted inventory before it surfaces a96 real discovery problem, and reads as urgent precisely because the numbers are extreme; delisted97 items often remain linked from a sale category's deep pagination long after removal. Inspect the98 top few rows of any such ranking manually before quoting the number — the extreme tail is where99 population-definition artefacts concentrate. Filtering can invert the headline: "two items are100 reachable only via pagination" becomes "no live item is", which redirects the analysis from101 *whether* pagination is a sole access path to *how much* of an item's link equity depends on it.1025. Attribute each orphan to its category, and — on multi-market or multilingual sites — to its103 market. Orphan rates commonly differ sharply per market because catalogue depth and taxonomy104 depth differ.1056. Report per category: total items, items on page 1, orphan count, orphan share, page count —106 as a breakdown of the item-level coverage set (see *Measure coverage at item level* below),107 never as a ranking of listings by size.108109### What the crawl cannot tell you110111A dataset gathered by traversal cannot evidence the absence of pages it was never able to reach,112and its internal consistency actively conceals the gap. Two reasoning errors follow, and both must113be closed before any pruning decision rests on crawl data.114115**The unreachable band is the middle, not the end.** Use the link pattern from Step 0 (b) to116work out which pages a depth-limited crawl could actually reach. With next+last pagination, page 1117links to page 2 *and* to page N, so both ends of the sequence sit at shallow depth; depth then118accumulates inward from page 2 forwards and — where a previous-link or `rel=prev` exists — from the119last page backwards. Under a depth limit, the reachable set is the first several pages plus a band120near the end, and the unreachable region is the **middle**. High page numbers routinely record121shallow depths for exactly this reason. Sorting by page number and assuming depth rises122monotonically gets this precisely backwards.123124**Missing items may be missing from the crawl, not from the site.** Without sitemap seeding, an item125linked only from an unreachable middle page is absent from the crawl entirely — not merely absent126from its link graph — so every crawl-derived metric agrees the item does not exist and the analysis127returns a confident, self-consistent, wrong answer. Seeding inverts the signature: with it, such128items appear in the crawl with zero inlinks; without it, they are missing altogether. The seeding129setting therefore determines which query finds them.130131So never state "no item depends on pagination for discovery" from crawl data alone. Qualify the132conclusion with the depth limit and the seeding configuration, and confirm it by set difference133against the independent inventory from step 2. The comparison, not the crawl, is the instrument.134135**The crawl also has a vintage, and the two deliverables it supports age at different rates.**136The structural conclusion this workflow exists to produce — whether any item depends on137pagination as its sole access path — is about the *shape* of the link graph and survives weeks138of drift. The orphan table's per-URL rows are claims about specific pages right now, and decay139continuously from the moment the crawl completes. The same export therefore supports one140deliverable and not the other, and the distinction is invisible unless the vintage is known;141nothing in the workflow forces the question, which is why it belongs in the pre-flight.142143- Resolve and record the crawl's **completion date** alongside every figure, and state the144 orphan list's as-of date in the deliverable. A report labelled "Last Crawl" is not a date —145 resolve the tool's actual crawl-date / crawl-name value.146- Ship aggregate and structural conclusions on an older crawl if need be. Do **not** ship the147 per-URL orphan or low-inlink list that way: live-verify a stratified sample first and report148 the observed stale rate, or regenerate from a fresh crawl.149150The failure shape: a verified orphan list whose crawl date was never established. A151stratified sample finds a sizeable share of the "orphans" linked today — from prose inside152their category's description block, `index,follow`, self-canonical, no nofollow, exactly the153links the report says do not exist — because the descriptions were written after the crawl154ran. Part of the list is already fixed at delivery, and nobody can say which part.155156### Test reachability at anchor level, not by substring157158Any manual check of whether an item is linked from a listing page must match the anchor that159carries the link — a pattern including `<a` and `href=` with the item's URL — never a bare item-id160substring. Modern listing templates ship a JSON product payload (dataLayer, hydration state) that161mentions far more item ids than the grid actually links, so a substring test answers "does this id162appear in the HTML", which is not the question. Useful cross-check: count product anchors per page163and compare against the count of id substring matches — a large gap means the substring test is164reading the payload, not the grid. And treat an arithmetically impossible result — an item165"appearing" on more pages of a stable-sorted sequence than the sequence can contain — as a166measurement fault, not a site fault: re-examine the method before hypothesising about the site,167because impossibility is the cheapest error signal available and it is absent from every plausible168wrong answer. The corrected test can invert the diagnosis entirely — from "hidden behind deep169pagination" to "linked from nowhere in its own category", a different problem with a different fix.170171### Source classification and measurement traps172173Every per-item inlink count rests on a definition of "source", and the definition's failure cases174apply to every item uniformly — they shift the whole distribution by one and never look like175outliers. Apply these rules before quoting any count:176177- **Remove self-links first.** Crawler inlink totals commonly include the page itself (breadcrumb178 last node, share links, a language switcher pointing at the same URL), so every item carries a179 phantom +1 and "exactly one non-paginated source" is, for most of its members, zero plus self.180- **Label near-universal structural sources separately** — HTML sitemap page, sitewide menus,181 footer "all products". They make every item look linked. Report **reachability** (any182 same-domain, non-self, non-paginated source) and **editorial linking** (a category body,183 related-item or content link) as two numbers.184- **Multi-domain crawl projects: separate same-domain from cross-domain sources.** Language or185 country switcher and alternate links from sister domains are links between separate sites and186 do not count toward internal pagination-independence. They are also why a multi-domain tool can187 show no item below one non-paginated source while a single-domain crawl, which cannot see them188 at all, shows hundreds.189- **Both sides of any subtraction from one snapshot.** Never subtract a live-fetched count from a190 crawl-stored count on a page set containing rotating listings (sale, new-in, bestseller,191 featured): everything that changed between the two time points becomes a finding, and the192 false negatives are silent. Same crawl for both sides, or exclude the rotating listings from193 both and say so.194- **Apply the crawl-depth bound as a free pre-filter.** A pagination-only item has crawl depth ≥ 3195 (home → page 1 → page N → item). Items at depth ≤ 2 are excluded without any inlink data, and a196 method that flags a depth-≤2 item as pagination-only is broken — reject the method, not the item.197 Check what makes the bound pass before reading it as a property of the architecture: if a198 complete HTML sitemap (or similar all-items page) is linked at depth 1, every item passes199 trivially and the bound proves reachability only. Then measure page-1 coverage before200 concluding that listings carry discovery, and report the HTML sitemap as the sole201 non-paginated source for the items off page 1.202- **Test listing stability before treating a pagination-only set as a fixed remediation list.**203 An item on two pages of one listing within a crawl (once the anchor-level method is confirmed204 sound), or on different pages across crawls, means the sort drifts. Then "pagination-only" is a205 snapshot of a process, and the remedy is structural — the page-1 ceiling, stable related-item206 links, a complete HTML sitemap as backstop — not linking one snapshot's list, because a207 different set drifts onto page 2+ next week.208- **Read an absence off a fetched page only after checking the response is complete.** When209 verifying that a parent listing links a subcategory, filter page or item, confirm a marker that210 must be present (filter panel, product grid, footer) and a size in line with sibling fetches,211 test the link class on several pairs, and cross-check against crawl depth and actual inlink212 sources rather than the inlink count alone. A partial response contains no links to anything.213- **Read paginated URLs' analytics from a dimension that keeps the query string.** GA4's214 "Landing page" dimension strips the query string, so `?p=N` landings fold into the base URL215 and a zero for them is an instrument artefact, not evidence that paginated pages get no216 entries. Use "Landing page + query string", or Search Console page data.217218**The cheap test that precedes any aggregate:** pull the actual inlink sources of the single219weakest item the method produced and classify each by hand — self, paginated, cross-domain,220structural, editorial. One read of the right row exposes a phantom source in every row. And when221an early verification turns up something odd but apparently harmless, write down what it would222imply if it were systematic before deciding it is minor; an anomaly noticed and rationalised is223costlier than one never seen. On one mid-sized catalogue, exactly this inverted a "no item depends224on pagination" finding into a material orphan share. Source-class table and the observed225cases: `references/measurement-recipes.md` ("Classifying inlink sources") and226its sibling sections on same-snapshot consistency, listing stability and fetched-page absence.227228### Measure coverage at item level229230The pass condition is "every item linked from page 1 of at least one listing", so measure it231directly, at item level, as Phase 1's primary metric: fetch page 1 of every listing at one232moment, extract only the product grid (not menu, footer or cross-sell blocks), union the grids,233and diff against the independent inventory from step 2. The residual is the coverage gap. Two234rules travel with it:235236- **Validate the extractor against a count the page declares about itself** — an inline size237 variable, a "showing 1–36 of 412" element, a payload field. Extracted grid count must equal238 min(declared size, items-per-page) on every listing; a mismatch means the extractor is reading239 something other than the grid. The same check exposes zero-item hub pages that look like top240 categories and provide no coverage.241- **Do not rank listings by size to find uncovered items.** Listing-size tables double-count242 when facets cross-cut: the largest "leaf" listings in a big catalogue are attribute facets243 over the same items, many already on page 1 of a smaller listing, and cross-cutting facets are244 leaves by URL. The size table misleads most in exactly the catalogues where the problem is245 biggest.246247Record, per uncovered item, the earliest page it appears on in any listing — that distribution248sizes the items-per-page lever in Phase 2. Recipe:249`references/measurement-recipes.md` ("Item-level page-1 coverage from validated grid250extraction").251252The output of Phase 1 is the item-level coverage gap and its per-category, per-market breakdown,253with each count's source classes, snapshot and queue state recorded. It is the input to every254later decision and the baseline the rollout is measured against.255256**Worked example (constructed).** A catalogue category holds 1,200 items at 30 items per listing257page: 40 pages, 30 items linked from page 1, 1,170 pagination-orphaned (97.5%). A directive applied258to pages 2–40 in this state removes the only internal path to 97.5% of that category. The same259category after Phase 2 holds eight leaf subcategories of 80–250 items each; at 30 per page none260are single-page, so the design is revised — either the per-page count rises or the subcategories261split further — until page-1 coverage is met.262263## Phase 2 — Design demand-backed leaf subcategories264265Subcategories are the alternative to deep pagination. Design them inside the existing main266category structure, not as a parallel taxonomy.267268Constraints every candidate subcategory must satisfy:269270- **Demand-backed.** Each subcategory maps to real search or user demand — query data, internal271 site-search logs, filter-usage data, or a keyword source. A subcategory nobody searches for is272 index bloat with extra steps. A keyword tool's volume commonly sits on one lead variant of a273 concept: query at least the singular, the plural and the common qualified form and take the274 max before calling a term demand-less, and report which variant carried it.275- **Above the floor.** A minimum item count, typically around six to eight, below which the page276 is too thin to justify indexation. Set the floor explicitly with the user and record it.277- **Below the ceiling.** Small enough that page 1 holds all or nearly all of its items. This is the278 binding constraint: derive the ceiling from the site's items-per-page setting, and **size the279 items-per-page lever before designing any subcategory** — from the earliest-page distribution280 of uncovered items (Phase 1), compute the share each candidate page size brings onto a page 1281 and where the residual concentrates, so design targets the residue and the listings that hold282 it. Price the lever on per-item markup and DOM plus server render time (a staging test at the283 target size); below-the-fold card images are commonly lazy-loaded already, and the cost must284 be measured as a function of item count, never generalised from the first cards, which are285 the above-the-fold exceptions by construction.286- **Covering.** Every item in the catalogue must be linked from page 1 of at least one287 subcategory. Coverage is the test the whole design passes or fails on.288- **Curated.** Where subcategories are generated from facet combinations, only the curated,289 demand-backed combinations become indexable and internally linked. Everything else stays290 non-indexable. This is index governance, and it is part of the deliverable.291292### From the coverage gap to a page list293294"Items not on page 1" is not the scope of subcategory work. Coverage decides what a subcategory295*could* fix; demand decides where one is *worth having*; the two are different sets, and296conflating them builds zero-demand pages for the wrong reason. Work in this order:2972981. **Inventory existing landing-page mechanisms first.** Clean-URL filter landing pages, "shop299 by" blocks or merchandising pages the platform already serves are the vehicle for new300 subcategories before any new mechanism is specified. A clean slug in markup does not mean a301 landing page exists — check each against the crawl's listing set, and verify parent → child302 links under the fetched-page rule from Phase 1.3032. **Pull exact facet membership** from the platform's filter endpoint where it exposes one304 (facet counts say how big a value is, not which uncovered items it lifts), and validate it305 against the panel's counts.3063. **Run a greedy set cover** over the uncovered items with candidates inside the floor–ceiling307 band, and present the coverage-vs-pages curve: it shows the ceiling (usually well below 100%)308 and the long tail of pages needed to reach it.3094. **Gate the candidates on search demand**, under the variant rule above.3105. **Deliver two tracks:** demand-led subcategories (the candidates that pass the gate — usually311 a handful, and the fundable part) and a coverage backstop for the rest (the items-per-page312 lever, stable related-item links, a complete HTML sitemap) so no zero-demand page is built to313 cover an item.314315Recipes: `references/subcategory-sizing.md` ("Size the items-per-page lever"), with sibling316sections on pricing the lever, exact facet membership and set cover, and the demand gate.317318Also specify, per subcategory: URL, title and heading pattern, where it is linked from (parent319category body, faceted navigation, breadcrumb), and its intended indexation state.320321Frame the upside explicitly when presenting: the new subcategories are a fresh ranking surface322addressing mid-tail demand the main category never ranked for. This is offence as well as defence,323and it is usually what makes the work fundable.324325## Phase 3 — Verify page-1 coverage326327Do not trust the design document. Verify against the built site, per category, before pruning328anything.3293301. Re-run the pagination-free crawl from Phase 1 against the environment with the new331 subcategories live.3322. Recompute the orphan table — as a set difference against the independent inventory, under the333 same depth limit and seeding configuration as Phase 1, both restated with the result. A zero334 computed inside the crawl alone is not a pass. The pass condition is zero pagination-orphaned335 items in the categories being signed off, or an explicitly accepted, enumerated residual.3363. Watch for **spillover**: a subcategory that sat just under the page-1 ceiling at design time and337 has since grown past it. Its overflow items are orphaned again the moment pagination is pruned.338 Spillover is the single most likely cause of a regression in this workflow.3394. Confirm the new subcategory links are in the initial HTML and are crawlable links, not340 script-driven handlers — from complete responses, on several parent/child pairs, under the341 fetched-page rule in Phase 1. Recompute the coverage gap on the same-moment page-1 grids, with342 the queue guard and source classification restated.343344## Phase 4 — Conditional pruning, rolled out per category345346Only categories that passed Phase 3 get pagination handling. This is a per-category rollout, not a347site-wide switch.348349Precondition, before any link-removal recommendation — three reads of the template, in order,350then one of the rendered page:3513521. **Enumerate every paginated anchor the listing template emits**, not "the pagination control".353 One template commonly carries several: a "load more" anchor, a "load a lot more" variant that354 jumps ahead by a multiplier, and a hidden (`display:none`) no-JS block whose only output is a355 next-page link. Usually only the last is genuinely crawler-only — which is what people tend to356 believe about the first. A removal ticket naming just the element under discussion leaves the357 others live and the change silently fails.3582. **For each anchor, record whether the crawl path and the user-facing control are the same359 element.** A progressively-enhanced "load more" button is commonly an anchor to the next360 paginated URL — JS intercepts the click and appends results, while the `href` is simultaneously361 the no-JS fallback and the crawl path — so "remove the links crawlers use, keep the control users362 use" is one change, not two. A belief about why a link exists ("this is only here for crawlers")363 is a claim about the template, and templates are cheap to inspect and frequently contradict it.3643. **Where they are one element, read the click handler and record whether the JS path depends on365 the `href`.** Same element is not the same as inseparable. The common pattern —366 `preventDefault()` followed by an AJAX append — never reads the `href`, so stripping it leaves367 the control identical for every JS-enabled visitor and the whole cost falls on no-JS visitors368 and crawlers; on a JS-dependent shop that is approximately nobody plus the path being removed on369 purpose. Removal is then cheap. Look one layer further into the script for an existing seam: if370 the next-page URL resolves as `data-href || href` (or similar), the change is renaming an371 attribute and the JS needs no edit at all. Only where the handler *does* read the `href`, or no372 handler is bound and the anchor is the navigation, is the cost real — link removal is then a373 front-end change with UX consequences and must be scoped, estimated and risk-assessed as one,374 not presented as an SEO-only tweak.3754. **Load the listing in a real browser before any claim about what users see or can do.**376 Markup shows what the server sends, not what the user sees: a numbered block can be377 `display:none` while a scroll script loads the next pages and rewrites the address bar. Read378 the pagination block's computed display, scroll to the grid end, count the items, and watch379 the address bar and the network log. For each scroll or AJAX load, record the requested URL380 and whether it differs from the crawlable paginated URL (consequence: the AJAX carve-out381 below). A UX statement is never made from static HTML alone. Recipe:382 `references/pruning-evidence.md` ("Rendered-page check").383384The coupling finding and the cost finding come from the same inspection. Stopping after the first385turns a good check into a bad estimate, and an overstated cost argues against a change the client386may have good reason to make — the more damaging error, because it looks like rigour.387388**Evidence for pruning — what the paginated URLs do in search today.** Before choosing a lever,389check the paginated URLs' indexability (canonical, robots, hreflang) and pull, per market, the390ranking tool's keywords whose ranking URL matches the pagination parameter and Search Console391page × query for the paginated set. Paginated URLs ranking for head terms or item queries are the392cannibalisation case that justifies pruning. **A paginated URL holding head-term traffic needs a393deliberately chosen redirect target and its own monitoring; a blanket "301 to page 1" rule must394not cover it by default.** Where paginated URLs carry hreflang to sister domains' paginated URLs,395pruning ships on all of those domains together. Recipe: `references/pruning-evidence.md`396("Evidence for pruning").397398- Once items are genuinely pagination-independent, a **self-referencing canonical on each paginated399 page** is usually sufficient: the pages remain crawlable, the items no longer depend on them, and400 no signal is misdirected. Treat **canonicalising deep pages to page 1** as a secondary lever, used401 where the client specifically needs consolidation and accepts that it is a hint rather than a402 directive.403- Blocking pagination in robots.txt cuts the crawler's path through those pages entirely. Reserve404 it for cases with a demonstrated crawl-budget problem, and only after coverage has been verified.405- **301-redirecting `?p=N` to the base listing URL is the most aggressive lever of all** — not a406 gentler alternative to a robots block. The ladder orders by how much discoverability is removed,407 not by how forceful the mechanism feels: a robots-blocked page is merely unfetched, whereas a408 redirected one resolves to different content, so any item appearing only on page 2+ becomes409 unreachable the instant it ships, with no grace period and no partial behaviour. The property410 that makes the redirect attractive — a directive where a canonical is a hint — is the property411 that makes it final. It arrives in conversation looking like a tidy-up after link removal, or as412 the fix for the stranding risk of a robots block; it inherits none of that option's caution by413 default and deserves more. The escalation case that justifies it: a canonical-to-page-1414 demonstrably ignored on URLs with long indexation history, after coverage is verified. Include415 the page-1 parameter variant (`?p=1`) in the redirect — a pure duplicate that is typically never416 linked. Two operational requirements travel with it:417 - **Sequencing inverts.** The redirect goes in *before* the internal links are removed, not418 after. Once every fetch resolves to the base URL, removing the links is tidiness rather than a419 precondition, and the window in which URLs are unlinked-but-not-yet-consolidated never opens.420 - **The AJAX carve-out is a requirement of the dev instruction, not a caveat.** A421 progressively-enhanced listing typically fetches its own paginated URL space for partial422 responses — `?p=N&infScroll=1` or similar — so a redirect matching `[?&]p=\d+` alone catches the423 site's own requests and the listing silently serves page 1 forever. Enumerate every parameter424 the listing's scripts append (one template has been observed using two different such425 parameters, one per load-more control) and exempt each by name in the ticket. Where the426 rendered-page check (read 4) shows a scroll load requesting the plain paginated URL with no427 marker, there is nothing to exempt: the dev instruction first adds a marker parameter to the428 script, and the redirect exempts that marker.429- Before finalising the directive, **check current search engine guidance on pagination handling430 and rel next/prev**, and verify it against the vendor's dated changelog or announcement rather431 than the documentation page alone — documentation pages are edited silently, changelogs carry432 dates. Record the date and source of what was checked in the deliverable.433- Sequence the rollout: one or two categories first, monitor indexation and organic coverage of434 their items for a full crawl-and-index cycle, then extend. Define in advance what a regression435 looks like and what rolls it back.436437## Phase 5 — Governance438439Hand over the rules that keep the architecture true after the project ends.440441- The page-1 ceiling is an ongoing constraint. Specify a recurring check that flags subcategories442 which have grown past it, and the action when one does (split, or raise items-per-page).443- New items must land in at least one demand-backed subcategory at creation. Specify who enforces444 this and where.445- The curated facet-combination allow-list needs an owner and a review cadence. Without one it446 either ossifies or leaks.447- Re-run the Phase 1 item-level coverage diff periodically as the standing health metric: the448 uncovered count should stay at the accepted residual. On drifting listings this is the only449 measure that holds — a fixed list of items does not.450451## Anti-patterns452453- **Directive first.** Recommending noindex, canonical-to-page-1, a robots.txt block or a redirect454 on pagination before the orphan audit exists. This is the failure this skill exists to prevent.455- **Ranking the redirect below the robots block.** A 301 to page 1 is proposed as the tidy-up456 after link removal or as the safe fix for a robots block's stranding risk, and priced beside457 "canonical, but it works". Order levers by what becomes unreachable the moment they ship: a458 redirect removes more discoverability than any of them, and a redirect that does not exempt the459 listing's own AJAX parameters by name breaks the listing.460- **Defining the paginated set by regex.** Classifying sources from a URL pattern rather than from461 the links the templates emit. It admits parameter variants nothing links to and misses linked462 variants the pattern never anticipated, and both errors survive into every downstream number.463- **Asserting the separation between crawl path and control without reading the template — in464 either direction.** A belief about the purpose of a piece of markup is not evidence about the465 markup. Assuming the load-more link exists "only for crawlers" and is removable for free is one466 error, and it survives longest among the people who built the thing; assuming that because it is467 one element the removal must be expensive is the other, and it looks like rigour. Enumerate the468 anchors, read the handler, then price it.469- **Testing whether an item is linked by grepping for its id.** An id substring hits the JSON470 product payload, not the link; only an anchor-level match (`<a` + `href=`) answers "is this471 item linked from here". An impossible result (one item on more pages than a stable sort472 allows) is the instrument failing, not the site.473- **Shipping a per-URL orphan list without its vintage.** An undated list implicitly claims to474 describe the present. Structural conclusions survive an ageing crawl; per-URL remediation475 lists do not, and the stale rate is measurable on a sample before delivery.476- **Reading a crawl's silence as evidence.** Treating "not in the crawl data" as "not on the site"477 without stating the depth limit and seeding configuration and checking a non-traversal source.478 A depth-limited crawl of next+last pagination is blind to the middle of every long sequence.479- **Ranking by link deficiency without filtering dead targets.** The top of an unfiltered480 "worst-linked" list is mostly removed pages, so the finding is an artefact of the population481 definition rather than a fact about the site.482- **Reading an absence off an unfinished crawl or a partial response.** A listing's rounded 100%483 with URLs still queued, or one truncated fetch of a parent page, both produce a clean,484 plausible, wrong "not linked". Check the queue and the response before the diff.485- **Counting a self-link or a sister-domain switcher link as a non-paginated source.** Both apply486 to every item uniformly, so they never surface as outliers; "exactly one source" is then "zero487 plus self". Classify the weakest item's sources by hand before trusting the aggregate.488- **Joining a live fetch against a crawl snapshot on a rotating listing.** Every item that moved489 into or out of the sale listing between the two time points becomes a finding. Both sides from490 one snapshot, and the depth bound before any of it.491- **Ranking listings by size to find uncovered items.** Cross-cutting facets double-count the492 same items; measure item-level page-1 coverage directly with a validated grid extractor.493- **Pricing the items-per-page lever from the first cards, or treating "items not on page 1" as494 the scope of subcategory work.** The first cards are the above-the-fold exceptions — measure495 as a function of item count. Demand decides where a subcategory is worth having; the uncovered496 remainder gets a coverage backstop, not zero-demand pages.497- **Noindexing paginated pages that are still the only route to items.** Beyond the coverage498 problem, a noindexed page's onward links carry less weight over time — the orphaning is499 progressive and easy to miss.500- **Subcategories invented from the catalogue tree alone.** Splitting by whatever attribute the501 product data happens to carry, with no demand check, produces pages nobody searches for and a502 larger index to govern.503- **Auto-generating every facet combination.** Turning the whole facet matrix into indexable pages504 replaces a pagination problem with a combinatorial one. Curate.505- **Site-wide rollout on one category's evidence.** Coverage is per category; so is the rollout.506- **Ignoring per-market variance.** Signing off a market with a shallow catalogue and applying the507 same directive to a deep one re-orphans thousands of items.508- **Treating design sign-off as verification.** The design document is a plan. Only a re-crawl is509 evidence.510- **Dropping the ranking-surface framing.** Presented purely as index hygiene, the work reads as511 cost with no upside and stalls.512513## Pre-flight verification514515Before delivering, the agent runs this against its own output:516517- [ ] The paginated-source class was derived from the links the templates actually emit, with the518 emitted pattern, the linked page set (full list / next-only / next+last), each paginated page's519 canonical target and the status of any page-1 parameter variant recorded.520- [ ] Where a segmentation layer drives the analysis, pagination is its own segment class.521- [ ] A pagination-free crawl was actually run (or its absence is flagged as a blocking gap), not522 assumed from sitemap data alone.523- [ ] The crawl's depth limit and sitemap-seeding setting are stated, the unreachable band of the524 sequence is reasoned about from the link pattern, and no "nothing depends on pagination"525 conclusion rests on crawl data alone.526- [ ] Both endpoints of any link ranking are constrained to live 200 URLs, and the top rows of any527 "worst-linked" ordering were inspected before the number was quoted.528- [ ] The item inventory came from at least two reconciled sources, at least one of them not529 assembled by traversal.530- [ ] The crawl's queued/waiting count was read from its live progress state before any absence531 diff; the deliverable states that the queue was empty or how many queued URLs were532 subtracted from the absent set.533- [ ] Every per-item source count excludes self-links, labels structural sources separately534 (reachability and editorial linking reported as two numbers), counts same-domain sources535 only in a multi-domain project, takes both sides of any subtraction from one snapshot, and536 applies the depth-≥3 bound; the weakest item's sources were classified by hand.537- [ ] A depth bound passed via a depth-1 HTML sitemap is reported as reachability only, with the538 sitemap named as sole non-paginated source off page 1; analytics for paginated URLs keep539 the query string (never GA4's "Landing page").540- [ ] Listing stability was tested before any pagination-only set was presented as a fixed541 remediation list, and where listings drift the remedy is structural.542- [ ] Item-level page-1 coverage was measured from same-moment page-1 grids with the extractor543 validated against a declared per-listing count, and the per-category, per-market table is544 a breakdown of that set, with counts and shares, not a size ranking of listings.545- [ ] The crawl's completion date is resolved and recorded, and the orphan table carries an546 as-of date. Any per-URL list shipped for item-by-item action was live-v547548…(truncated)
Run npx get-skill add rebelytics/pagination-independence-architecture in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Design category and listing architecture so that a site's indexable value does not depend on paginated pages, using demand-backed indexable subcategories as the alternative to deep pagination. Load this skill when the user asks whether to noindex, canonicalise, block or prune paginated URLs (`?page=`, `?p=`, `/page/2/`); when they mention pagination handling, page-2+ indexation, infinite scroll and crawlability, orphaned product or article pages, "products only reachable from page 7", crawl budget wasted on listing pages, thin or duplicate category pages, faceted navigation index bloat, or splitting a large category into subcategories. Also load for "how do we cut paginated URLs without losing products" and listing-page taxonomy redesign for large catalogues. Applies to e-commerce catalogues, editorial archives and directories. It is listed under Finance & Business on SkillMD.
This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free. This skill is licensed under CC-BY-4.
rebelytics (@rebelytics) published this skill. Their other Agent Skills are listed on their SkillMD profile.