Ecommerce Cro Experimentation

E-Commerce CRO Experimentation

Overview

This skill covers how to run a valid conversion experiment end-to-end on a WooCommerce/WordPress store — from deciding whether a test can even reach significance, through design, instrumentation, cache-safe delivery, and readout. It is the HOW-to-test layer; the surrounding skills own the other layers.

This skill is a thin adapter onto woocommerce-developer's GA4 event-mapping table — it references that table, it never reproduces it.

Traffic-Power Gate — run this FIRST

Before designing anything, decide whether a classical A/B test can reach significance inside a window your store can hold stable. Do not default to a fixed-horizon A/B test for a low-volume store.

Inputs: baseline conversion p; weekly eligible units (unique visitors or accounts — not sessions); relative MDE r (the smallest lift worth shipping).

Planning approximation (equal arms, α = 0.05 two-sided, 80% power): n_arm ≈ 15.7 × (1 − p) / (p × r²), total ≈ 2 × n_arm. Sample grows ~`1/r²— halving the MDE needs about **4×** the traffic.weeks ≈ total eligible units ÷ weekly eligible units`.

Time-to-significance (worked examples):

Decision branch:

• Powered — horizon fits an 8-12 week window in which traffic mix, implementation, and promotions stay stable → run a classical fixed-horizon test (see Experiment Design).
• Under-powered — horizon exceeds that window → do not run a fixed-horizon A/B. Pick a named fallback:
  1. Sequential or Bayesian design (alpha-spending / confidence sequences / mSPRT, or a posterior-loss rule) — the prior, threshold, and stopping rule must be set in advance; neither method manufactures information from thin traffic.
  2. Test a bigger change / larger MDE, or switch the primary to a higher-frequency proximal metric (add-to-cart rate) with revenue guardrails.
  3. Pre/post with guardrail metrics — a weaker causal claim; control for seasonality and confounds.
  4. Adopt a well-evidenced best practice without testing (from ecommerce-growth / conversion-psychology) and monitor guardrails.
  5. Time-boxed readout — work with the data actually available. Run for the window the operator has, then report what that data supports, at the confidence it supports, and state what a longer window would add. This is the default when the operator needs a decision now; the others are refinements on it.

Under-powered never means "no answer". A horizon that does not fit is a statement about certainty, not about usefulness — the operator still has to decide something this quarter, and "come back in 49 months" is not a decision aid. Always deliver the best available read, labelled:

DIRECTIONAL (under-powered) — 6 weeks, n≈1,180 units/arm
  observed:     +14% add-to-cart (95% CI −3% to +31%)
  supports:     ship it if the change is cheap and guardrails hold — the downside case is small
  does NOT support: a revenue-per-visitor claim, or attributing a specific lift figure
  a further 10 weeks would: narrow the CI to roughly ±8% and separate +14% from zero

State the direction, the interval, what it does and does not license, and the cost of more certainty. Then let the operator choose. An honest wide interval is worth more than silence, and far more than a confident number the data cannot carry.

Still never run a classical fixed-horizon test whose computed horizon exceeds ~8-12 weeks — the store will have changed underneath it before it concludes. That is a constraint on the test design, not permission to withhold the analysis.

Hypothesis Framing

• Evidence sources: heatmaps and session replays (Clarity — rage/dead clicks, scroll depth), funnel drop-offs (GA4 — see seo-data-analyst recipes), reviews and support tickets.
• Template: Observation → Change → Predicted effect → Metric. e.g. "Users rage-click the spec tabs (obs) → surface key specs as bullets above the fold (change) → add-to-cart rate rises (predicted) → add_to_cart / view_item (metric)."
• Prioritize with ICE (Impact, Confidence, Ease) or PIE (Potential, Importance, Ease). One hypothesis per test.

Experiment Design

• One primary metric + guardrails. Predefine guardrails: revenue/visitor, AOV, margin, refunds/cancellations, checkout errors, payment success rate, and Core Web Vitals.
• Sample size comes from the Traffic-Power Gate. Pre-register MDE, α, power, duration, and the primary metric before launch.
• Conventions: two-sided α = 0.05 (95%), 80% power (β = 0.20); use 90% power or a stricter α when a wrong decision is expensive.
• Anti-peeking: a fixed-horizon test is read once, at the planned horizon. To look early or stop early you must use a sequential design — repeatedly stopping when p < .05 inflates the Type-I error rate. Watching for operational breakage is fine; making the efficacy call early is not.
• Duration: run whole weeks / whole business cycles; never stop mid-week. Cover weekday + weekend, paydays, and promo cycles.
• Discipline: one change per variant (or a deliberate factorial design). Check sample-ratio mismatch (SRM). Randomize on visitor/account, never session. Avoid post-hoc segmentation (it is a multiple-comparisons trap).

Instrumentation — GA4 via GTM/dataLayer on Woo

woocommerce-developer §Analytics Integration owns the Woo-action → GA4-event mapping table — use it; do not copy it here. This section adds the procedure around it:

• One data contract: stable item_id, numeric price/quantity, currency, and a unique transaction_id; decide the randomization/analysis unit (visitor / account / session) up front.
• Push, don't scrape. Initialize window.dataLayer before the GTM container. Push atomic objects carrying the exact event name plus its ecommerce object — never read values from rendered HTML. Clear the previous ecommerce object before each push (GTM's data model persists).
• Fire at the real state transition. For AJAX/block carts, push after WooCommerce confirms success, not on the click.
• Purchase from truth. Emit purchase from the authoritative completed-order state; send refunds against the original transaction_id. Stamp experiment_id + variant onto both exposure and outcome events so arms can be analyzed independently.
• QA: GTM Preview / Tag Assistant (push order, consent state, fired vs blocked tags, outgoing hits) → then GA4 DebugView for event- and item-level parameters.
• De-duplicate at source: use GTM or direct gtag.js, not both; bind each interaction once; guard confirmation-page refreshes; keep transaction_ids unique. GA4 ignores a repeated transaction_id, but ordinary funnel events have no equivalent auto-dedup.
• Consent Mode v2: set denied-by-default consent (ad_storage, analytics_storage, ad_user_data, ad_personalization) before other tags — normally a CMP template on GTM's Consent Initialization trigger — then update after the visitor's choice. EEA advertising enforcement began early March 2024. Consent Mode signals a choice; it is not a CMP and not legal advice.
• Express wallets bypass the funnel — segment before you read it. Apple Pay / Google Pay / Shop Pay / PayPal express purchases launched from the PDP or cart never fire begin_checkout, and wallets are ~65% of mobile conversions. Before running any checkout experiment: confirm checkout_type (classic | express_wallet) is stamped on checkout and purchase events, and analyse the two funnels separately. A blended checkout-completion rate on a wallet-enabled store is wrong for most mobile orders, and it will make a checkout-page variant look like it moved a metric it never touched. Implementation detail: woocommerce-developer §Analytics Integration.
• Qualitative layer: Clarity/heatmaps explain the WHY behind a win or a flat result — see seo-data-analyst.

Cache-Safe Variation Delivery

Three delivery models on a cached WordPress site — pick per the change and the store's caching stack:

• Server-side is safe only if the full-page cache key also varies on a small-valued experiment cookie. Without that vary, the first cached A or B response is served to everyone.
• Edge: assign before the cache lookup, store a stable first-party A/B cookie, and rewrite to /control/... | /test/... or add the bucket to the cache key. Keep cardinality to A/B — never a user ID — and never cache a personalized Set-Cookie object as the shared page.
• Anti-flicker vs performance: hiding content until assignment cuts the control-to-variant flash and CLS but adds a blank interval and can worsen LCP; a synchronous snippet cuts flicker but delays parsing; async/defer isolates failure but risks flash. Always use a short fail-open timeout. Measure RUM + Web Vitals by arm (include the variant's own images/fonts/layout). CWV thresholds live in seo-structure-architect.
• WooCommerce cache specifics: DONOTCACHEPAGE=true bypasses the page cache (runs PHP → no variant leakage) but forfeits cache performance and does not control an upstream CDN/host cache. WP Rocket auto-uncaches Cart/Checkout/My-Account/Woo-REST while still caching product/category/home — "Never Cache Cookies" bypasses caching, it does not create one cached object per variant. LiteSpeed Cache (LSCache 6.0+) "Vary Cookies" can create per-variant cached objects and propagate the vary to QUIC.cloud (never vary on a unique visitor ID). Purge WP + host + CDN caches on every test start/stop/change, and exclude the experiment loader from JS combine/defer/delay unless the vendor supports it.

2026 tool landscape (Google Optimize sunset 30 Sep 2023): WP-native / Woo-aware options include Nelio A/B Testing (v8.x, Woo product/order goals), Convert (WP plugin + Woo revenue), VWO (official WP plugin tracks Woo events), and Thrive Optimize (landing pages). Developer/full-stack + edge options include GrowthBook (PHP + Cloudflare/Fastly/Lambda@Edge SDKs), Optimizely, AB Tasty/Flagship, Statsig, PostHog, and Harness FME (formerly Split). Generic client-side tools install easily via <head>/GTM, but Woo-event measurement is a separate integration.

QA, Rollout & Readout

Pre-launch QA checklist:

• Both arms render correctly on mobile + desktop, logged-in + guest, with caches warm.
• Instrumentation fires once per interaction in both arms (confirm in DebugView); SRM check is green; guardrails are wired.
• Consent state is correct; no console errors; the fail-open timeout works.

Ramp: start at a small traffic share to catch breakage, then move to full allocation and keep it fixed.

Readout decision rules (only after the pre-registered horizon or sequential boundary):

• Ship — primary metric wins with significance and no guardrail regressed.
• Iterate — flat/ambiguous but directional and cheap to refine.
• Kill — it loses or breaks a guardrail. A flat result is a valid, informative outcome, not a failure.

Log every experiment (hypothesis, dates, allocation, primary + guardrail results, decision). The log is your institutional memory and prevents re-running settled questions.

Ethics Guardrails

• No dark-pattern variants: fake scarcity/urgency, confirm-shaming, hidden costs, or pre-ticked add-ons are out of bounds even if they "win". See conversion-psychology ethical-urgency rules; back urgency/social-proof with real data (the UK CMA actively investigates fake urgency).
• Do not test consent flows into non-compliance, and keep accessibility parity across arms.

Anti-Patterns