# Creative Pattern Validator > Creative Pattern Validator - Skill: `raneqbarber/creative-pattern-validator` (Agent Skill) - Install (CLI): `npx skillmds@latest add raneqbarber/creative-pattern-validator` - Raw SKILL.md: https://api.skillmd.com/api/skills/raneqbarber/creative-pattern-validator/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: raneqbarber (https://skillmd.com/u/raneqbarber) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/raneqbarber/creative-pattern-validator --- # Creative Pattern Validator Takes a list of creative patterns someone found — "soft CTAs with large type performed 34% above benchmark" — and decides which ones are real. This is the safeguard between pattern mining and briefing. Without it, a creative team gets briefed on coin flips, produces work against them, and the work underperforms for reasons nobody can trace back to the analysis that caused it. --- ## 1 · Operating contract - **Never validate a pattern without the underlying ad counts.** A lift percentage alone cannot be assessed. Halt and ask. - **Never test at impression level.** The unit of analysis is the ad. This is not a preference — see §5. Getting it wrong makes every pattern significant, including the false ones. - **Never apply a single-test threshold to a multiple-test problem.** If more than one pattern was examined, the threshold must be corrected. Report how many were tested. - **Never present a surviving pattern as proven.** Surviving means *hard to explain as chance*. It does not mean large, important, or causal. Say so in the output. - **Never soften a verdict on request.** If the user asks for the cut list to be shorter, explain what the threshold means and leave the verdicts alone. - **Never claim a result you did not compute.** Where the runtime cannot run the arithmetic, emit the formula and the inputs and say the number is uncomputed. - **Never claim one global FDR guarantee.** Correction runs within declared families. Say which. - **Never treat ad-level independence as established.** Ads inside a campaign share conditions. Report the cluster-robustness tag on every finding, including `too few clusters`. - **Every finding carries a tier** from `reference/evidence-standards.md`. A finding without its tier is presented as stronger than it is. --- ## 2 · When to run this Run immediately after `creative-pattern-miner`, before anything reaches a brief, a guidelines document, or a slide. Also run when someone hands you findings from any other source — an agency deck, a platform's "insights" tab, a colleague's spreadsheet — and you need to know which of them to act on. Do not run this to compare two specific ads. That is an A/B test with one comparison, and the multiple-comparison machinery here does not apply. --- ## 3 · Required inputs and preflight ### Must have | Input | Notes | |---|---| | `patterns` | Each with: the trait or trait combination, the group's value, the comparison group's value | | `ads_in_group` | Number of distinct **ads** carrying the pattern — not impressions, not rows | | `ads_in_comparison` | Number of distinct ads in the same segment without it | | `per_ad_values` | The ad-level metric values for both groups. Group means alone are not enough | | `patterns_tested_total` | How many patterns were examined to produce this list, including the ones already discarded | ### Improves the output Distinct campaign and month counts per pattern, spend share, the segment definition, and whether patterns were enumerated as singles, pairs and trios (which determines the test families). ### Preflight 1. **Ad counts present.** *Halt if absent.* Lift with no denominator cannot be assessed. 2. **Unit of analysis.** Confirm counts are distinct ads. *Halt if they are impressions or ad-days.* 3. **Total tested known.** If `patterns_tested_total` is missing, ask. If the user cannot supply it, proceed but mark every verdict `correction: uncorrected` and state prominently that the threshold is not trustworthy. 4. **Per-ad values present.** *Halt if only group aggregates were supplied* — a rank test needs the distribution, not the mean. 5. **Comparison group defined.** Confirm the comparison is the same platform and ad objective without the trait. A pattern compared against the whole account is confounded by environment. --- ## 4 · The procedure ### Step 1 — Establish the test families Group the patterns by how they were enumerated: single traits, pairs, trios. Each is its own family. Correction is applied **within each family separately**, because a family of 366 singles and a family of 37,634 trios are different-sized haystacks and deserve different thresholds. Report the size of each family. ### Step 2 — Test each pattern For each pattern, ask one question: *if this trait made no difference at all, how often would we see a gap this big by chance?* Use a rank-based test on the per-ad values — Mann-Whitney U for a simple two-group comparison, stratified (van Elteren) where segments are involved. Rank tests are used rather than t-tests because ad-level performance is heavily skewed and small groups are common. The output is a p-value. Interpret it as: | p-value | Reads as | Verdict | |---|---|---| | 0.90 | 9 times in 10, even with no real effect | ignore | | 0.36 | about 1 time in 3 | ignore | | 0.05 | about 1 time in 20 | interesting | | 0.002 | about 1 time in 500 | hard to call luck | ### Step 3 — Correct for how many things were tested "p below 0.05" means one in twenty by luck. Test 44,587 patterns and roughly **2,200 will clear that bar with no real effect behind them**, every one of them looking like a finding. Apply Benjamini-Hochberg within each family at `q = 0.10`, meaning no more than 10% of what you publish is expected to be a false alarm: 1. **Declare the families before looking at results.** Singles, pairs, trios. Writing them down after seeing which findings survive is how a correction procedure becomes decoration. 2. Sort the family's p-values ascending, rank them `1…m`. 3. Find the largest rank `k` where `p(k) ≤ (k / m) × q`. 4. Everything up to and including rank `k` survives. Everything after is cut. 5. **Report an adjusted q-value per finding**, not just the family cutoff: `q(i) = min over j ≥ i of ( p(j) × m ÷ j )`, capped at 1. 6. **Report the effective cutoff** too — the p-value at rank `k`. On a real 44,587-pattern run this moved the bar from 0.05 to about 0.023. Per-finding q-values matter because a shared cutoff is unstable: adding a few ads next month moves the threshold and flips findings, and nobody can see why. `p = 0.018 · q = 0.071` travels with the finding and stays legible. **State the guarantee precisely.** FDR is controlled at `q` **within each declared family**, not across the union of everything published. Correcting separately is a defensible product decision — a family of 37,000 trios would otherwise annihilate a family of 366 singles — but it is not one global guarantee and must not be described as one. ### Step 3b — Test robustness to clustering Benjamini-Hochberg controls chance selection. It does not fix the fact that ads are **not independent observations**: ads inside a campaign share targeting, budget, bid strategy, optimization state and timing. A rank test that assumes independence will be too optimistic. Run a **campaign-level permutation test** on every surviving pattern: 1. Hold the segment structure fixed (platform × ad objective × period). 2. Shuffle the pattern label **at campaign level**, not at ad level — every ad in a campaign moves together, because the campaign is the unit that actually shares conditions. 3. Recompute the effect statistic — median per-ad lift difference. 4. Repeat 2,000+ times to build the null distribution. 5. Compare the observed effect against it. Where campaign is too coarse or unavailable, use the next meaningful shared unit: campaign × month, asset family, or copy family. Say which was used. Tag each finding: | Tag | Meaning | |---|---| | `campaign-robust` | Cleared the clustered permutation test | | `not robust to clustering` | Survived FDR but not the cluster test — likely one campaign's effect | | `too few clusters` | Under 5 distinct campaigns. Cannot assess. Report, do not claim | This is conservative by design. It asks whether the pattern would still look unusual if the effective independent units were campaigns rather than ads. The generalization gate in Step 4 is a heuristic guard against the same problem; this replaces the heuristic with a measurement. ### Step 4 — Apply the generalization gate Statistical survival is not enough. A pattern that is real inside one forgiving campaign is not guidance. **Test stratified, not pooled.** Use a segment-blocked rank test — van Elteren — which ranks each ad only against others *in its own segment*, then combines across segments. A pattern that lives in an easy segment earns nothing from living there. Pooling first and testing second lets segment composition masquerade as a trait effect. **The gate.** A pattern must be: - present in **3 or more segments** - with **no single segment supplying more than 70%** of its ads - across **3 or more distinct campaigns** - across **3 or more distinct months** A pattern failing this is not cut — it is reclassified as `local`, with the segment named. Local findings are usable; they just cannot be written as portfolio guidance. **Two safeguards when pooling is unavoidable.** Splitting by platform × ad objective costs power — 450 ads across six segments leaves ~75 per cell, and most findings then rest on five or six ads. A pooled read buys that power back, because the lift index already divides the environment out: a CAS of 1.2 means 20% above expectation whether it is LinkedIn awareness or Meta sales, which raw CTR could never do. But naive pooling invites Simpson's paradox — a trait can look good purely by being concentrated in a forgiving segment. So pooled analysis requires both: 1. **The stratified test above**, so segment ease earns nothing. 2. **The generalization gate**, so a pattern must appear broadly before it can be pooled at all. With both applied, pooled findings rest on 22–193 ads instead of five, which is the difference between a portfolio rule and an anecdote. ### Step 5 — Apply the actionability gate A trait appearing on more than **85% of the portfolio** describes what you already do. It cannot advise anything, however real its lift. Mark it `actionable: no` and keep it out of briefs. This gate catches the most common false victory in creative analysis: "high production value performs well" on a portfolio that is 91% high production value. ### Step 6 — Assign confidence | Tier | Requires | How to use it | |---|---|---| | **High** | ≥10 ads **and** ≥10% of spend | Recommend directly. Safe to default into briefs. | | **Medium** | 5–9 ads, or 3–9% of spend | Suggest with a hedge. Pair with a second signal. | | **Low** | below that | A hypothesis to test, never a recommendation. | Be honest when most of a set lands at Medium. That is a function of dataset size and improves as more months land — it is not a flaw to hide. ### Step 7 — Return the verdict list Brief only what passes all three: **significant AND generalizes AND actionable.** State how many of the original candidates that is. On a real run it was 16 of 201 survivors. --- ## 5 · Computation rules **Non-negotiable.** ### Count ads, not impressions The single most important rule in this skill. Impressions inside one ad share a creative and an audience, so they are not independent observations. Test at impression level and a 16-million-row sample makes a 0.01% CTR gap "significant". The demonstration to run on your own data: take one segment, split it in half **at random**, and test. There is no real effect — it is the same ads. At impression level that split returns `p = 1e-21`. Testing per-ad returns a properly calibrated result. ### Calibrate against noise before trusting the procedure Run 400 random splits of your own data. Roughly 5% should land under p<0.05. If far more do, the unit of analysis is wrong. If far fewer, the test is too conservative for the data. A measured run of this procedure produced 6.2%, which is what chance should produce. Publish that number alongside the findings. ### Recompute ratios from totals A group's rate is `Σ numerator ÷ Σ denominator` for the group. Never average the ad-level ratios — that weights a $50 test ad the same as a $50,000 evergreen. Within a single quarter the two methods tend to agree; across a longer window they diverge by up to 26%. ### Correct within families, not across Singles, pairs and trios are separate families. Pooling them applies the trio family's brutal correction to the singles and cuts real findings. ### Drop redundant patterns If a pattern's ad set is identical to a simpler pattern's, keep the simpler one. A pair that contains exactly the same ads as one of its component traits is that trait wearing a longer name. --- ## 6 · Output contract Structured output: `reference/output-schema.json`. Markdown fallback: ``` PATTERN VALIDATION · 2026-Q1 · 446 ads families declared in advance: singles 366 · pairs 6,587 · trios 37,634 correction: Benjamini-Hochberg q=0.10, WITHIN EACH FAMILY (not across the union) effective cutoff: singles p ≤ 0.023 · pairs p ≤ 0.008 · trios p ≤ 0.002 noise calibration: 6.2% of 400 random splits under p<0.05 — as expected cluster test: campaign-level permutation, 2,000 shuffles BRIEF THESE — significant · generalizes · actionable · campaign-robust pattern ads segs camps lift p q robustness soft caption CTA 62 5 9 +28% 0.0014 0.041 campaign-robust customer testimonial 42 4 7 +14% 0.0000 0.003 campaign-robust AVOID THESE no CTA on the image 22 3 6 −41% 0.0130 0.088 campaign-robust heavy on-image text 102 3 4 −19% 0.0003 0.011 campaign-robust SURVIVED FDR, FAILED CLUSTERING — do not brief urgency framing 31 3 2 +21% 0.0110 0.079 not robust → 74% of its ads sit in two campaigns. The pattern is those campaigns, not the trait. A test candidate, not guidance. LOCAL — real, but only in one place urgency_framing · 86% of its ads sit in one campaign · not portfolio guidance NOT ACTIONABLE — describes you, cannot advise you high_production_value · 91% of portfolio · +16% lift on 380 ads CUT — 185 patterns, could not be told apart from chance ``` --- ## 7 · Failure and degraded states | Situation | What to do | |---|---| | Ad counts missing | Halt. Name the patterns missing them. | | Counts are impressions | Halt. Explain the unit-of-analysis problem and ask for per-ad values. | | `patterns_tested_total` unknown | Proceed uncorrected, mark every verdict `correction: uncorrected`, and lead the output with the warning. | | Only group means supplied | Halt for rank tests. Offer a descriptive summary with no verdicts, clearly labeled as not validated. | | A family has fewer than 5 patterns | Skip correction for that family, say so, and report raw p-values. | | Every pattern is cut | Report that plainly. An empty brief list is a real and useful result — it means the dataset is too small to support guidance yet. | --- ## 8 · Worked example **Input:** 201 candidate patterns from 446 ads. 44,587 patterns were tested to produce them. **Two patterns, worked:** `cs_05 = none` in LinkedIn Decision. Lift index 1.16 — above the 1.10 bar the original framework used — on 51 ads. Not a thin sample. But **p = 0.94**: a gap that size appears 96 times in 100 with no real effect behind it. The old threshold would have briefed it. **Cut.** `vd_07 = large` + `vd_09 = soft` in LinkedIn Awareness. Lift index 1.34 on **6 ads**. Far fewer ads, and by headcount it looks like the weaker finding. But **p = 0.0009**. **Kept** — at Medium confidence, because 6 ads is under the High threshold. The lesson to carry: it is the size of the gap against the noise that matters, not the headcount. --- ## 9 · Self-check 1. Were counts distinct ads, never impressions or ad-days? 2. Was correction applied within each declared family, with families declared before results? 3. Does every finding carry an adjusted q-value, not just a family cutoff? 4. Was the campaign-level permutation run, with a robustness tag on every surviving finding? 5. Is `patterns_tested_total` stated in the output? 6. Does every surviving pattern carry a confidence tier with its ad count and spend share? 7. Were the generalization and actionability gates applied, with failures reclassified rather than silently dropped? 8. Does the output say what surviving means — hard to explain as chance, not proven, not causal? 9. If the noise calibration was run, is the number published? 10. Does every published finding carry its evidence tier? --- *Executes the evidence standards of the Performance Creative Intelligence measurement system. MIT licensed.*