Creative Pattern Validator

Creative Pattern Validator

Takes a list of creative patterns someone found — "soft CTAs with large type performed 34% above benchmark" — and decides which ones are real.

This is the safeguard between pattern mining and briefing. Without it, a creative team gets briefed on coin flips, produces work against them, and the work underperforms for reasons nobody can trace back to the analysis that caused it.

1 · Operating contract

• Never validate a pattern without the underlying ad counts. A lift percentage alone cannot be assessed. Halt and ask.
• Never test at impression level. The unit of analysis is the ad. This is not a preference — see §5. Getting it wrong makes every pattern significant, including the false ones.
• Never apply a single-test threshold to a multiple-test problem. If more than one pattern was examined, the threshold must be corrected. Report how many were tested.
• Never present a surviving pattern as proven. Surviving means hard to explain as chance. It does not mean large, important, or causal. Say so in the output.
• Never soften a verdict on request. If the user asks for the cut list to be shorter, explain what the threshold means and leave the verdicts alone.
• Never claim a result you did not compute. Where the runtime cannot run the arithmetic, emit the formula and the inputs and say the number is uncomputed.
• Never claim one global FDR guarantee. Correction runs within declared families. Say which.
• Never treat ad-level independence as established. Ads inside a campaign share conditions. Report the cluster-robustness tag on every finding, including too few clusters.
• Every finding carries a tier from reference/evidence-standards.md. A finding without its tier is presented as stronger than it is.

2 · When to run this

Run immediately after creative-pattern-miner, before anything reaches a brief, a guidelines document, or a slide.

Also run when someone hands you findings from any other source — an agency deck, a platform's "insights" tab, a colleague's spreadsheet — and you need to know which of them to act on.

Do not run this to compare two specific ads. That is an A/B test with one comparison, and the multiple-comparison machinery here does not apply.

3 · Required inputs and preflight

Must have

Improves the output

Distinct campaign and month counts per pattern, spend share, the segment definition, and whether patterns were enumerated as singles, pairs and trios (which determines the test families).

Preflight

1. Ad counts present. Halt if absent. Lift with no denominator cannot be assessed.
2. Unit of analysis. Confirm counts are distinct ads. Halt if they are impressions or ad-days.
3. Total tested known. If patterns_tested_total is missing, ask. If the user cannot supply it, proceed but mark every verdict correction: uncorrected and state prominently that the threshold is not trustworthy.
4. Per-ad values present. Halt if only group aggregates were supplied — a rank test needs the distribution, not the mean.
5. Comparison group defined. Confirm the comparison is the same platform and ad objective without the trait. A pattern compared against the whole account is confounded by environment.

4 · The procedure

Step 1 — Establish the test families

Group the patterns by how they were enumerated: single traits, pairs, trios. Each is its own family. Correction is applied within each family separately, because a family of 366 singles and a family of 37,634 trios are different-sized haystacks and deserve different thresholds.

Report the size of each family.

Step 2 — Test each pattern

For each pattern, ask one question: if this trait made no difference at all, how often would we see a gap this big by chance?

Use a rank-based test on the per-ad values — Mann-Whitney U for a simple two-group comparison, stratified (van Elteren) where segments are involved. Rank tests are used rather than t-tests because ad-level performance is heavily skewed and small groups are common.

The output is a p-value. Interpret it as:

Step 3 — Correct for how many things were tested

"p below 0.05" means one in twenty by luck. Test 44,587 patterns and roughly 2,200 will clear that bar with no real effect behind them, every one of them looking like a finding.

Apply Benjamini-Hochberg within each family at q = 0.10, meaning no more than 10% of what you publish is expected to be a false alarm:

1. Declare the families before looking at results. Singles, pairs, trios. Writing them down after seeing which findings survive is how a correction procedure becomes decoration.
2. Sort the family's p-values ascending, rank them 1…m.
3. Find the largest rank k where p(k) ≤ (k / m) × q.
4. Everything up to and including rank k survives. Everything after is cut.
5. Report an adjusted q-value per finding, not just the family cutoff: q(i) = min over j ≥ i of ( p(j) × m ÷ j ), capped at 1.
6. Report the effective cutoff too — the p-value at rank k. On a real 44,587-pattern run this moved the bar from 0.05 to about 0.023.

Per-finding q-values matter because a shared cutoff is unstable: adding a few ads next month moves the threshold and flips findings, and nobody can see why. p = 0.018 · q = 0.071 travels with the finding and stays legible.

State the guarantee precisely. FDR is controlled at q within each declared family, not across the union of everything published. Correcting separately is a defensible product decision — a family of 37,000 trios would otherwise annihilate a family of 366 singles — but it is not one global guarantee and must not be described as one.

Step 3b — Test robustness to clustering

Benjamini-Hochberg controls chance selection. It does not fix the fact that ads are not independent observations: ads inside a campaign share targeting, budget, bid strategy, optimization state and timing. A rank test that assumes independence will be too optimistic.

Run a campaign-level permutation test on every surviving pattern:

1. Hold the segment structure fixed (platform × ad objective × period).
2. Shuffle the pattern label at campaign level, not at ad level — every ad in a campaign moves together, because the campaign is the unit that actually shares conditions.
3. Recompute the effect statistic — median per-ad lift difference.
4. Repeat 2,000+ times to build the null distribution.
5. Compare the observed effect against it.

Where campaign is too coarse or unavailable, use the next meaningful shared unit: campaign × month, asset family, or copy family. Say which was used.

Tag each finding:

This is conservative by design. It asks whether the pattern would still look unusual if the effective independent units were campaigns rather than ads. The generalization gate in Step 4 is a heuristic guard against the same problem; this replaces the heuristic with a measurement.

Step 4 — Apply the generalization gate

Statistical survival is not enough. A pattern that is real inside one forgiving campaign is not guidance.

Test stratified, not pooled. Use a segment-blocked rank test — van Elteren — which ranks each ad only against others in its own segment, then combines across segments. A pattern that lives in an easy segment earns nothing from living there. Pooling first and testing second lets segment composition masquerade as a trait effect.

The gate. A pattern must be:

• present in 3 or more segments
• with no single segment supplying more than 70% of its ads
• across 3 or more distinct campaigns
• across 3 or more distinct months

A pattern failing this is not cut — it is reclassified as local, with the segment named. Local findings are usable; they just cannot be written as portfolio guidance.

Two safeguards when pooling is unavoidable. Splitting by platform × ad objective costs power — 450 ads across six segments leaves ~75 per cell, and most findings then rest on five or six ads. A pooled read buys that power back, because the lift index already divides the environment out: a CAS of 1.2 means 20% above expectation whether it is LinkedIn awareness or Meta sales, which raw CTR could never do.

But naive pooling invites Simpson's paradox — a trait can look good purely by being concentrated in a forgiving segment. So pooled analysis requires both:

1. The stratified test above, so segment ease earns nothing.
2. The generalization gate, so a pattern must appear broadly before it can be pooled at all.

With both applied, pooled findings rest on 22–193 ads instead of five, which is the difference between a portfolio rule and an anecdote.

Step 5 — Apply the actionability gate

A trait appearing on more than 85% of the portfolio describes what you already do. It cannot advise anything, however real its lift. Mark it actionable: no and keep it out of briefs.

This gate catches the most common false victory in creative analysis: "high production value performs well" on a portfolio that is 91% high production value.

Step 6 — Assign confidence

Be honest when most of a set lands at Medium. That is a function of dataset size and improves as more months land — it is not a flaw to hide.

Step 7 — Return the verdict list

Brief only what passes all three: significant AND generalizes AND actionable. State how many of the original candidates that is. On a real run it was 16 of 201 survivors.

5 · Computation rules

Non-negotiable.

Count ads, not impressions

The single most important rule in this skill. Impressions inside one ad share a creative and an audience, so they are not independent observations. Test at impression level and a 16-million-row sample makes a 0.01% CTR gap "significant".

The demonstration to run on your own data: take one segment, split it in half at random, and test. There is no real effect — it is the same ads. At impression level that split returns p = 1e-21. Testing per-ad returns a properly calibrated result.

Calibrate against noise before trusting the procedure

Run 400 random splits of your own data. Roughly 5% should land under p<0.05. If far more do, the unit of analysis is wrong. If far fewer, the test is too conservative for the data. A measured run of this procedure produced 6.2%, which is what chance should produce.

Publish that number alongside the findings.

Recompute ratios from totals

A group's rate is Σ numerator ÷ Σ denominator for the group. Never average the ad-level ratios — that weights a $50 test ad the same as a $50,000 evergreen. Within a single quarter the two methods tend to agree; across a longer window they diverge by up to 26%.

Correct within families, not across

Singles, pairs and trios are separate families. Pooling them applies the trio family's brutal correction to the singles and cuts real findings.

Drop redundant patterns

If a pattern's ad set is identical to a simpler pattern's, keep the simpler one. A pair that contains exactly the same ads as one of its component traits is that trait wearing a longer name.

6 · Output contract

Structured output: reference/output-schema.json. Markdown fallback:

PATTERN VALIDATION · 2026-Q1 · 446 ads
families declared in advance: singles 366 · pairs 6,587 · trios 37,634
correction: Benjamini-Hochberg q=0.10, WITHIN EACH FAMILY (not across the union)
effective cutoff: singles p ≤ 0.023 · pairs p ≤ 0.008 · trios p ≤ 0.002
noise calibration: 6.2% of 400 random splits under p<0.05 — as expected
cluster test: campaign-level permutation, 2,000 shuffles

BRIEF THESE — significant · generalizes · actionable · campaign-robust
  pattern                   ads segs camps   lift      p      q    robustness
  soft caption CTA           62    5     9   +28%  0.0014  0.041   campaign-robust
  customer testimonial       42    4     7   +14%  0.0000  0.003   campaign-robust

AVOID THESE
  no CTA on the image        22    3     6   −41%  0.0130  0.088   campaign-robust
  heavy on-image text       102    3     4   −19%  0.0003  0.011   campaign-robust

SURVIVED FDR, FAILED CLUSTERING — do not brief
  urgency framing            31    3     2   +21%  0.0110  0.079   not robust
  → 74% of its ads sit in two campaigns. The pattern is those campaigns,
    not the trait. A test candidate, not guidance.

LOCAL — real, but only in one place
  urgency_framing   ·  86% of its ads sit in one campaign  ·  not portfolio guidance

NOT ACTIONABLE — describes you, cannot advise you
  high_production_value  ·  91% of portfolio  ·  +16% lift on 380 ads

CUT — 185 patterns, could not be told apart from chance

7 · Failure and degraded states

8 · Worked example

Input: 201 candidate patterns from 446 ads. 44,587 patterns were tested to produce them.

Two patterns, worked:

cs_05 = none in LinkedIn Decision. Lift index 1.16 — above the 1.10 bar the original framework used — on 51 ads. Not a thin sample. But p = 0.94: a gap that size appears 96 times in 100 with no real effect behind it. The old threshold would have briefed it. Cut.

vd_07 = large + vd_09 = soft in LinkedIn Awareness. Lift index 1.34 on 6 ads. Far fewer ads, and by headcount it looks like the weaker finding. But p = 0.0009. Kept — at Medium confidence, because 6 ads is under the High threshold.

The lesson to carry: it is the size of the gap against the noise that matters, not the headcount.

9 · Self-check

1. Were counts distinct ads, never impressions or ad-days?
2. Was correction applied within each declared family, with families declared before results?
3. Does every finding carry an adjusted q-value, not just a family cutoff?
4. Was the campaign-level permutation run, with a robustness tag on every surviving finding?
5. Is patterns_tested_total stated in the output?
6. Does every surviving pattern carry a confidence tier with its ad count and spend share?
7. Were the generalization and actionability gates applied, with failures reclassified rather than silently dropped?
8. Does the output say what surviving means — hard to explain as chance, not proven, not causal?
9. If the noise calibration was run, is the number published?
10. Does every published finding carry its evidence tier?

Executes the evidence standards of the Performance Creative Intelligence measurement system. MIT licensed.