Creative pre-flight panel (Claude + Jev)
Third step of the Claude + Jev creative system: parse the market (competitor-creative-parser), write the briefs (market-to-creative-brief), pre-flight the concepts (this skill). Every concept meets a simulated buyer panel before anyone opens a design tool, weak ideas are cut early, and after launch the panel is checked against real results so the team learns where it was right and where it was wrong.
A panel score is a cheap filter and a signal, not proof of performance. It never replaces a live test and never touches budgets.
| Who |
Does what |
| Claude |
Writes 12 to 30 buyer profiles from the team's ICP notes and the snapshot's audience signals |
| Jev |
Answers typed questions for every concept and profile pair: a Choice (scroll, stop, click) with probabilities, three yes/no checks, a fit Score |
| Code |
Rotates option order, builds the heatmap, ranks with guards, applies the go, rework, drop rules, compares with real results later |
| A person |
Decides what is produced and what goes to the test |
When to use
- A brief pack has more concepts or hooks than the team can produce this sprint.
- Before a UGC shoot or a motion job, to pick which of three hooks to produce first.
- To see which buyer profiles no concept reaches.
- After a test, to measure whether the panel's order matched the real order and set its weight for next time.
Not for: deciding budgets, declaring winners, replacing a live test, or judging what only the finished asset shows (picture quality, a voice, pacing).
Pipeline
concepts.jsonl (brief_pack.py --panel-concepts) profiles.jsonl (Claude)
1 Profiles (Claude) -> profiles.jsonl
2 Panel (Jev, run_panel.py) -> answers-<track>.jsonl, run-<track>.json, order-check-<track>.json
3 Matrix (code) -> matrix-<track>.csv, matrix.md, verdicts.json
4 Decide (a person) -> brief_pack.py --panel-verdicts verdicts.json: the table goes into the pack
5 Test (meta-creative-testing), then calibrate (panel_vs_results.py) with real results
Static track and video track
|
Static |
Video |
| Panel item |
one concept (S1) |
one hook variant of a concept (M1-A, M1-B, M1-C) |
State ad |
headline, primary text, button, offer line, image description |
format, first 3 seconds (seen, on screen, spoken), the rest of the beat sheet as text, end card |
| Choice |
keeps scrolling, stops and reads, taps |
swipes away in 3 s, keeps watching, taps |
| yes/no |
headline about the buyer's problem; offer and terms clear from headline, image and button; would distrust a claim |
first 3 s about the buyer's problem; product clear from the first 3 s; would distrust a claim |
| Score |
fit with situation and goal, 4 levels |
the same |
| Extra rule |
|
a hook that trails its concept's best hook by 0.10 or more is reworked |
| Question file |
scripts/questions_static.py |
scripts/questions_video.py |
Inputs to ask for
concepts.jsonl from market-to-creative-brief (brief_pack.py --panel-concepts).
- The team's ICP notes: who buys, who churns, what reviews, support and sales hear.
snapshot.md from competitor-creative-parser, for audience signals.
- Confirmation that concepts and profiles may be sent to the vendor (TypeSafe). Written by the guide, they hold no personal data.
Step 1. Write the profiles
Follow references/profile-writing.md: 12 to 30 profiles, each with situation, goal, objections and what makes them stop scrolling; at least two skeptics, one price-sensitive buyer and one control outside the target; situations, not demographics; no real persons.
Step 2. Run the panel
python3 scripts/run_panel.py --concepts concepts.jsonl --profiles profiles.jsonl --track static --dry-run --out-dir panel
python3 scripts/run_panel.py --concepts concepts.jsonl --profiles profiles.jsonl --track static --smoke \
--data-handling-confirmed --out-dir panel-smoke
python3 scripts/run_panel.py --concepts concepts.jsonl --profiles profiles.jsonl --track video --order-check 12 \
--max-usd 0.25 --data-handling-confirmed --out-dir panel
- Key from the environment only:
TYPESAFE_API_KEY (or --key-env). Never in chat, a file or a log. --base-url takes an https gateway URL.
--dry-run is an offline mock with meaningless numbers. --smoke runs 3 pairs live and checks the response shape.
- Spend guard
--max-usd (default 0.25); tokens come from the API's usage field. One request per pair with all five questions.
- Errors never become answers: a failed pair is marked
error, and an item with fewer than half of its profiles answered gets no verdict.
- Question design and why:
references/question-design.md.
Step 3. Build the matrix
python3 scripts/panel_matrix.py --answers panel/answers-static.jsonl panel/answers-video.jsonl \
--profiles profiles.jsonl --out-dir panel
Per item: stop-or-click mass, click mass, mean fit, the share of profiles for each yes/no, the share of pairs in the review band, the profiles it loses. Guards: no ranking under 12 profiles, no leader named on a lead under 0.10. Verdict rules and how to read the heatmap: references/reading-the-matrix.md.
Step 4. Back into the brief pack
python3 ../market-to-creative-brief/scripts/brief_pack.py --decisions decisions.json --snapshot snapshot.json \
--out brief-pack.md --panel-verdicts panel/verdicts.json
The pack gets the go, rework, drop table; dropped items are held out of the test plan names and rework items are marked. A person decides; any verdict can be overridden.
Step 5. Calibrate after launch
python3 scripts/panel_vs_results.py --verdicts panel/verdicts.json --results results.csv --metric ctr --out panel-vs-results.md
results.csv holds real per-item numbers the user exports (item_id, impressions, clicks, conversions, spend). Never invent or estimate them. The report gives rank agreement, the misses and advice on the panel's weight next time.
Output to the user
matrix.md: heatmaps, ranking with guards, verdicts with reasons, the profiles each item loses.
- The go, rework, drop table, and the brief pack rendered with it.
- Plain caveats: the panel read text, not the finished ads; it is a filter and a signal; the live test decides.
- After a test:
panel-vs-results.md and the advice on weight.
Guardrails
- A panel score is not proof of performance and is never presented as one.
- Never touches budgets, ad accounts or launches. A person decides.
- Profiles describe situations, never real or identifiable people.
- Thresholds are starting values until
panel_vs_results.py has run on the team's own results.
- No invented results: calibration uses only numbers the user supplies.
What was checked live and what was not
Checked on 2026-11-15 with the owner's key on the fictional brief pack (6 concepts, 12 panel items) and 12 fictional profiles on jev-1.14.0 through the direct vendor endpoint: a smoke run (3 pairs, 2,407 input tokens), then 144 pairs across both tracks with an order check on 12 pairs per track; 0 errors, 143,539 input tokens in total, $0.0060 at the listed price. Order agreement was 11 of 12 on each track. Saved output: scripts/sample_panel/live-run-2026-11-15/ (real model output on fictional concepts). Write-up: examples/competitor-parse-fictional-market.md in the repository.
Not checked: real concepts and real buyers, the gateway route, and panel_vs_results.py on real results (none exist for a fictional market; the script is covered by offline tests with synthetic fixtures). In the example, offer_clear and distrust_claim mostly stayed in the review band: rewrite or calibrate them on your own concepts before they gate anything.
Files
scripts/questions_static.py, scripts/questions_video.py: questions, thresholds, state builders.
scripts/run_panel.py: panel runner, standard library only.
scripts/panel_matrix.py: heatmap, ranking, verdicts.
scripts/panel_vs_results.py: calibration against real results.
scripts/test_offline.py: tests, no network, no key.
scripts/sample_panel/: fictional profiles and concepts, plus the saved live run.
references/profile-writing.md, references/question-design.md, references/reading-the-matrix.md.
Related skills: market-to-creative-brief (concepts in, verdicts back), meta-creative-testing (survivors go to the test plan), competitor-creative-parser (audience signals), jev-marketing-gates (runner pattern, hard rules).
Author
Vlad Yasinski, AI growth consultant for app and subscription businesses. Telegram: @ua5er (https://www.roimobi.com/tg?src=skill-creative-preflight-panel). Site: https://www.roimobi.com/?utm_source=github&utm_medium=skill&utm_campaign=user-acquisition-skills
1---2name: creative-preflight-panel3description: Claude + Jev pre-flight panel for ad concepts. Before designers or the motion team spend hours, every concept from the brief pack is shown to 12 to 30 simulated buyer profiles that Claude writes from the team's ICP notes and the market snapshot; Jev answers a small typed set per concept and profile (scroll, stop or click with probabilities, three yes/no checks, a fit score); code builds the concept x profile heatmap, ranks with small-sample and tie guards and gives go, rework or drop with reasons; a person decides. After launch, a calibration script checks the panel against real results. Static and video tracks. Use for "pre-test ad concepts", "screen creative concepts before production", "synthetic audience panel", "which hook should we produce", "concept testing with AI", "simulated buyer panel", "Jev panel".4license: MIT5metadata:6 source: https://skillmd.com/skills/ua5er/creative-preflight-panel7 author: Vlad Yasinski8 version: "1.0"9 site: https://www.roimobi.com10 telegram: https://t.me/ua5er11---1213# Creative pre-flight panel (Claude + Jev)1415Third step of the Claude + Jev creative system: parse the market (`competitor-creative-parser`), write the briefs (`market-to-creative-brief`), pre-flight the concepts (this skill). Every concept meets a simulated buyer panel before anyone opens a design tool, weak ideas are cut early, and after launch the panel is checked against real results so the team learns where it was right and where it was wrong.1617A panel score is a cheap filter and a signal, not proof of performance. It never replaces a live test and never touches budgets.1819| Who | Does what |20|---|---|21| Claude | Writes 12 to 30 buyer profiles from the team's ICP notes and the snapshot's audience signals |22| Jev | Answers typed questions for every concept and profile pair: a Choice (scroll, stop, click) with probabilities, three yes/no checks, a fit Score |23| Code | Rotates option order, builds the heatmap, ranks with guards, applies the go, rework, drop rules, compares with real results later |24| A person | Decides what is produced and what goes to the test |2526## When to use2728- A brief pack has more concepts or hooks than the team can produce this sprint.29- Before a UGC shoot or a motion job, to pick which of three hooks to produce first.30- To see which buyer profiles no concept reaches.31- After a test, to measure whether the panel's order matched the real order and set its weight for next time.3233Not for: deciding budgets, declaring winners, replacing a live test, or judging what only the finished asset shows (picture quality, a voice, pacing).3435## Pipeline3637```38concepts.jsonl (brief_pack.py --panel-concepts) profiles.jsonl (Claude)39 1 Profiles (Claude) -> profiles.jsonl40 2 Panel (Jev, run_panel.py) -> answers-<track>.jsonl, run-<track>.json, order-check-<track>.json41 3 Matrix (code) -> matrix-<track>.csv, matrix.md, verdicts.json42 4 Decide (a person) -> brief_pack.py --panel-verdicts verdicts.json: the table goes into the pack43 5 Test (meta-creative-testing), then calibrate (panel_vs_results.py) with real results44```4546## Static track and video track4748| | Static | Video |49|---|---|---|50| Panel item | one concept (S1) | one hook variant of a concept (M1-A, M1-B, M1-C) |51| State `ad` | headline, primary text, button, offer line, image description | format, first 3 seconds (seen, on screen, spoken), the rest of the beat sheet as text, end card |52| Choice | keeps scrolling, stops and reads, taps | swipes away in 3 s, keeps watching, taps |53| yes/no | headline about the buyer's problem; offer and terms clear from headline, image and button; would distrust a claim | first 3 s about the buyer's problem; product clear from the first 3 s; would distrust a claim |54| Score | fit with situation and goal, 4 levels | the same |55| Extra rule | | a hook that trails its concept's best hook by 0.10 or more is reworked |56| Question file | `scripts/questions_static.py` | `scripts/questions_video.py` |5758## Inputs to ask for59601. `concepts.jsonl` from `market-to-creative-brief` (`brief_pack.py --panel-concepts`).612. The team's ICP notes: who buys, who churns, what reviews, support and sales hear.623. `snapshot.md` from `competitor-creative-parser`, for audience signals.634. Confirmation that concepts and profiles may be sent to the vendor (TypeSafe). Written by the guide, they hold no personal data.6465## Step 1. Write the profiles6667Follow `references/profile-writing.md`: 12 to 30 profiles, each with situation, goal, objections and what makes them stop scrolling; at least two skeptics, one price-sensitive buyer and one control outside the target; situations, not demographics; no real persons.6869## Step 2. Run the panel7071```72python3 scripts/run_panel.py --concepts concepts.jsonl --profiles profiles.jsonl --track static --dry-run --out-dir panel73python3 scripts/run_panel.py --concepts concepts.jsonl --profiles profiles.jsonl --track static --smoke \74 --data-handling-confirmed --out-dir panel-smoke75python3 scripts/run_panel.py --concepts concepts.jsonl --profiles profiles.jsonl --track video --order-check 12 \76 --max-usd 0.25 --data-handling-confirmed --out-dir panel77```7879- Key from the environment only: `TYPESAFE_API_KEY` (or `--key-env`). Never in chat, a file or a log. `--base-url` takes an https gateway URL.80- `--dry-run` is an offline mock with meaningless numbers. `--smoke` runs 3 pairs live and checks the response shape.81- Spend guard `--max-usd` (default 0.25); tokens come from the API's usage field. One request per pair with all five questions.82- Errors never become answers: a failed pair is marked `error`, and an item with fewer than half of its profiles answered gets no verdict.83- Question design and why: `references/question-design.md`.8485## Step 3. Build the matrix8687```88python3 scripts/panel_matrix.py --answers panel/answers-static.jsonl panel/answers-video.jsonl \89 --profiles profiles.jsonl --out-dir panel90```9192Per item: stop-or-click mass, click mass, mean fit, the share of profiles for each yes/no, the share of pairs in the review band, the profiles it loses. Guards: no ranking under 12 profiles, no leader named on a lead under 0.10. Verdict rules and how to read the heatmap: `references/reading-the-matrix.md`.9394## Step 4. Back into the brief pack9596```97python3 ../market-to-creative-brief/scripts/brief_pack.py --decisions decisions.json --snapshot snapshot.json \98 --out brief-pack.md --panel-verdicts panel/verdicts.json99```100101The pack gets the go, rework, drop table; dropped items are held out of the test plan names and rework items are marked. A person decides; any verdict can be overridden.102103## Step 5. Calibrate after launch104105```106python3 scripts/panel_vs_results.py --verdicts panel/verdicts.json --results results.csv --metric ctr --out panel-vs-results.md107```108109`results.csv` holds real per-item numbers the user exports (item_id, impressions, clicks, conversions, spend). Never invent or estimate them. The report gives rank agreement, the misses and advice on the panel's weight next time.110111## Output to the user1121131. `matrix.md`: heatmaps, ranking with guards, verdicts with reasons, the profiles each item loses.1142. The go, rework, drop table, and the brief pack rendered with it.1153. Plain caveats: the panel read text, not the finished ads; it is a filter and a signal; the live test decides.1164. After a test: `panel-vs-results.md` and the advice on weight.117118## Guardrails119120- A panel score is not proof of performance and is never presented as one.121- Never touches budgets, ad accounts or launches. A person decides.122- Profiles describe situations, never real or identifiable people.123- Thresholds are starting values until `panel_vs_results.py` has run on the team's own results.124- No invented results: calibration uses only numbers the user supplies.125126## What was checked live and what was not127128Checked on 2026-11-15 with the owner's key on the fictional brief pack (6 concepts, 12 panel items) and 12 fictional profiles on `jev-1.14.0` through the direct vendor endpoint: a smoke run (3 pairs, 2,407 input tokens), then 144 pairs across both tracks with an order check on 12 pairs per track; 0 errors, 143,539 input tokens in total, $0.0060 at the listed price. Order agreement was 11 of 12 on each track. Saved output: `scripts/sample_panel/live-run-2026-11-15/` (real model output on fictional concepts). Write-up: `examples/competitor-parse-fictional-market.md` in the repository.129130Not checked: real concepts and real buyers, the gateway route, and `panel_vs_results.py` on real results (none exist for a fictional market; the script is covered by offline tests with synthetic fixtures). In the example, `offer_clear` and `distrust_claim` mostly stayed in the review band: rewrite or calibrate them on your own concepts before they gate anything.131132## Files133134- `scripts/questions_static.py`, `scripts/questions_video.py`: questions, thresholds, state builders.135- `scripts/run_panel.py`: panel runner, standard library only.136- `scripts/panel_matrix.py`: heatmap, ranking, verdicts.137- `scripts/panel_vs_results.py`: calibration against real results.138- `scripts/test_offline.py`: tests, no network, no key.139- `scripts/sample_panel/`: fictional profiles and concepts, plus the saved live run.140- `references/profile-writing.md`, `references/question-design.md`, `references/reading-the-matrix.md`.141142Related skills: `market-to-creative-brief` (concepts in, verdicts back), `meta-creative-testing` (survivors go to the test plan), `competitor-creative-parser` (audience signals), `jev-marketing-gates` (runner pattern, hard rules).143144## Author145146Vlad Yasinski, AI growth consultant for app and subscription businesses. Telegram: @ua5er (https://www.roimobi.com/tg?src=skill-creative-preflight-panel). Site: https://www.roimobi.com/?utm_source=github&utm_medium=skill&utm_campaign=user-acquisition-skills