Survey Flow Audit
Audit a survey the way it will actually run, not the way its build files say it
should. Everything here reads the LIVE definition over the API and, optionally,
walks the respondent path in a real browser. The audit is read-only on the
survey definition: it produces findings, never fixes. The optional browser
walk (Phase H) does generate test responses — data-plane writes with their
own cleanup obligations, and possibly test hits on a vendor dashboard — so it
is opt-in and announced, never silent. Repairs are a separate, explicitly
authorized step (see the qualtrics-ops skill for how to make them without
new damage).
Fielding now happens in an environment where AI agents complete surveys at
scale and pass conventional attention checks (documented since 2025 in
peer-reviewed and platform validations), panel vendors bill on redirect
passbacks, and platforms silently stage rather than publish edits. Each of those failure classes is invisible in a casual preview
and cheap to catch here.
When to use
Immediately before a soft launch or full launch; after any live patch to a
fielding instrument; when a vendor reports a broken redirect or "different
content"; when handed an unfamiliar survey to take over. Inputs: API
credentials and the survey id; ideally also the pre-registration or PAP (for
the report-only-vs-terminating posture), the vendor's integration sheet
(redirect URLs, ID parameter name), and the quota targets. A browser MCP
(claude-in-chrome or Playwright) enables Phase H; without it, run A–G and say
so in the report.
Posture
- Read-only.
GET everything; PUT/POST nothing. If the platform offers a
no-op write check for token scope, that is the only write.
- Evidence or it didn't happen: every PASS cites the object read back (flow
element, option key, quota logic), never the absence of an error.
- The registered design wins. Where a PAP declares an item report-only, a live
branch that terminates on it is a blocking finding even if well-built.
Phase A — identity and publish state
Confirm the survey id, name, and active/inactive state match intent. An
inactive instrument scheduled for launch is fine; an active one nobody meant
to open is a finding.
Publish state: the working definition and the published version must match.
On Qualtrics, do not trust an in_sync flag alone — question- and
quota-level writes can leave it true while respondents see an older version.
The proof is the version list: a published entry, created by a publish call
someone can vouch for, with the published content read back where the API
exposes it (a description is provenance, not proof of content).
Staged-but-unpublished edits to a fielding survey are a blocking finding.
Response settings that shape the data: partial-response window, multiple-
submission prevention, anonymization/IP recording, link type, expiration —
and whether in-progress respondents stay pinned to the version they started.
Phase B — consent before anything
- The first substantive screen a respondent reaches is consent (or a language
selector whose every arm leads first to consent).
- Nothing evaluates or acts before affirmative consent: no terminating gates,
no quality branches, no telemetry collectors on or before the consent page.
(If the approved protocol places a minimal eligibility screener before
consent, audit that instead for authorization, minimization, and whether
pre-consent data are retained.)
Location/device capture nodes may write earlier (platforms populate them at
session start), but every branch that reads them must sit after consent.
- Decline path: declining consent must route to the vendor's screen-out (or the
study's stated exit), not dead-end or count as a complete.
- Consent text ↔ configuration consistency, both directions: if invisible
scoring or fingerprinting is enabled (reCAPTCHA, device checks), the text
discloses it; if the text promises skippable questions, optional questions
actually exist. A consent page describing a survey that isn't this one is a
finding whichever direction the drift runs.
Phase C — question integrity
- Force-response completeness: enumerate every question; classify descriptive
(no answer possible), forced, requested, and unvalidated. The check is
consistency, not a universal forced-by-default norm (optional is often the
right call for sensitive items): every unvalidated answerable item must be
one the design names optional, and if any exist, Phase B's
consent-consistency check must see them.
- Attention and manipulation checks: present where the design says, and their
consequence (terminate vs record-only) matches the registration. In the
current environment, terminating on an attention check screens out humans
while catching almost no agents — flag it as a design smell even when it
matches the PAP.
- Multilingual instruments: first identify the architecture. Qualtrics'
native translation layer keeps one block structure (audit translations for
coverage); a branch-per-language build duplicates every block per arm — there,
every item, choice set, validation setting, and embedded JS must exist
symmetrically in each arm. A check present in one arm only, or logic testing
"correct option NOT selected" on a twin build (the unanswered twin matches
trivially and ejects the whole other arm), is a blocking finding.
Phase D — flow structure
Walk the full flow tree, at every nesting depth:
- Block order matches the intended instrument; randomizers present with the
intended settings (even presentation, subset size).
- Every embedded-data field is written before the first element that reads it
(capture-before-gate). A guarded condition on a never-yet-written field is
silently dead — it fails safe, which is exactly why nobody notices.
- Terminating branches: condition logic decodes to the intended trigger; inner
flow sets the exit redirect before the End-of-Survey element; unique flow
IDs throughout; the terminal "completion" redirect node is the last element.
- On branched (language/arm) instruments, structural checks run per arm, not
once globally.
Phase E — vendor integration
- Redirect pattern: a pre-consent default carrying the screen-out URL, a
terminal overwrite carrying the complete URL, end-of-survey set to redirect
to the piped field. Early leavers must exit as screen-outs, completers as
completes, quota-fulls (if hard quotas exist) as quota-fulls — each URL
byte-exact against the vendor's sheet.
- The vendor's respondent-ID parameter is captured as embedded data and echoed
back on every exit path, including declines. Note: query parameters resolve
into piped references at session start regardless of where (or whether) the
field is declared — but declaration is what makes the value reliably saved
and exported, so treat a missing declaration as a minor finding and a wrong
parameter name as fatal.
- Enumerate which vendor endpoints can receive traffic and which are dead by
design, and check that against what the vendor was told in writing. A quality
or quota-full endpoint the vendor expects to fire, wired to nothing, is a
relationship problem waiting for fieldwork.
Phase F — quotas
- Decode every quota's logic against the live question's choices and audit it
against the RATIFIED grid, not an assumed one: marginal-family designs
should partition each frame exactly once with per-family targets summing to
the commissioned N; interlocked or deliberately overlapping designs have
their own intended structure (check the multiple-match setting). Screening
categories ("I don't live here") belong to no quota either way.
- Hard vs soft actions match the ratified design; group labels say which is
which truthfully.
- All counts are zero before fielding (test responses leave phantom counts even
after deletion-with-decrement — read the actual counters).
- Platform-specific: on Qualtrics, choice conditions need
ChoiceLocator as
well as LeftOperand — the engine evaluates the latter, the editor renders
from the former, and an editor save over a missing locator can blank live
logic. Read both. Follow pagination: quota lists truncate.
Phase G — anti-automation layer
- Platform toggles (bot-detection scoring, device fingerprinting, geo capture)
are on if the design says so — and disclosed per Phase B.
- Behavioral instrumentation (interaction paradata, honeypots, page timers) is
present on the pages the design instruments, in every language arm.
- The live-terminating set is restricted to signals that cannot plausibly be a
real person: ineligibility, duplicate device, machine signature. Anything
scored or graded (bot-score thresholds, fraud scores, speed cutoffs,
attention items) belongs to analysis, not to a live gate — a scored live gate
is a finding.
Phase H — browser walk (optional, needs a browser MCP)
- Use the LIVE distribution link, never the preview (preview banners change
rendering and skip embedded-data population). Append a test value for the
vendor ID parameter.
- Walk at minimum: one decline (assert the screen-out redirect fires with the
ID echoed), one complete per language arm (assert the complete redirect), one
mobile-viewport pass (conjoint tables and stacked layouts render; nothing
clips). Where feasible add: the quota-full path, one pass per experimental
arm, a missing-vendor-ID entry, and validation/back-button behavior on one
forced item.
- Confirm no screen precedes consent, and that the consent page renders in the
right language for each arm.
- Clean up: delete the test responses with quota decrement, then re-read quota
counts (Phase F) — and note that in-progress partials usually cannot be
deleted via API and must expire or be cleared in the UI.
Report
Rank findings blocking / major / minor, each with the evidence read back
and the phase that produced it. State explicitly: live version vs working
version; which phases ran (and that H was skipped, if it was); which findings
the registered design forces you to leave alone. End with the test-response
cleanup confirmation if Phase H ran. Hand fixes to a separate authorized
change — with its own backup, read-back, and publish-with-proof — rather than
folding them into the audit.
1---2name: survey-flow-audit-23description: Pre-fielding audit of a live survey over the platform API, with an optional browser walk. Consent-before-anything gates, publish state, force-response completeness, quotas, vendor redirects, anti-bot instrumentation, language-arm symmetry.4argument-hint: "[survey id or platform, plus the registration and vendor sheet if available]"5metadata:6 source: https://skillmd.com/skills/majiayu000/survey-flow-audit-27---89# Survey Flow Audit1011Audit a survey the way it will actually run, not the way its build files say it12should. Everything here reads the LIVE definition over the API and, optionally,13walks the respondent path in a real browser. The audit is **read-only on the14survey definition**: it produces findings, never fixes. The optional browser15walk (Phase H) does generate test *responses* — data-plane writes with their16own cleanup obligations, and possibly test hits on a vendor dashboard — so it17is opt-in and announced, never silent. Repairs are a separate, explicitly18authorized step (see the `qualtrics-ops` skill for how to make them without19new damage).2021Fielding now happens in an environment where AI agents complete surveys at22scale and pass conventional attention checks (documented since 2025 in23peer-reviewed and platform validations), panel vendors bill on redirect24passbacks, and platforms silently stage rather than publish edits. Each of those failure classes is invisible in a casual preview25and cheap to catch here.2627## When to use2829Immediately before a soft launch or full launch; after any live patch to a30fielding instrument; when a vendor reports a broken redirect or "different31content"; when handed an unfamiliar survey to take over. Inputs: API32credentials and the survey id; ideally also the pre-registration or PAP (for33the report-only-vs-terminating posture), the vendor's integration sheet34(redirect URLs, ID parameter name), and the quota targets. A browser MCP35(claude-in-chrome or Playwright) enables Phase H; without it, run A–G and say36so in the report.3738## Posture3940- Read-only. `GET` everything; `PUT`/`POST` nothing. If the platform offers a41 no-op write check for token scope, that is the only write.42- Evidence or it didn't happen: every PASS cites the object read back (flow43 element, option key, quota logic), never the absence of an error.44- The registered design wins. Where a PAP declares an item report-only, a live45 branch that terminates on it is a **blocking** finding even if well-built.4647## Phase A — identity and publish state4849- Confirm the survey id, name, and active/inactive state match intent. An50 inactive instrument scheduled for launch is fine; an active one nobody meant51 to open is a finding.52- Publish state: the working definition and the published version must match.53 On Qualtrics, do not trust an `in_sync` flag alone — question- and54 quota-level writes can leave it true while respondents see an older version.55 The proof is the version list: a published entry, created by a publish call56 someone can vouch for, with the published content read back where the API57 exposes it (a description is provenance, not proof of content).58 Staged-but-unpublished edits to a fielding survey are a **blocking** finding.5960- Response settings that shape the data: partial-response window, multiple-61 submission prevention, anonymization/IP recording, link type, expiration —62 and whether in-progress respondents stay pinned to the version they started.6364## Phase B — consent before anything6566- The first substantive screen a respondent reaches is consent (or a language67 selector whose every arm leads first to consent).68- Nothing evaluates or acts before affirmative consent: no terminating gates,69 no quality branches, no telemetry collectors on or before the consent page.70 (If the approved protocol places a minimal eligibility screener before71 consent, audit that instead for authorization, minimization, and whether72 pre-consent data are retained.)73 Location/device capture nodes may *write* earlier (platforms populate them at74 session start), but every branch that *reads* them must sit after consent.75- Decline path: declining consent must route to the vendor's screen-out (or the76 study's stated exit), not dead-end or count as a complete.77- Consent text ↔ configuration consistency, both directions: if invisible78 scoring or fingerprinting is enabled (reCAPTCHA, device checks), the text79 discloses it; if the text promises skippable questions, optional questions80 actually exist. A consent page describing a survey that isn't this one is a81 finding whichever direction the drift runs.8283## Phase C — question integrity8485- Force-response completeness: enumerate every question; classify descriptive86 (no answer possible), forced, requested, and unvalidated. The check is87 consistency, not a universal forced-by-default norm (optional is often the88 right call for sensitive items): every unvalidated answerable item must be89 one the design *names* optional, and if any exist, Phase B's90 consent-consistency check must see them.91- Attention and manipulation checks: present where the design says, and their92 *consequence* (terminate vs record-only) matches the registration. In the93 current environment, terminating on an attention check screens out humans94 while catching almost no agents — flag it as a design smell even when it95 matches the PAP.96- Multilingual instruments: first identify the architecture. Qualtrics'97 native translation layer keeps one block structure (audit translations for98 coverage); a branch-per-language build duplicates every block per arm — there,99 every item, choice set, validation setting, and embedded JS must exist100 symmetrically in each arm. A check present in one arm only, or logic testing101 "correct option NOT selected" on a twin build (the unanswered twin matches102 trivially and ejects the whole other arm), is a **blocking** finding.103104## Phase D — flow structure105106Walk the full flow tree, at every nesting depth:107108- Block order matches the intended instrument; randomizers present with the109 intended settings (even presentation, subset size).110- Every embedded-data field is written before the first element that reads it111 (capture-before-gate). A guarded condition on a never-yet-written field is112 silently dead — it fails safe, which is exactly why nobody notices.113- Terminating branches: condition logic decodes to the intended trigger; inner114 flow sets the exit redirect *before* the End-of-Survey element; unique flow115 IDs throughout; the terminal "completion" redirect node is the last element.116- On branched (language/arm) instruments, structural checks run per arm, not117 once globally.118119## Phase E — vendor integration120121- Redirect pattern: a pre-consent default carrying the screen-out URL, a122 terminal overwrite carrying the complete URL, end-of-survey set to redirect123 to the piped field. Early leavers must exit as screen-outs, completers as124 completes, quota-fulls (if hard quotas exist) as quota-fulls — each URL125 byte-exact against the vendor's sheet.126- The vendor's respondent-ID parameter is captured as embedded data and echoed127 back on every exit path, including declines. Note: query parameters resolve128 into piped references at session start regardless of where (or whether) the129 field is declared — but declaration is what makes the value reliably saved130 and exported, so treat a missing declaration as a minor finding and a wrong131 parameter name as fatal.132- Enumerate which vendor endpoints can receive traffic and which are dead by133 design, and check that against what the vendor was told in writing. A quality134 or quota-full endpoint the vendor expects to fire, wired to nothing, is a135 relationship problem waiting for fieldwork.136137## Phase F — quotas138139- Decode every quota's logic against the live question's choices and audit it140 against the RATIFIED grid, not an assumed one: marginal-family designs141 should partition each frame exactly once with per-family targets summing to142 the commissioned N; interlocked or deliberately overlapping designs have143 their own intended structure (check the multiple-match setting). Screening144 categories ("I don't live here") belong to no quota either way.145- Hard vs soft actions match the ratified design; group labels say which is146 which truthfully.147- All counts are zero before fielding (test responses leave phantom counts even148 after deletion-with-decrement — read the actual counters).149- Platform-specific: on Qualtrics, choice conditions need `ChoiceLocator` as150 well as `LeftOperand` — the engine evaluates the latter, the editor renders151 from the former, and an editor save over a missing locator can blank live152 logic. Read both. Follow pagination: quota lists truncate.153154## Phase G — anti-automation layer155156- Platform toggles (bot-detection scoring, device fingerprinting, geo capture)157 are on if the design says so — and disclosed per Phase B.158- Behavioral instrumentation (interaction paradata, honeypots, page timers) is159 present on the pages the design instruments, in every language arm.160- The live-terminating set is restricted to signals that cannot plausibly be a161 real person: ineligibility, duplicate device, machine signature. Anything162 scored or graded (bot-score thresholds, fraud scores, speed cutoffs,163 attention items) belongs to analysis, not to a live gate — a scored live gate164 is a finding.165166## Phase H — browser walk (optional, needs a browser MCP)167168- Use the LIVE distribution link, never the preview (preview banners change169 rendering and skip embedded-data population). Append a test value for the170 vendor ID parameter.171- Walk at minimum: one decline (assert the screen-out redirect fires with the172 ID echoed), one complete per language arm (assert the complete redirect), one173 mobile-viewport pass (conjoint tables and stacked layouts render; nothing174 clips). Where feasible add: the quota-full path, one pass per experimental175 arm, a missing-vendor-ID entry, and validation/back-button behavior on one176 forced item.177- Confirm no screen precedes consent, and that the consent page renders in the178 right language for each arm.179- Clean up: delete the test responses with quota decrement, then re-read quota180 counts (Phase F) — and note that in-progress partials usually cannot be181 deleted via API and must expire or be cleared in the UI.182183## Report184185Rank findings **blocking / major / minor**, each with the evidence read back186and the phase that produced it. State explicitly: live version vs working187version; which phases ran (and that H was skipped, if it was); which findings188the registered design forces you to leave alone. End with the test-response189cleanup confirmation if Phase H ran. Hand fixes to a separate authorized190change — with its own backup, read-back, and publish-with-proof — rather than191folding them into the audit.