Set up BQAA tracing for this user
Goal: get agent-event rows flowing from this Claude Code session into a BigQuery agent_events table. The plugin's hooks already fire automatically once enabled — your job is to fill in the GCP/IAM gap. Do not mutate the user's GCP project without explicit approval at each step.
What's already automatic vs. what isn't
After claude plugin install bigquery-agent-analytics-tracing@… and a Claude Code restart:
- Automatic: SessionStart / UserPromptSubmit / PreToolUse / PostToolUse / Stop / SubagentStop hooks fire and try to write rows.
- Not automatic: enabling GCP APIs, creating the BigQuery dataset/table, creating a service account, granting IAM, or running the bootstrap. The hook code calls
google-cloud-bigquerydirectly — it never invokesgcloudorbq. IfBQAA_PROJECT_ID/BQAA_DATASETare missing or IAM is wrong, every hook silently no-ops and writes the error toBQAA_LOG_FILE(default/tmp/bqaa-agent-tracing.log).
So if rows aren't showing up, you are almost always one of: (a) env vars unset, (b) ADC not configured, (c) IAM grants missing, (d) wrong dataset/location. This skill walks the user through fixing all four.
Step 1 — Gather inputs (don't run anything yet)
Ask the user (or read from environment) and confirm before continuing:
| Input | Default / source | When to ask |
|---|---|---|
| GCP project ID | $BQAA_PROJECT_ID, $GCP_PROJECT_ID, $GOOGLE_CLOUD_PROJECT, or gcloud config get-value project |
Always confirm — wrong project is the most expensive mistake |
| BigQuery dataset | $BQAA_DATASET or agent_analytics |
Ask if multi-team or shared dataset |
| Table name | $BQAA_TABLE or agent_events |
Rarely changed |
| Location | $BQAA_LOCATION or US |
Ask if outside US (must match existing dataset if it exists) |
| Identity to grant (the principal) | user:$(gcloud config get-value account) (the user's own account) |
See "Choosing the principal" below |
| Consumer | Ask: Claude Code TUI, Claude Agent SDK (Python), or Codex CLI? | Always ask — Step 6 emits different config for each |
Choosing the principal
The default for local Claude Code / SDK / Codex on a developer workstation is
--principal user:$(gcloud config get-value account) — grant the developer's
own Google identity. Why: the BQAA hooks read ADC, and ADC == that same identity
on a normal gcloud auth application-default login setup. No impersonation, no
extra wiring; runtime "just works" once env is set in Step 6.
Multi-account warning. The default grants the active gcloud CLI account (
gcloud config get-value account), but runtime hooks use ADC (gcloud auth application-default login). On most setups they're the same identity, but if the user has multiple Google accounts logged in, those two can diverge — the IAM grant lands on account A and the hooks keep writing as account B (which has no grant). The preflight surfaces both: if theADC: OKline and thegcloud CLI auth: OKline don't show the same email, stop and confirm with the user which account should be granted, then have them rungcloud auth application-default loginwith that account before --execute.
A service account (--service-account bqaa-writer) is the right choice
when:
- Multiple developers / a shared workstation will share the same writer identity.
- The agent will run unattended on a server / GKE / CI where there's no human to OAuth.
- Your org centralizes credential management on SAs.
If you go that route, the hooks will not automatically use the SA — ADC is
still whoever is logged in. The BQAA hooks call google-cloud-bigquery
directly, which reads ADC; they do not consult gcloud config. So whatever
routes the SA into runtime has to route it into ADC. Pick one:
# Option A — SA impersonation via ADC (recommended, no JSON key):
gcloud auth application-default login \
--impersonate-service-account=bqaa-writer@PROJECT.iam.gserviceaccount.com
# Caller still needs roles/iam.serviceAccountTokenCreator on the SA.
# Option B — service-account JSON key (least preferred; rotate on schedule):
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/bqaa-writer.json
# Option C — Workload Identity Federation (for CI / unattended; no key file):
# See https://cloud.google.com/iam/docs/workload-identity-federation
# The agent picks up SA credentials from GOOGLE_APPLICATION_CREDENTIALS
# pointing at the WIF credential-config JSON.
Do NOT confuse with
gcloud config set auth/impersonate_service_account. That setting redirects the gcloud / bq CLI account only. It does not affect the BQAA Python hooks — they read ADC. Setting it alone leaves the hooks running under whoever ADC is configured for, even thoughgcloudcommands you type yourself will run as the SA. Use it as a complement to Option A if you want CLI commands to also impersonate; never use it as the only step.
If the user picked the SA path without picking A/B/C, stop and tell them the SA was created and granted but the runtime hooks will keep using their own identity until ADC routes through the SA. Most local-dev users should just take the user-principal default.
Echo the values back to the user before any subprocess call. Phrasing like
"I'll set up BQAA tracing on project-xyz / dataset agent_analytics /
location US, granting your own account you@example.com (so local Claude
Code's ADC writes directly) — sound right?" works.
Step 2 — Check ADC
Run only:
gcloud auth application-default print-access-token >/dev/null 2>&1 && echo "ADC OK" || echo "ADC NOT CONFIGURED"
gcloud config get-value project 2>/dev/null
If ADC is not configured, stop here. Tell the user verbatim:
ADC isn't configured — you'll need to run
gcloud auth application-default loginin your own terminal (it's an interactive browser flow that I can't complete for you). Reply when it's done and I'll continue.
Wait for them to confirm. Don't try gcloud auth application-default login yourself; the OAuth flow needs a browser the user controls.
Step 3 — Dry-run the bootstrap
Always run dry-run first. Show the full plan to the user.
Default (recommended for local Claude Code / SDK / Codex on a workstation):
python plugins/bigquery-agent-analytics-tracing/scripts/setup_gcp_prereqs.py \
--project "$BQAA_PROJECT_ID" \
--dataset "$BQAA_DATASET" \
--table "${BQAA_TABLE:-agent_events}" \
--location "${BQAA_LOCATION:-US}" \
--principal "user:$(gcloud config get-value account)" \
--non-interactive
Service-account path (advanced — only when user opted in per Step 1; remind them they still need to wire impersonation/key per Step 1's Option A/B/C):
python plugins/bigquery-agent-analytics-tracing/scripts/setup_gcp_prereqs.py \
--project "$BQAA_PROJECT_ID" \
--dataset "$BQAA_DATASET" \
--table "${BQAA_TABLE:-agent_events}" \
--location "${BQAA_LOCATION:-US}" \
--service-account bqaa-writer \
--non-interactive
The script prints a numbered plan of every gcloud / bq / google-cloud-bigquery call it will make, plus a preflight summary of ADC + gcloud CLI auth + project. Show the printed plan to the user verbatim and ask: "Does this plan look right? Want me to apply it?"
Common variations:
- Existing dataset, don't touch it: add
--no-create-dataset --no-create-table. - Runtime auto-creates datasets: add
--runtime-auto-create-datasetto also grantroles/bigquery.user.
Step 4 — Wait for explicit approval, then execute
Hard rule: re-run the EXACT command from Step 3 with --execute appended.
Do not switch the principal, the service-account flag, or any other option
between dry-run and execute. The user approved one plan; running anything
different — even a "more sensible" variant — bypasses approval and can
silently apply a different identity path than what was reviewed.
In practice this means: if Step 3 ran with --principal "user:...", Step 4
runs the same command + --execute; if Step 3 ran with
--service-account bqaa-writer, Step 4 runs that same command + --execute.
Any change to the plan goes back through Step 3 first.
If a step fails at execute time (most often Permission denied: serviceusage.services.enable or IAM propagation latency), the script's own
retry covers IAM propagation. For permission errors, tell the user which
role they're missing — it's almost always one of:
roles/serviceusage.serviceUsageAdminto enable APIsroles/iam.serviceAccountAdminto create the SAroles/bigquery.admin(ordataOwneron dataset +jobUseron project) to create dataset / set IAM
Step 5 — Verify with a real BigQuery round-trip
Run the e2e smoke once --execute returns 0:
python plugins/bigquery-agent-analytics-tracing/scripts/e2e_bigquery_smoke.py \
--project "$BQAA_PROJECT_ID" \
--dataset "$BQAA_DATASET" \
--table "${BQAA_TABLE:-agent_events}" \
--location "${BQAA_LOCATION:-US}"
Expected: JSON output ends with "missing_event_types": [] and "row_count": 4. Show the user the four event types written.
Step 6 — Wire the env so the consumer auto-traces
Required keys: BQAA_PROJECT_ID, BQAA_DATASET, BQAA_LOCATION. Optional but
recommended: BQAA_TABLE, BQAA_AGENT_NAME, BQAA_WRITER_LABEL (use a
per-team / per-product suffix like bqaa-coding-agent-plugin/0.1.0/team-foo
so adoption queries can break down by deployment).
The "right place" depends on the consumer the user chose in Step 1. Don't pick one for them. Emit the snippet that fits and tell them how to apply it.
If the consumer is Claude Code TUI
Project-scoped is best — .claude/settings.local.json is gitignored and only
this repo gets the env. Write or update the env block:
{
"env": {
"BQAA_PROJECT_ID": "<PROJECT>",
"BQAA_DATASET": "<DATASET>",
"BQAA_TABLE": "agent_events",
"BQAA_LOCATION": "<LOCATION>",
"BQAA_AGENT_NAME": "claude-code",
"BQAA_WRITER_LABEL": "bqaa-coding-agent-plugin/0.1.0/<deployment>"
},
"permissions": { "...": "...preserve any existing block..." }
}
Use the Edit tool (not Write) if the file already has other settings so
you don't clobber existing permissions / enabledPlugins / etc. Hand-off:
"Restart Claude Code in this directory to pick up the env."
If the consumer is Claude Agent SDK (Python)
Settings.local.json is not read by the SDK. The SDK spawns its own
claude subprocess with the env you pass via ClaudeAgentOptions. Emit:
import os
from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
options = ClaudeAgentOptions(
cwd="<your-project-path>",
env={
**os.environ,
"BQAA_PROJECT_ID": "<PROJECT>",
"BQAA_DATASET": "<DATASET>",
"BQAA_TABLE": "agent_events",
"BQAA_LOCATION": "<LOCATION>",
"BQAA_AGENT_NAME": "claude-agent-sdk",
"BQAA_WRITER_LABEL": "bqaa-coding-agent-plugin/0.1.0/<deployment>",
},
plugins=[{"type": "local",
"path": "<absolute-path-to>/plugins/bigquery-agent-analytics-tracing"}],
)
async with ClaudeSDKClient(options=options) as client:
...
Hand-off: "Use this options block on every ClaudeSDKClient. The SDK does
not inherit your shell env — every BQAA_* must be in options.env."
If the consumer is Codex CLI (bqaa-codex wrapper)
Shell-scoped is the only path. Emit:
export BQAA_PROJECT_ID="<PROJECT>"
export BQAA_DATASET="<DATASET>"
export BQAA_TABLE="agent_events"
export BQAA_LOCATION="<LOCATION>"
export BQAA_AGENT_NAME="codex-cli"
export BQAA_WRITER_LABEL="bqaa-coding-agent-plugin/0.1.0/<deployment>"
Tell the user: "Add these to ~/.zshrc / ~/.bashrc and restart your shell,
or source them in the same terminal where you'll run
bqaa-codex. Then run a real codex exec through the wrapper and re-check
BigQuery."
Python runtime caveat (all consumers)
The plugin's hooks shell out to bash hook.sh, which runs
${BQAA_PYTHON:-python3}. Whatever Python that resolves to must have
google-cloud-bigquery (and ideally google-cloud-bigquery-storage and
pyarrow) importable. If the user installed deps in a venv, point hooks at
that interpreter:
export BQAA_PYTHON="/path/to/venv/bin/python"
Add BQAA_PYTHON to the env block too (settings.local.json env or
ClaudeAgentOptions.env or shell). When verifying in Step 5/7, if rows aren't
landing despite IAM looking right, check cat /tmp/bqaa-agent-tracing.log —
ModuleNotFoundError: google.cloud.bigquery is the dead giveaway.
Step 7 — Sanity query (optional)
Once the next session has been used, suggest the adoption query:
SELECT JSON_VALUE(attributes, '$.writer.agent') AS agent,
JSON_VALUE(attributes, '$.writer.label') AS writer_label,
COUNT(*) AS events,
COUNT(DISTINCT session_id) AS sessions
FROM `<PROJECT>.<DATASET>.agent_events`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
GROUP BY agent, writer_label
ORDER BY events DESC
If agent='claude-code' rows appear, the loop is closed.
Troubleshooting jumps
If the user says "no rows", "rows not landing", "stuck", check in this order:
cat /tmp/bqaa-agent-tracing.log— last few lines have the actual error.ls /tmp/bqaa-agent-tracing/spool/dead-letter/— permanent failures live here as JSON.bq query --use_legacy_sql=false "SELECT COUNT(*) FROM \..agent_events` WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 MINUTE)"` — proves whether anything has landed.- If env vars look right but no rows, run the e2e smoke (Step 5) — it bypasses spool/drainer and writes inline, so it tells you whether IAM is the issue independent of the hook plumbing.
What NOT to do
- Don't run
gcloud auth loginorapplication-default loginyourself — they're interactive browser flows. - Don't run
--executewithout showing the dry-run plan first. - Don't grant
roles/bigquery.admin"to be safe" — the bootstrap defaults to the narrow runtime roles (dataEditoron dataset +jobUseron project) for a reason. - Don't generate service-account JSON keys. ADC + impersonation is the safer default; if the user insists, point them at Workload Identity Federation instead.