# Debug CI

> Diagnose failed Spellbook PR CI workflows, dbt slim CI steps, missing seeds, stale manifests, cluster issues, and CI test table queries.

- Skill: `duneanalytics/debug-ci` (Agent Skill)
- Install (CLI): `npx skillmds@latest add duneanalytics/debug-ci`
- Raw SKILL.md: https://api.skillmd.com/api/skills/duneanalytics/debug-ci/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: duneanalytics (https://skillmd.com/u/duneanalytics)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/duneanalytics/debug-ci

---


# Debug CI Failures

Use this skill when a Spellbook PR's CI tests fail and you need to diagnose the issue.

## Step 1: Identify Which Sub-Project Failed

Each sub-project has its own CI workflow (`daily_spellbook.yml`, `dex.yml`, `hourly_spellbook.yml`, `nft.yml`, `solana.yml`, `tokens.yml`). All call the shared reusable workflow in `.github/workflows/dbt_run.yml`.

Check the PR's "Checks" tab to see which workflow(s) failed.

## Step 2: Identify Which Step Failed

The CI pipeline runs these steps in order:

1. **`dbt seed`** — builds seed CSVs modified in the PR
2. **`dbt run initial model(s)`** — full historical build of modified models
3. **`dbt test initial model(s)`** — runs tests on the initial build
4. **`dbt run incremental model(s)`** — incremental run to test incremental logic
5. **`dbt test incremental model(s)`** — tests again post-incremental run

Click into the failed workflow run in the "Actions" tab to expand each step's logs.

## Step 3: Diagnose Common Failures

### Compile / Seed Failures
- **Missing source or ref**: Check that all `source()` and `ref()` calls point to valid models. Run `dbt compile` locally to verify.
- **Missing seed metadata**: If error says "Metadata is not found for ___", the seed likely doesn't exist in prod. Force a change on the seed file and include it in the PR so CI rebuilds it.

### Model Run Failures
- **SQL syntax errors**: Check the compiled SQL in CI logs. Run `dbt compile` locally and test the output on Dune or via `dune_query.py`.
- **Timeout (90-minute limit)**: Simplify the model, add tighter date filters, or check for missing partition filters.
- **Schema not assigned**: Every model must have `schema` in its config block. The CI check-schemas script will fail otherwise.

### Test Failures
- **`unique_combination_of_columns` failing**: The model is producing duplicate rows for the unique key. Check your join logic, filters, and whether NULLs are present in key columns.
- **`not_null` failing**: A unique key column contains NULLs. Add `coalesce()` or filter out NULL rows.
- **Seed test mismatch**: Model output doesn't match seed expected values. Either update the seed CSV or fix the model logic. Query the CI test table on Dune to compare: `dune.<ci_schema>.<model_name>` (see Step 4).

### Manifest / Extra Models Running
- **More models running than expected**: The manifest file on main may be out of date. Check if the `commit manifest` workflow in Actions completed successfully. If it's still running or failed, wait for it to finish, then re-trigger CI.
- **Fix**: The Dune team's `commit manifest` workflow auto-runs after merges. If it failed, the Dune team needs to re-run it.

### Cluster Issues
- **DuneSQL cluster down**: The CI cluster may be offline. This requires the Dune team to fix internally. Check if other PRs are also failing.

## Step 4: Query CI Test Tables

CI tables are available on Dune for ~24 hours after the run. **Always use the `dune.` catalog prefix.**

**Current format** (from `dbt run initial model(s)` logs):

```sql
select *
from dune.dune_spellbook_ci__tmp_pr<PR>_<run_id>_<attempt>.<model_name>
limit 100
```

Example: `dune.dune_spellbook_ci__tmp_pr9744_27302471352_1.zeroex_bnb_api_fills`

**Legacy format:**

```sql
select *
from dune.test_schema.git_dunesql_<GIT_HASH>_<schema>_<alias>
limit 100
```

For legacy tables, the suffix matches dbt **`schema`** + **`_`** + **`alias`** (e.g. `tokens.transfers` → `tokens_transfers`). For current CI, use the dbt **model name** from logs.

Use these to:
- Verify data quality before expanding date ranges
- Compare CI output against production tables
- Debug seed test mismatches by joining CI table against the seed

If the CI table can't be found, the ~24-hour window may have elapsed. Re-run CI to rebuild.

## Step 5: Common Fixes

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| "Metadata is not found" | Seed missing from prod and PR | Include seed file in PR with a small change |
| Duplicate rows in unique test | NULLs in unique key or bad join | Add `coalesce()` / fix join logic |
| Seed test values mismatch | Model logic changed or seed stale | Update seed CSV or fix model SQL |
| Extra models running | Manifest out of date | Wait for `commit manifest` workflow, re-run CI |
| Timeout at 90 min | Query too expensive | Add tighter date filters, check partition pruning |
| `prices.usd` token not showing | Backend backfill pending | Wait a few days for historical pricing to populate |