# Troubleshoot

> Diagnose and fix common ABCA issues: deployment failures, preflight errors, authentication problems, agent failures, and build issues. Use when the user says "troubleshoot", "debug", "not working", "error", "failed", "help me fix", "preflight_failed", "task failed", "deploy failed", "auth error", "401", "422", "503", or describes something not working as expected.

- Skill: `aws-samples/troubleshoot` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aws-samples/troubleshoot`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aws-samples/troubleshoot/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: aws-samples (https://skillmd.com/u/aws-samples)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/aws-samples/troubleshoot

---


# ABCA Troubleshooting

You are diagnosing an issue with the ABCA platform. Follow a systematic approach: gather symptoms, check the most common causes, and apply targeted fixes.

> **Running the CLI:** commands below call `node cli/lib/bin/bgagent.js …`. In a non-interactive or mise-managed shell `node` may not be on `PATH` — prefix with `mise exec --`. Ironically, `node: command not found` is itself a common symptom (the shell hasn't activated mise); that's a missing prefix, not a broken install.

## Step 1: Identify the Problem Category

Determine which area the issue falls into:

1. **Build/Compilation** — TypeScript errors, test failures, lint issues
2. **Deployment** — CDK deploy/synth failures, CloudFormation errors
3. **Authentication** — Cognito errors, token issues, 401 responses
4. **Task Submission** — 422 errors, validation failures, guardrail blocks
5. **Task Execution** — Preflight failures, agent failures, timeouts
6. **Local Agent Testing** — Docker issues, run.sh problems

## Build/Compilation Issues

```bash
export MISE_EXPERIMENTAL=1
mise //cdk:compile 2>&1 | tail -50  # TypeScript errors
mise //cdk:test 2>&1 | tail -50     # Test failures
```

**Common causes:**
- Missing `mise run install` after pulling changes
- `yarn: command not found` — Run `corepack enable && corepack prepare yarn@1.22.22 --activate`
- Type mismatches after editing `cdk/src/handlers/shared/types.ts` without updating `cli/src/types.ts`

## Deployment Issues

```bash
# Check CloudFormation events for the failed stack
aws cloudformation describe-stack-events --stack-name backgroundagent-dev \
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED` || ResourceStatus==`UPDATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' \
  --output table
```

**Common causes:**
- Docker not running — Required for CDK asset bundling
- Missing CDK bootstrap — Run `mise //cdk:bootstrap`
- IAM permission issues — Check `aws sts get-caller-identity`
- Region mismatch — Ensure consistent region across all commands

## Authentication Issues

```bash
# Verify credentials
aws sts get-caller-identity

# Check Cognito user exists
aws cognito-idp admin-get-user \
  --user-pool-id $USER_POOL_ID \
  --username user@example.com
```

**Common causes:**
- "App client does not exist" — Region mismatch between CLI config and stack deployment
- Token expired — Re-authenticate with `bgagent login`
- 401 on API calls — Token not included or malformed in Authorization header
- User not created — Self-signup is disabled; admin must create users

## Task Submission Issues (422 / 400)

**"Repository not onboarded" / `REPO_NOT_ONBOARDED` (422):**
- The repo isn't registered. Fastest fix: `bgagent repo onboard <owner/repo>` (operator path — writes the RepoTable record at runtime, no redeploy). A CDK Blueprint is only needed for declarative config. Use the `onboard-repo` skill for details.
- Also confirm the `owner/repo` matches **exactly** what you pass to `bgagent submit --repo`.

**"GUARDRAIL_BLOCKED" (400):**
- Task description triggered Bedrock Guardrails content screening
- Review and rephrase the task description to remove potentially flagged content

**Validation errors:**
- Check required fields: `repo` is required, plus at least one of `issue_number`, `task_description`, `pr_number`
- `max_turns` range: 1-500
- `max_budget_usd` range: $0.01-$100

## Task Execution Issues

```bash
# Check task events for details
node cli/lib/bin/bgagent.js events <TASK_ID> --output json
```

**`preflight_failed`:**
- GitHub PAT lacks permissions for the repo
- Repository doesn't exist or is private without proper token scope
- Check event `reason` and `detail` fields for specifics
- Verify PAT: fine-grained token must include the target repository with Contents (read/write), Pull Requests (read/write), Issues (read)

**`task_failed` / task completes with 0 tokens and no PR:**
- Agent encountered an error during execution
- Check CloudWatch logs for the session:
  ```bash
  aws logs filter-log-events \
    --log-group-name "/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/jean_cloude" \
    --filter-pattern "<TASK_ID>" \
    --region us-west-2 --query 'events[*].message' --output text
  ```
- Common: repo build/test commands not documented in CLAUDE.md

**403 "not authorized to perform bedrock:InvokeModelWithResponseStream":**
- The repo's `model_id` is a model the runtime IAM role wasn't granted. The runtime only has `grantInvoke` for the models in the stack's configured set — read it from the `BedrockModelIds` stack output rather than a list here (Sonnet 4.6, Opus 4.8, Opus 5, Haiku 4.5 by default).
- **Quick fix:** point the repo at an already-granted model — `bgagent repo onboard <owner/repo> --model global.anthropic.claude-opus-5` (no redeploy).
- **To add a new model to the runtime:** grant it in the stack and redeploy. The model set is the shared list in `cdk/src/constructs/bedrock-models.ts` — add the model via the `bedrockModels` CDK context (`cdk.json`) so both the AgentCore and ECS backends grant it (#433). Adding a model also requires **account-level Bedrock access** for it (separate from IAM — see the next row).

**Model not enabled / "not available on your Bedrock deployment" (often immediate failure, few turns, zero or near-zero tokens):**
- **IAM is necessary but not sufficient.** The AgentCore role may already have `bedrock:InvokeModel*`, but the **account** must also satisfy [Amazon Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html): Marketplace subscription flow on first serverless use (with `aws-marketplace:Subscribe` / `ViewSubscriptions` where needed), Anthropic **first-time use** details (`PutUseCaseForModelAccess` or the console model catalog), and a valid payment method for Marketplace-backed models.
- **Use an inference profile ID** in the Blueprint / DynamoDB `model_id` when Bedrock requires it for on-demand invocation (for example `global.anthropic.claude-opus-5` for global Opus 5). See [Use an inference profile in model invocation](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-use.html). Raw `anthropic.*` IDs often hit "on-demand not supported" or wrong routing — see the **400** row below.
- **Cross-Region profiles** route across Regions in a geography; ensure IAM and any SCPs allow Bedrock in **all destination Regions** for that profile. See [Supported Regions and models for inference profiles](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html).
- **Task status:** When the Claude CLI reports a terminal error via `ResultMessage.is_error`, the agent marks the task **FAILED** (not COMPLETED) and persists `error_message` in DynamoDB.

**400 "Invocation with on-demand throughput isn't supported":**
- The Blueprint `modelId` uses a raw foundation model ID (e.g. `anthropic.claude-opus-4-8`)
- Fix: change to the inference profile ID, prefixed with the geography the stack grants —
  its `BedrockGeoRegion` output (e.g. `global.anthropic.claude-opus-4-8`) — then update
  DynamoDB via redeploy. A prefix from a *different* geography raises `AccessDenied`
  rather than this 400, since the IAM grant is scoped per geography.

**503 "Too many connections" / task completes with 0 tokens after long duration:**
- Bedrock is throttling model invocations. The agent retries for minutes then gives up.
- Symptoms: task runs for 10-15 minutes, may end with `COMPLETED` if the SDK does not flag `ResultMessage.is_error` (unlike hard Bedrock entitlement errors, which surface as **FAILED** once the CLI sets `is_error` on the result)
- Diagnosis:
  1. Check application logs for `"text": "API Error: 503 Too many connections"`
  2. **Check what model_id is actually being passed** — the DynamoDB record may have a stale model override:
     ```bash
     aws dynamodb get-item \
       --table-name <RepoTableName> \
       --key '{"repo": {"S": "owner/repo"}}' \
       --query 'Item.model_id' --output text
     ```
- Causes:
  - **Stale model_id in DynamoDB** (most common) — the Blueprint `onUpdate` only sets fields present in props; removing a `modelId` prop does NOT remove the field from DynamoDB. The task keeps using the old model.
  - Bedrock service-level throttling for the specific model (Opus-class models have tighter limits than Sonnet or Haiku)
  - Account quota limits reached
- Fix:
  1. **Check and fix the DynamoDB record first** — remove stale `model_id` if present:
     ```bash
     aws dynamodb update-item \
       --table-name <RepoTableName> \
       --key '{"repo": {"S": "owner/repo"}}' \
       --update-expression "REMOVE model_id"
     ```
  2. If model_id is correct, wait and retry — throttling is often transient
  3. Switch to a model with higher availability (Haiku 4.5 > Sonnet 4.6 > Opus)
  4. Request a Bedrock quota increase for `InvokeModel` RPM on your model

**`task_timed_out`:**
- 9-hour maximum exceeded
- Consider reducing scope or increasing `max_turns` for complex tasks
- Check if the agent is stuck in a loop (review logs)

**Concurrency limit:**
- Default: 3 concurrent tasks per user
- Wait for running tasks to complete or cancel them

## Local Agent Testing Issues

```bash
# Verify Docker is running
docker info

# Test locally with dry run
DRY_RUN=1 ./agent/run.sh "owner/repo" "Test task"
```

**Common causes:**
- Missing environment variables: `GITHUB_TOKEN`, `AWS_REGION`
- Docker not running or insufficient resources (needs 2 vCPU, 8 GB RAM)
- Missing AWS credentials for Bedrock access

## Diagnostic Commands Quick Reference

```bash
# Stack status
aws cloudformation describe-stacks --stack-name backgroundagent-dev --query 'Stacks[0].StackStatus'

# Stack outputs
aws cloudformation describe-stacks --stack-name backgroundagent-dev --query 'Stacks[0].Outputs' --output table

# Task status (use --verbose for HTTP-level debug output)
node cli/lib/bin/bgagent.js --verbose status <TASK_ID>
node cli/lib/bin/bgagent.js events <TASK_ID> --output json

# Watch task progress in real time
node cli/lib/bin/bgagent.js watch <TASK_ID>

# Download full execution trace (task must have been submitted with --trace)
node cli/lib/bin/bgagent.js trace download <TASK_ID>

# List running tasks
node cli/lib/bin/bgagent.js list --status RUNNING

# Build health
mise run build
```

**Tip:** Add `--verbose` to any `bgagent` command to see the full HTTP request/response cycle on stderr. This is the fastest way to diagnose auth, network, or API contract issues.