Amazon Bedrock Diagnostics
When to use
Any Amazon Bedrock investigation where the console alone is insufficient — model access and invocation issues, knowledge base configuration, agent orchestration, custom model training, guardrails configuration, performance optimization, security setup, or service integration.
Investigation workflow
Step 1 — Collect and triage
aws bedrock list-foundation-models --query 'modelSummaries[*].{Id:modelId,Name:modelName,Provider:providerName,Status:modelLifecycle.status}'
version: "1.0.0"
last_updated: "2025-04-12"
aws bedrock get-model-invocation-logging-configuration
aws bedrock list-custom-models
aws bedrock-agent list-agents --query 'agentSummaries[*].{Id:agentId,Name:agentName,Status:agentStatus}'
version: "1.0.0"
last_updated: "2025-04-12"
Step 2 — Domain deep dive
aws bedrock-agent list-knowledge-bases --query 'knowledgeBaseSummaries[*].{Id:knowledgeBaseId,Name:name,Status:status}'
version: "1.0.0"
last_updated: "2025-04-12"
aws bedrock-agent get-knowledge-base --knowledge-base-id <kb-id>
aws bedrock get-guardrail --guardrail-identifier <guardrail-id>
aws bedrock list-provisioned-model-throughputs
Step 3 — Detailed investigation
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=bedrock.amazonaws.com --max-results 20
aws cloudwatch get-metric-statistics --namespace AWS/Bedrock --metric-name Invocations --dimensions Name=ModelId,Value=<model-id> --start-time <start> --end-time <end> --period 300 --statistics Sum
aws bedrock-agent get-agent --agent-id <agent-id>
aws bedrock-agent list-agent-action-groups --agent-id <agent-id> --agent-version DRAFT
Read references/bedrock-guardrails.md before concluding on any Bedrock issue.
Tool quick reference
| Tool / API |
When to use |
bedrock list-foundation-models |
Check available models and status |
bedrock invoke-model |
Test model invocation directly |
bedrock-agent list-knowledge-bases |
List knowledge bases and status |
bedrock-agent get-knowledge-base |
Get KB configuration details |
bedrock-agent list-agents |
List agents and status |
bedrock-agent get-agent |
Get agent configuration |
bedrock get-guardrail |
Check guardrail configuration |
bedrock list-provisioned-model-throughputs |
Check provisioned throughput |
bedrock list-custom-models |
List fine-tuned models |
| CloudWatch metrics |
Monitor invocation counts, latency, errors |
Gotchas: Amazon Bedrock
- Model access must be explicitly requested and approved. Each foundation model requires separate access approval in the Bedrock console. Access is per-region. Without approval, InvokeModel returns AccessDeniedException.
- On-demand throughput has rate limits per model. Each model has different tokens-per-minute (TPM) and requests-per-minute (RPM) limits. Exceeding limits returns ThrottlingException. Provisioned throughput provides dedicated capacity.
- Knowledge base sync is asynchronous. After creating a KB or updating data sources, you must trigger a sync. Sync can take minutes to hours depending on data volume. Queries return stale results until sync completes.
- Agents use a multi-step orchestration loop. Agents break down tasks into steps, invoke action groups, and synthesize responses. Orchestration can loop if the agent cannot determine the next step. Max iterations and timeout settings control this.
- Guardrails evaluate both input and output. Content filters, topic denials, and word filters apply to both user input and model output. Overly restrictive guardrails block legitimate use cases. Test guardrails thoroughly before production.
- Custom model training requires specific S3 data format. Training data must be in JSONL format with specific schema. Training can take hours. Failed training jobs often stem from data format issues or insufficient IAM permissions.
- Provisioned throughput is billed even when idle. Provisioned model throughput provides dedicated capacity but incurs charges regardless of usage. Commitment terms (1-month, 6-month) affect pricing. No auto-scaling.
- VPC endpoints are required for private access. Bedrock API calls from private subnets require VPC endpoints. Both bedrock and bedrock-runtime endpoints may be needed. Security groups must allow HTTPS traffic.
- Streaming responses require specific client handling. InvokeModelWithResponseStream returns chunks. Client must handle the event stream protocol. Timeouts must account for full response generation time.
Anti-hallucination rules
- Always cite specific model IDs, KB IDs, agent IDs, or API responses as evidence.
- Model access requires explicit approval. Never assume a model is accessible.
- On-demand has rate limits. Never claim unlimited throughput without provisioned capacity.
- KB sync is async. Never assume data is immediately available after upload.
- Guardrails evaluate both input AND output. Never claim they only filter one direction.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
24 runbooks
| Category |
IDs |
Covers |
| A — Model |
A1-A3 |
Model access, invocation errors, throttling |
| B — Knowledge Base |
B1-B3 |
KB creation, sync failures, retrieval issues |
| C — Agents |
C1-C3 |
Agent creation, action group errors, orchestration |
| D — Fine-Tuning |
D1-D2 |
Custom model training, provisioned throughput |
| E — Guardrails |
E1-E2 |
Content filtering, topic denial |
| F — Performance |
F1-F2 |
Latency, token limits |
| G — Security |
G1-G2 |
IAM permissions, VPC config |
| H — Integration |
H1-H2 |
Lambda integration, streaming responses |
| Z — Catch-All |
Z1 |
General troubleshooting |
1---2name: bedrock-diagnostics3description: Use this skill to investigate and troubleshoot Amazon Bedrock problems by analyzing model access, invocation errors, throttling, knowledge base creation, sync failures, retrieval issues, agent creation, action group errors, orchestration problems, custom model training, provisioned throughput, content filtering, topic denial, latency, token limits, IAM permissions, VPC configuration, Lambda integration, streaming responses, and following structured runbooks. Activate when: model access denied, invocation errors, throttling/rate limits, knowledge base creation failures, data sync issues, poor retrieval quality, agent creation errors, action group failures, orchestration loops, fine-tuning failures, provisioned throughput issues, guardrail blocking legitimate content, latency problems, token limit exceeded, IAM permission errors, VPC endpoint issues, Lambda timeout with Bedrock, streaming response errors, or the user says something is wrong with Bedrock without naming specific symptoms.4---56# Amazon Bedrock Diagnostics78## When to use910Any Amazon Bedrock investigation where the console alone is insufficient — model access and invocation issues, knowledge base configuration, agent orchestration, custom model training, guardrails configuration, performance optimization, security setup, or service integration.1112## Investigation workflow1314### Step 1 — Collect and triage1516```17aws bedrock list-foundation-models --query 'modelSummaries[*].{Id:modelId,Name:modelName,Provider:providerName,Status:modelLifecycle.status}'18version: "1.0.0"19last_updated: "2025-04-12"20aws bedrock get-model-invocation-logging-configuration21aws bedrock list-custom-models22aws bedrock-agent list-agents --query 'agentSummaries[*].{Id:agentId,Name:agentName,Status:agentStatus}'23version: "1.0.0"24last_updated: "2025-04-12"25```2627### Step 2 — Domain deep dive2829```30aws bedrock-agent list-knowledge-bases --query 'knowledgeBaseSummaries[*].{Id:knowledgeBaseId,Name:name,Status:status}'31version: "1.0.0"32last_updated: "2025-04-12"33aws bedrock-agent get-knowledge-base --knowledge-base-id <kb-id>34aws bedrock get-guardrail --guardrail-identifier <guardrail-id>35aws bedrock list-provisioned-model-throughputs36```3738### Step 3 — Detailed investigation3940```41aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=bedrock.amazonaws.com --max-results 2042aws cloudwatch get-metric-statistics --namespace AWS/Bedrock --metric-name Invocations --dimensions Name=ModelId,Value=<model-id> --start-time <start> --end-time <end> --period 300 --statistics Sum43aws bedrock-agent get-agent --agent-id <agent-id>44aws bedrock-agent list-agent-action-groups --agent-id <agent-id> --agent-version DRAFT45```4647Read `references/bedrock-guardrails.md` before concluding on any Bedrock issue.4849## Tool quick reference5051| Tool / API | When to use |52|------------|-------------|53| `bedrock list-foundation-models` | Check available models and status |54| `bedrock invoke-model` | Test model invocation directly |55| `bedrock-agent list-knowledge-bases` | List knowledge bases and status |56| `bedrock-agent get-knowledge-base` | Get KB configuration details |57| `bedrock-agent list-agents` | List agents and status |58| `bedrock-agent get-agent` | Get agent configuration |59| `bedrock get-guardrail` | Check guardrail configuration |60| `bedrock list-provisioned-model-throughputs` | Check provisioned throughput |61| `bedrock list-custom-models` | List fine-tuned models |62| CloudWatch metrics | Monitor invocation counts, latency, errors |6364## Gotchas: Amazon Bedrock6566- Model access must be explicitly requested and approved. Each foundation model requires separate access approval in the Bedrock console. Access is per-region. Without approval, InvokeModel returns AccessDeniedException.67- On-demand throughput has rate limits per model. Each model has different tokens-per-minute (TPM) and requests-per-minute (RPM) limits. Exceeding limits returns ThrottlingException. Provisioned throughput provides dedicated capacity.68- Knowledge base sync is asynchronous. After creating a KB or updating data sources, you must trigger a sync. Sync can take minutes to hours depending on data volume. Queries return stale results until sync completes.69- Agents use a multi-step orchestration loop. Agents break down tasks into steps, invoke action groups, and synthesize responses. Orchestration can loop if the agent cannot determine the next step. Max iterations and timeout settings control this.70- Guardrails evaluate both input and output. Content filters, topic denials, and word filters apply to both user input and model output. Overly restrictive guardrails block legitimate use cases. Test guardrails thoroughly before production.71- Custom model training requires specific S3 data format. Training data must be in JSONL format with specific schema. Training can take hours. Failed training jobs often stem from data format issues or insufficient IAM permissions.72- Provisioned throughput is billed even when idle. Provisioned model throughput provides dedicated capacity but incurs charges regardless of usage. Commitment terms (1-month, 6-month) affect pricing. No auto-scaling.73- VPC endpoints are required for private access. Bedrock API calls from private subnets require VPC endpoints. Both bedrock and bedrock-runtime endpoints may be needed. Security groups must allow HTTPS traffic.74- Streaming responses require specific client handling. InvokeModelWithResponseStream returns chunks. Client must handle the event stream protocol. Timeouts must account for full response generation time.7576## Anti-hallucination rules77781. Always cite specific model IDs, KB IDs, agent IDs, or API responses as evidence.792. Model access requires explicit approval. Never assume a model is accessible.803. On-demand has rate limits. Never claim unlimited throughput without provisioned capacity.814. KB sync is async. Never assume data is immediately available after upload.825. Guardrails evaluate both input AND output. Never claim they only filter one direction.836. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.8485## 24 runbooks8687| Category | IDs | Covers |88|----------|-----|--------|89| A — Model | A1-A3 | Model access, invocation errors, throttling |90| B — Knowledge Base | B1-B3 | KB creation, sync failures, retrieval issues |91| C — Agents | C1-C3 | Agent creation, action group errors, orchestration |92| D — Fine-Tuning | D1-D2 | Custom model training, provisioned throughput |93| E — Guardrails | E1-E2 | Content filtering, topic denial |94| F — Performance | F1-F2 | Latency, token limits |95| G — Security | G1-G2 | IAM permissions, VPC config |96| H — Integration | H1-H2 | Lambda integration, streaming responses |97| Z — Catch-All | Z1 | General troubleshooting |
aws-samples