tools-only dbt Job Failure Debugging
Goal
Test whether the LLM can effectively use MCP tools to diagnose a dbt Cloud job failure.
Expected Behavior
The LLM should:
- Use the dbt MCP server to list recent job runs
- Identify the failed run from January 14th
- Get detailed error information using
get_job_run_error - Classify the error type and investigate appropriately
- Either find the root cause or document what was checked
Evaluation Criteria
- Tool Usage: Did it use the MCP tools effectively?
- Systematic: Did it follow a logical debugging process?
- Documentation: Did it document findings appropriately?
Prerequisites
This scenario requires:
- dbt MCP server running with valid credentials
- Access to a dbt Cloud account with job history