lobehub OpenResponses Compliance Test
Run the official OpenResponses compliance test suite against the local (or remote) Response API endpoint.
Quick Start
# From the openapi package directory
cd lobehub/packages/openapi
# Run all tests (dev mode, localhost:3010)
APP_URL=http://localhost:3010 bun run test:response-compliance -- \
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1
# Run specific tests only
APP_URL=http://localhost:3010 bun run test:response-compliance -- \
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1 \
--filter basic-response,streaming-response
# Verbose mode (shows request/response details)
APP_URL=http://localhost:3010 bun run test:response-compliance -- \
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1 -v
# JSON output (for CI)
APP_URL=http://localhost:3010 bun run test:response-compliance -- \
--auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1 --json
Prerequisites
- Dev server running with
ENABLE_MOCK_DEV_USER=truein.env - The
api/v1/responsesroute registered (viasrc/app/(backend)/api/v1/[[...route]]/route.ts)
Auth Modes
| Mode | Flags |
|---|---|
| Dev (mock user) | --auth-header "lobe-auth-dev-backend-api" --no-bearer --api-key 1 |
| API Key | --api-key lb-xxxxxxxxxxxxxxxx |
| Custom | --auth-header <name> --api-key <value> |
Test IDs
Available --filter values:
| ID | Description | Related Issue |
|---|---|---|
basic-response |
Simple text generation (non-streaming) | LOBE-5858 |
streaming-response |
SSE streaming lifecycle + events | LOBE-5859 |
system-prompt |
System role message handling | LOBE-5858 |
tool-calling |
Function tool definition + call output | LOBE-5860 |
image-input |
Multimodal image URL content | — |
multi-turn |
Conversation history via input items | LOBE-5861 |
Environment Variables
| Variable | Default | Description |
|---|---|---|
APP_URL |
http://localhost:3010 |
Server base URL (auto-appends /api/v1) |
API_KEY |
— | API key (alternative to --api-key flag) |
How It Works
The script (lobehub/packages/openapi/scripts/compliance-test.sh) clones the official openresponses/openresponses repo into scripts/openresponses-compliance/ (gitignored) and runs its CLI test runner. First run clones; subsequent runs update from upstream.
Debugging Failures
- Run with
-vto see full request/response payloads - Common failure patterns:
- "Failed to parse JSON": Auth failed, server returned HTML redirect
- "Response has no output items": LLM execution not yet implemented
- "Expected number, received null": Missing required field in response schema
- "Invalid input": Zod validation on response schema — check field format
Key Files
- Types:
lobehub/packages/openapi/src/types/responses.type.ts - Service:
lobehub/packages/openapi/src/services/responses.service.ts - Controller:
lobehub/packages/openapi/src/controllers/responses.controller.ts - Route:
lobehub/packages/openapi/src/routes/responses.route.ts - Test script:
lobehub/packages/openapi/scripts/compliance-test.sh - Cloud route:
src/app/(backend)/api/v1/[[...route]]/route.ts