majiayu000 UCP Conformance Testing
Before running tests
Fetch the latest test suite: Web-search github Universal-Commerce-Protocol conformance and fetch the README for current setup instructions, test data format, and CLI flags.
Repository: https://github.com/Universal-Commerce-Protocol/conformance
Conceptual Architecture
What the Conformance Suite Tests
The official suite contains ~13 test files covering:
| Test File | What It Validates |
|---|---|
| Checkout lifecycle | Full create → update → complete → verify flow |
| Order management | Order creation, status transitions, data integrity |
| Fulfillment | Shipping/pickup methods, group selection, option validation |
| Card credentials | Payment credential format, tokenization |
| Webhooks | Delivery, retry, signature verification |
| Idempotency | Duplicate request handling, cache behavior |
| Invalid input | Error responses for malformed requests |
| Protocol compliance | Headers, TLS, version negotiation |
| Data validation | Schema compliance, required fields, type correctness |
| Service bindings | REST/MCP/A2A transport correctness |
| Business logic | Totals calculation, tax, discount application |
| AP2 integration | Mandate generation, signing, verification |
| Security (simulation URLs) | URL validation, injection prevention |
How to Run
- Clone the conformance repo
- Install dependencies with
uv sync - Start your UCP server locally
- Run tests pointing at your server:
uv run checkout_lifecycle_test.py \
--server_url=http://localhost:8182 \
--simulation_secret=your-secret \
--conformance_input=test_data/your_store/conformance_input.json
Test Data
Tests require a conformance_input.json file that describes your store's products, prices, and expected behaviors. Check the sample test data in the repo for the format.
Writing Custom Conformance Tests
When extending UCP with custom capabilities:
- Use the same test patterns as the official suite
- Test capability negotiation (extension present/absent)
- Test schema validation against your custom schemas
- Test error cases specific to your extension
- Test idempotency for all mutating operations
Integration with CI/CD
- Run conformance tests as part of your CI pipeline
- Use the
--server_urlflag to point at staging environments - Track test pass rates as a quality gate for deployments
Always fetch the latest test suite before running — new tests are added as the spec evolves.