Contract Tester

contract-tester

Description: API contract validation, breaking change detection, and consumer-driven contract testing

Category: API Quality Assurance

Complexity: Medium-High (schema validation + contract compatibility)

Purpose

Validate API implementations against CTR (Contract) documents, detect breaking changes, and ensure provider-consumer compatibility. Prevents API contract violations before deployment.

Capabilities

1. CTR YAML Schema Validation

• Validate YAML structure against schema
• Check required fields completeness
• Verify data type correctness
• Validate enum values
• Check format specifications (email, URL, UUID)

2. OpenAPI/AsyncAPI Compliance

• Convert CTR to OpenAPI 3.0/3.1 spec
• Validate against OpenAPI schema
• AsyncAPI validation for event-driven APIs
• JSON Schema validation
• GraphQL schema validation

3. Contract Versioning

• Semantic versioning validation (MAJOR.MINOR.PATCH)
• Version compatibility checking
• Deprecation warnings
• Migration path validation

4. Breaking Change Detection

• Removed endpoints/operations
• Changed request/response schemas
• Removed required fields
• Changed field types
• Stricter validation rules
• Changed authentication requirements

5. Mock Server Generation

• Generate mock API server from CTR
• Realistic response generation
• Configurable response delays
• Error scenario simulation
• State management for stateful APIs

6. Consumer-Driven Contract Testing

• Pact contract testing
• Provider verification
• Consumer contract generation
• Bidirectional contract testing
• Contract evolution tracking

7. Contract Compatibility Matrix

• Provider version compatibility
• Consumer version support
• Deprecation timeline tracking
• Migration requirements

8. API Documentation Generation

• Generate API documentation from CTR
• Interactive API explorer
• Code examples in multiple languages
• Authentication guides

Contract Testing Workflow

graph TD
    A[CTR Document] --> B[Parse YAML]
    B --> C{Valid YAML?}
    C -->|No| D[Report Parse Error]
    C -->|Yes| E[Schema Validation]

    E --> F{Schema Valid?}
    F -->|No| G[Report Schema Errors]
    F -->|Yes| H[Version Check]

    H --> I{Version Valid?}
    I -->|No| J[Report Version Error]
    I -->|Yes| K[Breaking Change Detection]

    K --> L{Previous Version Exists?}
    L -->|No| M[First Version]
    L -->|Yes| N[Compare Versions]

    N --> O{Breaking Changes?}
    O -->|Yes| P[Report Breaking Changes]
    O -->|No| Q[Compatible Change]

    M --> R[Generate Mock Server]
    Q --> R
    P --> R

    R --> S[Run Contract Tests]
    S --> T{Tests Pass?}
    T -->|No| U[Report Test Failures]
    T -->|Yes| V[Generate Documentation]

    V --> W[Validation Complete]

    D --> W
    G --> W
    J --> W
    U --> W

Usage Instructions

Validate CTR Document

contract-tester validate --file ctrs/CTR-USER-V1.yaml

Output:

=== Contract Validation Report: CTR-USER-V1 ===

Status: PASS

Schema Validation: ✓
- Service name: user-service
- Version: 1.0.0
- Protocol: REST
- Endpoints: 5
- Models: 3

OpenAPI Compliance: ✓
- Valid OpenAPI 3.1 specification
- All schemas properly defined
- Authentication specified (Bearer JWT)

Version Check: ✓
- Semantic versioning: 1.0.0
- No previous version (initial release)

Breaking Changes: N/A (initial release)

Warnings (1):
- Endpoint DELETE /users/{id} lacks rate limiting specification
  → Recommendation: Add rate limiting to prevent abuse

Summary: Contract is valid and ready for implementation

Detect Breaking Changes

contract-tester diff \
  --previous ctrs/CTR-USER-V1.yaml \
  --current ctrs/CTR-USER-V2.yaml

Output:

=== Breaking Change Analysis: V1 → V2 ===

BREAKING CHANGES (3):

1. [CRITICAL] Removed endpoint
   - DELETE /users/{id}
   - Impact: Consumers using this endpoint will fail
   - Migration: Use DELETE /users/{id}/deactivate instead

2. [CRITICAL] Changed response type
   - Endpoint: GET /users/{id}
   - Field: created_at
   - V1: string (format: date-time)
   - V2: integer (unix timestamp)
   - Impact: Date parsing will fail in consumers
   - Migration: Update consumers to handle unix timestamp

3. [MAJOR] Added required field
   - Endpoint: POST /users
   - Field: email_verified (boolean, required)
   - Impact: Existing POST requests missing this field will fail
   - Migration: Update all POST requests to include email_verified

NON-BREAKING CHANGES (2):

1. [MINOR] Added optional field
   - Endpoint: GET /users/{id}
   - Field: last_login (string, optional)
   - Impact: None (backward compatible)

2. [PATCH] Fixed typo in description
   - Model: User
   - Field: username
   - Impact: Documentation only

RECOMMENDATION: Version should be 2.0.0 (MAJOR bump due to breaking changes)
Current version: 1.1.0 ❌
Suggested version: 2.0.0 ✓

Generate Mock Server

contract-tester mock \
  --file ctrs/CTR-USER-V1.yaml \
  --port 8080 \
  --response-delay 100ms

Mock server features:

• Realistic data generation based on schemas
• Configurable response delays
• Error scenario injection
• Request validation
• Response logging

# Test against mock server
curl http://localhost:8080/api/users/123

# Response
{
  "user_id": "123e4567-e89b-12d3-a456-426614174000",
  "username": "john_doe",
  "email": "john.doe@example.com",
  "created_at": "2025-01-15T10:30:00Z",
  "status": "active"
}

Run Consumer Contract Tests

contract-tester test-consumer \
  --contract ctrs/CTR-USER-V1.yaml \
  --consumer user-frontend \
  --provider user-service \
  --output reports/contract-tests/

Generated Pact test:

import pytest
from pact import Consumer, Provider, Like, EachLike, Term

@pytest.fixture
def pact():
    return Consumer('user-frontend').has_pact_with(
        Provider('user-service'),
        pact_dir='./pacts'
    )

def test_get_user_contract(pact):
    """Verify GET /users/{id} contract"""
    expected_response = {
        'user_id': Like('123e4567-e89b-12d3-a456-426614174000'),
        'username': Like('john_doe'),
        'email': Term(r'.+@.+\..+', 'john.doe@example.com'),
        'created_at': Like('2025-01-15T10:30:00Z'),
        'status': Term(r'active|inactive|suspended', 'active')
    }

    (pact
     .given('user with id 123 exists')
     .upon_receiving('a request for user 123')
     .with_request('GET', '/api/users/123')
     .will_respond_with(200, body=expected_response))

    with pact:
        result = user_api.get_user('123')
        assert result.status_code == 200
        assert 'user_id' in result.json()

Verify Provider Implementation

contract-tester test-provider \
  --contract ctrs/CTR-USER-V1.yaml \
  --provider-url http://localhost:8000 \
  --output reports/provider-verification/

CTR Document Structure

Complete CTR Example

# ctrs/CTR-USER-V1.yaml
contract_id: CTR-USER-V1
version: 1.0.0
service_name: user-service
protocol: REST
base_path: /api

metadata:
  description: User management API contract
  owner: backend-team
  consumers:
    - user-frontend
    - admin-dashboard
  traceability:
    requirements:
      - REQ-USER-001
      - REQ-USER-002
    specifications:
      - SPEC-API-V1

authentication:
  type: bearer
  scheme: JWT
  token_location: header
  header_name: Authorization

endpoints:
  - operation_id: get_user
    method: GET
    path: /users/{id}
    summary: Retrieve user by ID
    parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          format: uuid
    responses:
      200:
        description: User found
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/User'
      404:
        description: User not found
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Error'
      401:
        description: Unauthorized
    rate_limit:
      requests: 100
      window: 60s

  - operation_id: create_user
    method: POST
    path: /users
    summary: Create new user
    request_body:
      required: true
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/CreateUserRequest'
    responses:
      201:
        description: User created
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/User'
      400:
        description: Invalid request
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Error'
      409:
        description: User already exists

  - operation_id: update_user
    method: PUT
    path: /users/{id}
    summary: Update user
    parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          format: uuid
    request_body:
      required: true
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/UpdateUserRequest'
    responses:
      200:
        description: User updated
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/User'
      404:
        description: User not found

  - operation_id: delete_user
    method: DELETE
    path: /users/{id}
    summary: Delete user
    parameters:
      - name: id
        in: path
        required: true
        schema:
          type: string
          format: uuid
    responses:
      204:
        description: User deleted
      404:
        description: User not found

  - operation_id: list_users
    method: GET
    path: /users
    summary: List all users
    parameters:
      - name: page
        in: query
        schema:
          type: integer
          minimum: 1
          default: 1
      - name: limit
        in: query
        schema:
          type: integer
          minimum: 1
          maximum: 100
          default: 20
    responses:
      200:
        description: User list
        content:
          application/json:
            schema:
              type: object
              properties:
                users:
                  type: array
                  items:
                    $ref: '#/components/schemas/User'
                pagination:
                  $ref: '#/components/schemas/Pagination'

components:
  schemas:
    User:
      type: object
      required:
        - user_id
        - username
        - email
        - created_at
        - status
      properties:
        user_id:
          type: string
          format: uuid
          description: Unique user identifier
        username:
          type: string
          minLength: 3
          maxLength: 20
          pattern: '^[a-zA-Z0-9_]+$'
          description: Unique username
        email:
          type: string
          format: email
          description: User email address
        full_name:
          type: string
          maxLength: 100
          description: User's full name
        created_at:
          type: string
          format: date-time
          description: Account creation timestamp
        updated_at:
          type: string
          format: date-time
          description: Last update timestamp
        status:
          type: string
          enum: [active, inactive, suspended]
          description: Account status

    CreateUserRequest:
      type: object
      required:
        - username
        - email
        - password
      properties:
        username:
          type: string
          minLength: 3
          maxLength: 20
          pattern: '^[a-zA-Z0-9_]+$'
        email:
          type: string
          format: email
        password:
          type: string
          minLength: 8
          maxLength: 128
        full_name:
          type: string
          maxLength: 100

    UpdateUserRequest:
      type: object
      properties:
        email:
          type: string
          format: email
        full_name:
          type: string
          maxLength: 100
        status:
          type: string
          enum: [active, inactive, suspended]

    Error:
      type: object
      required:
        - error_code
        - message
      properties:
        error_code:
          type: string
          description: Machine-readable error code
        message:
          type: string
          description: Human-readable error message
        details:
          type: object
          description: Additional error details

    Pagination:
      type: object
      required:
        - page
        - limit
        - total
      properties:
        page:
          type: integer
          description: Current page number
        limit:
          type: integer
          description: Items per page
        total:
          type: integer
          description: Total number of items
        has_next:
          type: boolean
          description: Whether there are more pages

Breaking Change Rules

MAJOR Version Changes (X.0.0)

1. Removed Endpoints

   # V1
   DELETE /users/{id}
   
   # V2 (BREAKING)
   # Endpoint removed
   

2. Removed Required Fields

   # V1
   required: [username, email, password]
   
   # V2 (BREAKING)
   required: [username, email]  # password removed
   

3. Changed Field Types

   # V1
   user_id:
     type: string
   
   # V2 (BREAKING)
   user_id:
     type: integer
   

4. Stricter Validation

   # V1
   username:
     type: string
   
   # V2 (BREAKING)
   username:
     type: string
     pattern: '^[a-zA-Z0-9_]+$'  # New constraint
   

5. Changed Authentication

   # V1
   authentication:
     type: basic
   
   # V2 (BREAKING)
   authentication:
     type: bearer
     scheme: JWT
   

MINOR Version Changes (x.Y.0)

1. New Endpoints

   # V1.0.0
   GET /users
   
   # V1.1.0 (Non-breaking)
   GET /users
   GET /users/search  # New endpoint
   

2. New Optional Fields

   # V1.0.0
   properties:
     username: ...
   
   # V1.1.0 (Non-breaking)
   properties:
     username: ...
     nickname: ...  # New optional field
   

3. Relaxed Validation

   # V1.0.0
   username:
     minLength: 5
   
   # V1.1.0 (Non-breaking)
   username:
     minLength: 3  # Less strict
   

PATCH Version Changes (x.y.Z)

1. Documentation Updates
2. Bug Fixes in Descriptions
3. Typo Corrections
4. Example Updates

Contract Compatibility Matrix

Provider Version Compatibility:

V1.0.0 consumers can use:
- V1.0.0, V1.0.1, V1.1.0, V1.2.0 providers ✓
- V2.0.0 providers ❌ (breaking changes)

V1.1.0 consumers can use:
- V1.1.0, V1.1.1, V1.2.0 providers ✓
- V1.0.0 providers ⚠️ (missing optional features)
- V2.0.0 providers ❌ (breaking changes)

V2.0.0 consumers can use:
- V2.0.0, V2.0.1, V2.1.0 providers ✓
- V1.x.x providers ❌ (incompatible)

Tool Access

Required tools:

• Read: Read CTR YAML files
• Write: Generate test files and documentation
• Bash: Execute validation and testing tools
• Grep: Search for contract patterns

Required libraries:

• pyyaml: YAML parsing
• jsonschema: Schema validation
• openapi-spec-validator: OpenAPI validation
• pact-python: Consumer-driven contract testing
• faker: Mock data generation

Integration Points

With doc-flow

• Validate CTR documents during generation
• Ensure traceability to requirements
• Update documentation on contract changes

With test-automation

• Generate contract tests automatically
• Integrate with test suites
• Track contract test coverage

With code-review

• Validate API implementation against contracts
• Detect contract violations in code
• Suggest contract-compliant fixes

With devops-flow

• Contract validation in CI/CD pipelines
• Block deployments with breaking changes
• Generate API gateway configurations

Best Practices

1. Semantic versioning: Follow MAJOR.MINOR.PATCH strictly
2. Backward compatibility: Prefer additive changes
3. Deprecation notices: Warn before removing features
4. Consumer feedback: Involve consumers in contract changes
5. Documentation: Clear migration guides for breaking changes
6. Testing: Test all contract changes before release
7. Versioning strategy: Support N-1 versions minimum
8. Change log: Maintain comprehensive contract changelog

Success Criteria

• 100% contract validation coverage
• Zero undetected breaking changes
• <5 minute contract validation time
• 100% consumer-provider compatibility in same MAJOR version
• All breaking changes documented with migration paths
• Mock servers available for all contracts
• All CTR documents include complete Document Control sections with:
  • Project metadata (name, version, date, owner, preparer, status)
  • Document Revision History table tracking all contract changes

Notes

• Contract tests run in CI/CD before deployment
• Breaking changes require MAJOR version bump
• Provider must maintain backward compatibility within MAJOR version
• Consumers should handle minor version differences gracefully
• Contract changes trigger automatic notification to consumers