# Gateway Integration Tests

> Gateway Integration Tests

- Skill: `wso2/gateway-integration-tests` (Agent Skill)
- Install (CLI): `npx skillmds@latest add wso2/gateway-integration-tests`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wso2/gateway-integration-tests/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: wso2 (https://skillmd.com/u/wso2)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/wso2/gateway-integration-tests

---


# Gateway Integration Tests

Runs the WSO2 API Platform gateway BDD integration suite (`gateway/it/`, Godog +
Docker Compose) for a chosen subset of feature files, diagnoses failures from
container logs, and cleans up afterward.

## When to use

- "Run the gateway integration tests" / "run `<x>.feature`"
- "Verify my new feature file passes"
- "Why is this Integration Test failing?" — triage from container logs
- After editing a `.feature` file, a policy, or gateway code, to confirm behavior

## Workflow

Always run these steps in order:

1. **Build the gateway images** — `cd gateway && make build-coverage` (see Step 1).
   Do this first, every time, so the test stack runs the current code.
2. **Run the tests** for the chosen feature files via `IT_FEATURE_PATHS` (see Step 2).
3. **Troubleshoot** from container logs if anything fails (see Step 3).
4. **Clean up** the stack with `make clean` (see Step 4).

## Layout

- Suite dir: `gateway/it/` — run all `make` commands from here.
- Feature files: `gateway/it/features/*.feature`. Default suite list (and how
  `IT_FEATURE_PATHS` overrides it) is in `gateway/it/suite_test.go` → `getFeaturePaths()`.
- Compose stack: `gateway/it/docker-compose.test.yaml`. Containers are named
  `it-<service>` (`it-gateway-controller`, `it-gateway-runtime`, `it-sample-backend`,
  `it-echo-backend`, `it-mock-platform-api`, `it-mock-jwks`, …).
- Failure logs: on any scenario failure the suite dumps all container logs to
  `gateway/it/logs/suite_failure_<timestamp>.log`.

## Step 1: Build the gateway images (do this first, always)

The test stack runs from `*-coverage` images. Build them **before every test run**
so the stack reflects the current code — any change to gateway-controller,
gateway-runtime, policy-engine, or SDK code is otherwise invisible to the tests:

```bash
cd gateway && make build-coverage      # builds gateway-{controller,runtime}-coverage:<VERSION>
```

This is slow (~5-15 min). `make test` later retags `:<VERSION>` → `:test` via its
`ensure-test-tags` step. Confirm the images exist afterward:

```bash
docker images --format '{{.Repository}}:{{.Tag}} {{.CreatedSince}}' | grep -E 'gateway-(runtime|controller)-coverage'
```

If a non-gateway test dependency is stale (e.g. `mock-platform-api`), rebuild that
image too — see the Common failure signatures table.

## Step 2: Run the chosen feature files

Run only the features you need with the `IT_FEATURE_PATHS` env var
(comma-separated, paths relative to `gateway/it/`):

```bash
cd gateway/it
IT_FEATURE_PATHS=features/dynamic-endpoint.feature make test
```

Multiple files:

```bash
IT_FEATURE_PATHS=features/health.feature,features/dynamic-endpoint.feature make test
```

`make test` first runs `ensure-test-tags` (retags `*-coverage:<VERSION>` images to
`:test`) then `go test`. Total wall time is dominated by Docker Compose startup
(~60-90s) plus the scenarios.

### Cold-start caveat — IMPORTANT

When a feature runs **first** against a freshly-started gateway, the first requests
can return `503 UH no_healthy_upstream` for several seconds while Envoy clusters
warm and xDS syncs. The per-feature readiness wait (`I wait for the endpoint ... to
be ready`) is only ~9s, which can be too short at cold start.

To verify a single feature reliably, prepend warm-up features so the gateway is hot
by the time yours runs:

```bash
IT_FEATURE_PATHS=features/health.feature,features/api_deploy.feature,features/<target>.feature make test
```

This mirrors how the feature runs inside the full default suite (where it executes
after ~40 other features).

## Step 3: Troubleshoot a failure

When scenarios fail, find the **root cause** — do not just rerun. Work through:

### 1. Read the suite output
The summary table lists which scenario failed and the failing step. A failure at
`Given the gateway services are running` or at a readiness wait means the stack
itself is unhealthy — go to step 2. A failed assertion means the gateway did
something wrong — go to step 4.

### 2. Check container health and exit codes

```bash
docker ps -a --filter 'name=it-' --format '{{.Names}}\t{{.Status}}'
```

If a container `Exited (1)`, read its logs:

```bash
docker logs it-gateway-controller 2>&1 | tail -50
docker logs it-gateway-runtime    2>&1 | tail -50
```

The gateway-runtime log interleaves three sources — filter by prefix:
- `[rtr]` — Envoy router (access logs, routing errors)
- `[pol]` — policy engine (ext_proc, policy execution)
- unprefixed — the container entrypoint / process manager

### 3. Use the dumped failure log
The suite writes every container's logs to one file on failure:

```bash
ls -t gateway/it/logs/suite_failure_*.log | head -1
```

Grep it for the API/policy under test, then narrow to errors:

```bash
latest=$(ls -t gateway/it/logs/suite_failure_*.log | head -1)
grep -iE '<api-or-policy-name>' "$latest" | grep -iE 'error|warn|fail|panic|reject|503|no_healthy' | head -40
```

### 4. Reproduce manually for assertion failures
Bring up only the services you need, deploy the artifact by hand, and inspect.
The controller exposes two REST APIs (basic auth `admin:admin` — the plaintext credential hardcoded in
the IT fixture `gateway/it/test-config.toml`, distinct from the shipped `config.toml`, which provisions
its admin credential via `setup.sh`); the router is `http://localhost:8080`; Envoy admin is
`http://localhost:9901`.

- Management API — `http://localhost:9090/api/management/v0.9` — data-plane
  artifacts: REST APIs, LLM providers, LLM proxies, MCP servers, …
- Admin API — `http://localhost:9090/api/admin/v0.9` — admin resources: secrets,
  API keys, …

**Do not hard-code resource paths.** Resolve the correct path, method, and
payload shape from the OpenAPI specs in the repo before issuing the request —
artifact types and field names drift, and the spec is the source of truth:

- `gateway/gateway-controller/api/management-openapi.yaml`
- `gateway/gateway-controller/api/admin-openapi.yaml`

Read the spec, pick the operation matching the artifact under test (e.g.
`POST /rest-apis`, `POST /llm-providers`, `POST /llm-proxies`, `POST /mcp-servers`,
`POST /secrets`, `POST /api-keys`), then issue it:

```bash
cd gateway/it
docker compose -f docker-compose.test.yaml up -d gateway-controller gateway-runtime sample-backend mock-platform-api mock-openapi
# Generic shape — substitute <base> and <resource> per the OpenAPI operation:
curl -s -u admin:admin -X POST '<base>/<resource>' \
  -H 'Content-Type: application/yaml' --data-binary @artifact.yaml
curl -s http://localhost:8080/<context>/<path>     # invoke via the router
# inspect each component's config — see step 5 for the config_dump endpoints
```

Then trace the request: policy execution in `[pol]` logs, the upstream chosen in
the `[rtr]` access log (`upstream_cluster`, `response_code`, `response_code_details`).

### 5. Compare config dumps across components

Each component exposes a `config_dump` showing the configuration it actually holds.
Comparing them pinpoints **which** component a deployment went wrong in — config
flows controller → (xDS) → policy engine and Envoy, so the first dump that looks
wrong is the culprit.

```bash
# Gateway Controller — the deployment snapshot it built (APIs, routes, clusters,
# policy chains). Also /xds_sync_status to confirm xDS push state.
curl -s http://localhost:9092/api/admin/v0.9/config_dump | jq .
curl -s http://localhost:9092/api/admin/v0.9/xds_sync_status | jq .

# Policy Engine — what it received via xDS (routes, policy chains, route metadata
# such as upstream definition paths). Admin port 9002.
curl -s http://localhost:9002/config_dump | jq .

# Envoy router — live listeners, routes, and clusters. Admin port 9901.
curl -s http://localhost:9901/config_dump | jq .
curl -s http://localhost:9901/clusters            # endpoint health per cluster
```

Typical use: a request is routed/rewritten wrong → check the controller dump for
the expected route + cluster; if correct there, check the policy engine dump for
the route metadata and policy chain; if correct there, check Envoy's dump. The
component where the expected config is missing or wrong is where the bug lives.

### 6. Pause a live scenario with Delve (last resort)

`make test` is the normal path — reach for Delve only when steps 1-5 above have
not pinpointed the bug and you need to freeze the actual failing scenario.

Run the suite under the Delve debugger (`dlv`). The Compose stack is
daemon-managed — started in `BeforeSuite`, torn down in `AfterSuite` — so it runs
independently of the test process. **While the test is paused at a breakpoint
between those two points the whole stack stays live**, with the scenario's exact
deployed APIs present, and you can `curl` config dumps / invoke APIs from another
terminal. Halting also pauses Go's `-test.timeout` clock, so inspection time is
unlimited.

`dlv test` bypasses the Makefile, so build/tag the images first:

```bash
cd gateway/it
make ensure-test-tags        # or: cd ../ && make build-coverage
IT_FEATURE_PATHS=features/<target>.feature COMPOSE_FILE=docker-compose.test.yaml \
  dlv test . -- -test.run TestFeatures -test.v -test.timeout 0
```

Both `break <file>:<line>` and `break <function>` work. Function-name
breakpoints survive edits to the file, so they're the more durable choice when
you're not sure how many iterations you'll go through — use either as it suits.
Godog step definitions are ordinary Go functions, so you can break inside any
step, mid-scenario — the most useful mode, since you pause *before* things go
wrong with the scenario's APIs already deployed. Map a Gherkin phrase to its
handler by grepping the step registrations:

```bash
grep -rn 'ctx.Step(' gateway/it/      # each step regex → its handler function
```

Then at the `(dlv)` prompt break on that function by name (`funcs <regexp>`
discovers exact names), and `continue` — BeforeSuite starts the stack (~60-90s),
then execution stops at the breakpoint:

```
(dlv) funcs steps.*AssertSteps             # list candidate function names
(dlv) break steps.(*AssertSteps).<method>  # pause on an assertion, response in hand
(dlv) break steps.(*HTTPSteps).<method>    # pause on a request (the deploy step calls SendPOSTToService)
(dlv) continue
```

When stopped inside a step you can `next` / `step` through its code and `print`
locals (`print testState`) — full Go debugging. Meanwhile, from a **second
terminal**, the live stack is reachable: hit any component's `config_dump` as in
step 5, or invoke the scenario's deployed API on the router (`:8080`).

Break at step boundaries, not mid-HTTP-call. Caveat: if you `quit` Delve before
the suite finishes, `AfterSuite` never runs and containers leak — run
`make clean` afterward.

### 7. Switch to source-level gateway debug (when step 6 isn't enough)

Step 6 pauses the **test process**. When the bug is in the **gateway code
itself** (controller deploy handler, xDS push, policy execution) you need to
pause the gateway process — hand off to the [[gateway-debug]] skill, which
runs gateway-controller and policy-engine as host-side processes under `dlv`.
The failing scenario's deploy YAML and request become the manual reproducer.
Map the scenario to one of [[gateway-debug]]'s two paths:

- **Scenario invokes the API on `:8080` through Envoy** (most `.feature` files
  — anything with `When I send a … request to "http://…8080…"`): take
  gateway-debug's **Path B (request-runtime)** and pick the **IT handoff**
  compose target at Step 2 so the IT mocks (`mock-jwks`,
  `mock-platform-api`, …) the scenario depends on stay live.
- **Scenario only hits the controller REST API** (api-keys, secrets, REST
  API CRUD, MCP / LLM-resource CRUD — never touches `:8080`): take
  gateway-debug's **Path A (deployment-only)** — no docker, just Step 3a.

### Common failure signatures

| Symptom | Likely cause | Action |
|---|---|---|
| `it-gateway-controller` exits, log shows `manifest ... 404` | stale `mock-platform-api` image missing an endpoint | `docker compose -f docker-compose.test.yaml build mock-platform-api` |
| `503 UH no_healthy_upstream` only at start | cold-start cluster warmup | add warm-up features (see Cold-start caveat) |
| Assertion mismatch on routed path/host | policy or controller xDS bug | reproduce manually, compare `[pol]` metadata vs `[rtr]` access log |
| `No image found for ...` | coverage images missing | `cd gateway && make build-coverage` |

## Step 4: Clean up — always run when finished

After tests pass (or after troubleshooting), tear the stack down:

```bash
cd gateway/it && make clean
```

`make clean` force-removes `it-*` containers, runs `docker compose down -v
--remove-orphans` for every compose file, and deletes `coverage/` and `reports/`
artifacts. If you manually started a partial stack, `make clean` also covers it.