Plugin Test

Run Plugin Tests Locally

This skill runs the same tests that CI runs, but locally. Ask the user what they want to test, or infer from recent changes.

Options

The user can specify:

• Scope: all, unit, a plugin name (e.g., flask, redis), a category (data, http, web), or e2e
• Python version: e.g., 3.11 (default: 3.11-slim for Docker image)
• Rebuild image: whether to rebuild the Docker image (default: only if not already built)

If the user just says /plugin-test with no args, ask what to test.

Prerequisites Check

Before running tests, verify:

# Check Docker is running
docker info > /dev/null 2>&1 || echo "ERROR: Docker is not running"

# Check Poetry is available
poetry --version || echo "ERROR: Poetry not found, run 'make poetry'"

# Check if environment is set up
poetry env info > /dev/null 2>&1 || echo "WARNING: Poetry env not set up, will run 'make env'"

If prerequisites fail, help the user fix them before proceeding.

Step 1: Setup Environment (if needed)

make env

This installs Poetry and all dependencies including dev and plugin groups.

Step 2: Build Plugin Test Docker Image

The plugin test image must be built before running any plugin integration tests (not needed for unit tests).

# Default Python version
docker build --build-arg BASE_PYTHON_IMAGE=3.11-slim \
  -t apache/skywalking-python-agent:latest-plugin --no-cache \
  . -f tests/plugin/Dockerfile.plugin

To test with a specific Python version:

docker build --build-arg BASE_PYTHON_IMAGE=3.12-slim \
  -t apache/skywalking-python-agent:latest-plugin --no-cache \
  . -f tests/plugin/Dockerfile.plugin

Important: The image tag is always apache/skywalking-python-agent:latest-plugin — this is what docker-compose files reference. Rebuilding with a different Python version replaces it.

Testing Multiple Python Versions

The Python version inside the Docker container (not the host) determines what Python the plugin runs under. To test multiple versions, rebuild the image for each:

# Test with Python 3.11
docker build --build-arg BASE_PYTHON_IMAGE=3.11-slim \
  -t apache/skywalking-python-agent:latest-plugin --no-cache . \
  -f tests/plugin/Dockerfile.plugin
poetry run pytest -v tests/plugin/web/sw_flask/

# Then test with Python 3.13
docker build --build-arg BASE_PYTHON_IMAGE=3.13-slim \
  -t apache/skywalking-python-agent:latest-plugin --no-cache . \
  -f tests/plugin/Dockerfile.plugin
poetry run pytest -v tests/plugin/web/sw_flask/

CI does this as a matrix (builds images for each Python version in parallel, then runs all tests against each). Locally, you rebuild and re-run sequentially.

Available Python base images: 3.9-slim, 3.10-slim, 3.11-slim, 3.12-slim, 3.13-slim, 3.14-slim

Check if image already exists:

docker images apache/skywalking-python-agent:latest-plugin --format "{{.ID}} {{.CreatedAt}}"

If it exists and is recent, ask the user if they want to rebuild or which Python version to use (rebuilding takes ~1-2 minutes).

Step 3: Run Tests

Unit Tests (no Docker needed)

poetry run pytest -v tests/unit/

Single Plugin Test

Map the plugin name to its test path:

• Web frameworks: tests/plugin/web/sw_<name>/
• HTTP clients: tests/plugin/http/sw_<name>/
• Data stores: tests/plugin/data/sw_<name>/

poetry run pytest -v tests/plugin/web/sw_flask/
poetry run pytest -v tests/plugin/data/sw_redis/
poetry run pytest -v tests/plugin/http/sw_requests/

Plugin Category

poetry run pytest -v tests/plugin/data/    # All data plugins
poetry run pytest -v tests/plugin/http/    # All HTTP plugins
poetry run pytest -v tests/plugin/web/     # All web plugins

All Tests

poetry run pytest -v tests/unit/ tests/plugin/data/ tests/plugin/http/ tests/plugin/web/

E2E Tests

E2E tests need a separate Docker image and use SkyWalking infra-e2e:

# Build E2E image
docker build --build-arg BASE_PYTHON_IMAGE=3.11-slim \
  -t apache/skywalking-python-agent:latest-e2e --no-cache \
  . -f tests/e2e/base/Dockerfile.e2e

# E2E tests use infra-e2e framework, not pytest directly
# They are defined in tests/e2e/case/*/e2e.yaml

Note: E2E tests require the e2e CLI tool from SkyWalking infra-e2e. They are typically only run in CI. Inform the user if they ask for E2E.

Version Format in support_matrix

Use .* wildcard to always test the latest patch of each minor version:

support_matrix = {
    'falcon': {
        '>=3.13': ['4.*'],         # latest falcon 4.x
        '>=3.10': ['3.1.*', '4.*'],
    }
}

• '4.*' → pip installs falcon==4.* → latest 4.x (e.g., 4.2.0 today, 4.3.0 when released)
• '4.2.*' → pip installs falcon==4.2.* → latest 4.2.x patch
• '4.2' → pip installs falcon==4.2 → always 4.2.0 (misses patches)

Convention: use major.* (e.g., '4.*') when the plugin supports the whole major version, or minor.* (e.g., '3.11.*') when only specific minors are tested. This keeps CI testing fresh and the Plugins.md doc meaningful.

Step 4: Interpret Results

Success

All tests pass with green output. Report the summary.

Failure - Docker Compose Timeout

Wait time exceeded 150 secs

This means services didn't start. Check:

1. Is Docker running with enough resources?
2. Is port 9090/9091/12800/19876 already in use? (lsof -i :9090)
3. Check Docker logs: docker compose -f tests/plugin/<category>/sw_<name>/docker-compose.yml logs

Failure - Validation Mismatch

The test prints a diff between expected and actual span data. Common causes:

• Wrong componentId in expected.data.yml
• Missing or extra spans
• Tag values don't match
• Operation name mismatch

Failure - Import/Install Error

Plugin library failed to install in Docker. Check:

• Library version compatibility with Python version
• requirements.txt content in the test directory
• Docker compose command for the service

Failure - Permanent 502 / Empty Segments

IMPORTANT: Check your HTTP proxy first! If http_proxy or https_proxy environment variables are set, ALL test HTTP requests (prepare, validate) go through the proxy instead of reaching Docker containers directly. This causes 502 responses and empty segment data.

# Check for proxy
echo $http_proxy $https_proxy

# Run tests without proxy
export http_proxy="" https_proxy="" no_proxy="*" NO_PROXY="*"
poetry run pytest -v tests/plugin/web/sw_flask/

If still failing after clearing proxy:

• Check container logs: docker logs sw_<name>-consumer-1
• Look for whether the service actually started
• Debug in foreground: docker compose -f <path>/docker-compose.yml up
• Check ports: lsof -i :9090 -i :9091 -i :12800

Plugin Name to Test Path Mapping

Useful Debug Commands

# Check what containers are running during a test
docker ps

# Check container logs for a stuck test
docker compose -f tests/plugin/<category>/sw_<name>/docker-compose.yml logs

# Clean up orphaned containers from failed tests
docker compose -f tests/plugin/<category>/sw_<name>/docker-compose.yml down --remove-orphans

# Check what the mock collector received (while containers are running)
curl http://localhost:12800/receiveData

# Validate expected data manually
curl -X POST http://localhost:12800/dataValidate -d @tests/plugin/<category>/sw_<name>/expected.data.yml

# Check if ports are in use
lsof -i :9090 -i :9091 -i :12800 -i :19876

# Run with verbose output
poetry run pytest -v -s tests/plugin/web/sw_flask/

Source: apache/skywalking-python — distributed by TomeVault.