Write Sglang Test

Writing SGLang CI / UT Tests

This skill covers how to write and register tests. For CI pipeline internals (stage ordering, fast-fail, gating, partitioning, debugging CI failures), see the CI workflow guide.

Core Rules

1. Always use CustomTestCase — never raw unittest.TestCase. It ensures tearDownClass runs even when setUpClass fails, preventing resource leaks in CI.
2. tearDownClass must be defensive — use hasattr/null checks before accessing resources (e.g. cls.process) that setUpClass may not have finished allocating.
3. Place tests in test/registered/<kind>/<subsystem>/ — <kind> is unit, kernel, e2e, accuracy, perf, or stress; hardware belongs in registrations, not directory names
4. Reuse server fixtures — inherit from DefaultServerBase or write setUpClass/tearDownClass with popen_launch_server
5. Mock boundaries, not SGLang behavior — mock slow or external dependencies only when the assertion still checks an observable result, state transition, or error. A test whose evidence is only assert_called* mirrors its mock and is not admissible. Launch a real server only when inference results or lifecycle behavior are the contract under test.

JIT kernel notes:

• If the task is adding or updating code under python/sglang/kernels/jit/, prefer the add-jit-kernel skill first.
• New JIT kernel correctness tests use test/registered/kernel/jit/**/test_*.py.
• New JIT kernel benchmarks use test/registered/kernel/jit/benchmark/**/bench_*.py.
• Those files are executed by test/run_suite.py through dedicated kernel suites (base-b-kernel-*); a register_*_ci(...) call placed under python/sglang/ is rejected by the check-no-registered-tests-in-package pre-commit hook.

Model & Backend Selection

Key principle for E2E tests: Do NOT add register_amd_ci unless the test specifically exercises AMD/ROCm code paths. Common E2E tests just need any GPU to run — duplicating across backends wastes CI time with no extra coverage.

All model constants

Defined in python/sglang/test/test_utils.py:

Naming Conventions

A per-commit suite name is generated from registration metadata as {stage}-test-{runner_config} — you don't hand-write it:

• stage — the CI stage (e.g. base-b, base-b-kernel-unit, base-c).
• runner_config — a runner-pool key from scripts/ci/runner_configs.yml, which maps it to the physical runner label (so 1-gpu-large runs on 1-gpu-h100). AMD/NPU use their own keys (e.g. amd).
• Suite — register_cuda_ci(stage="base-b", runner_config="1-gpu-small") → base-b-test-1-gpu-small, the name you pass to run_suite.py --suite. The -test- is just the connector; never put it in register_*_ci.

CUDA nightly uses the same shape with stage="nightly" (e.g. stage="nightly", runner_config="1-gpu-large" → nightly-test-1-gpu-large) and no nightly=True — the stage name carries the cadence, and setting the flag makes the test silently never run. Legacy single-string suite= is left only for stress and some AMD/CPU/NPU pools.

All CI Suites

Per-commit (CUDA)

Per-commit (AMD)

Per-commit (Ascend NPU)

Nightly

Nightly suites are listed in NIGHTLY_SUITES in test/run_suite.py. They run via nightly-test-nvidia.yml, nightly-test-amd.yml, and nightly-test-npu.yml, not pr-test.yml.

CUDA nightly suites are named nightly-test-{runner_config} — one per machine type, holding everything that runs nightly on it. There is no per-purpose split (kernel / eval / perf / precision all share their machine's suite); auto_partition splits the work. Examples:

• nightly-test-1-gpu-large (CUDA)
• nightly-test-2-gpu-large (CUDA)
• nightly-test-8-gpu-h200 (CUDA)
• nightly-test-4-gpu-gb300 (CUDA)
• nightly-amd (AMD)
• nightly-amd-8-gpu-mi35x (AMD)
• nightly-1-npu-a3 (NPU)
• nightly-2-npu-a3 (NPU)
• nightly-4-npu-a3 (NPU)
• nightly-8-npu-a3 (NPU)
• nightly-16-npu-a3 (NPU)

Note: Multimodal diffusion uses python/sglang/multimodal_gen/test/run_suite.py, not test/run_suite.py.

Choosing a Suite

Use the lightest suite that meets your test's needs:

• No GPU required → base-a-test-cpu
• Most small GPU tests → base-b-test-1-gpu-small (default choice)
• Need H100 memory or Hopper features → base-b-test-1-gpu-large
• JIT kernel correctness → base-b-kernel-unit-test-1-gpu-large
• JIT kernel correctness for B200 / SM100 paths → base-b-kernel-unit-test-4-gpu-b200
• JIT kernel benchmarks → base-b-kernel-benchmark-test-1-gpu-large
• Multi-GPU → only when the test actually needs multiple GPUs

Test File Templates

Unit Tests (no server / engine launch)

See test/registered/unit/README.md for quick-start and rules. Unit tests live in test/registered/unit/, mirroring python/sglang/srt/:

"""Unit tests for srt/<module>"""

import unittest
from unittest.mock import MagicMock, patch

from sglang.srt.<module> import TargetClass
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase

register_cpu_ci(est_time=5, suite="base-a-test-cpu")
# Unit tests are CPU-only. GPU operator tests belong under `kernel/`.

class TestTargetClass(CustomTestCase):
    def test_basic_behavior(self):
        obj = TargetClass(...)
        self.assertEqual(obj.method(), expected)

    @patch("sglang.srt.<module>.some_dependency")
    def test_with_mock(self, mock_dep):
        mock_dep.return_value = MagicMock()
        # test logic with dependency mocked
        ...


if __name__ == "__main__":
    unittest.main()

Use unittest.mock.patch / MagicMock only at dependency boundaries. Assert the resulting value, state, protocol output, or error—not merely that the mock was called. If the module transitively imports GPU-only packages (e.g. sgl_kernel), they can be stubbed so the test runs on CPU CI. Do not modify sys.modules at module level—use patch.dict (as a class decorator or with start/stop) to ensure cleanup and avoid cross-test pollution. See test/registered/unit/README.md for details and examples.

Quality bar — test real logic (validation boundaries, state transitions, error paths, branching, etc.). Skip tests that just verify Python itself works (e.g., "does calling an abstract method raise NotImplementedError?", "does a dataclass store the field I assigned?"). Consolidate repetitive patterns into parameterized tests. No production code changes in test PRs.

E2E test (small model, server needed)

import unittest

import requests

from sglang.srt.utils import kill_process_tree
from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import (
    DEFAULT_SMALL_MODEL_NAME_FOR_TEST,
    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
    DEFAULT_URL_FOR_TEST,
    CustomTestCase,
    popen_launch_server,
)

register_cuda_ci(est_time=60, suite="base-b-test-1-gpu-small")


class TestMyFeature(CustomTestCase):
    @classmethod
    def setUpClass(cls):
        cls.model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
        cls.base_url = DEFAULT_URL_FOR_TEST
        cls.process = popen_launch_server(
            cls.model,
            cls.base_url,
            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
            other_args=["--arg1", "value1"],  # feature-specific args
        )

    @classmethod
    def tearDownClass(cls):
        if hasattr(cls, "process") and cls.process:
            kill_process_tree(cls.process.pid)

    def test_basic_functionality(self):
        response = requests.post(
            self.base_url + "/generate",
            json={"text": "Hello", "sampling_params": {"max_new_tokens": 32}},
        )
        self.assertEqual(response.status_code, 200)


if __name__ == "__main__":
    unittest.main(verbosity=3)

E2E test (8B model, server needed, performance)

import time
import unittest

import requests

from sglang.srt.utils import kill_process_tree
from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import (
    DEFAULT_MODEL_NAME_FOR_TEST,
    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
    DEFAULT_URL_FOR_TEST,
    CustomTestCase,
    popen_launch_server,
)

register_cuda_ci(est_time=300, suite="base-b-test-1-gpu-large")


class TestMyFeaturePerf(CustomTestCase):
    @classmethod
    def setUpClass(cls):
        cls.model = DEFAULT_MODEL_NAME_FOR_TEST
        cls.base_url = DEFAULT_URL_FOR_TEST
        cls.process = popen_launch_server(
            cls.model,
            cls.base_url,
            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
        )

    @classmethod
    def tearDownClass(cls):
        if hasattr(cls, "process") and cls.process:
            kill_process_tree(cls.process.pid)

    def test_latency(self):
        start = time.perf_counter()
        response = requests.post(
            self.base_url + "/generate",
            json={"text": "Hello", "sampling_params": {"max_new_tokens": 128}},
        )
        elapsed = time.perf_counter() - start
        self.assertEqual(response.status_code, 200)
        self.assertLess(elapsed, 5.0, "Latency exceeded threshold")


if __name__ == "__main__":
    unittest.main(verbosity=3)

Server Fixture Reuse

For tests that only need a standard server, inherit from DefaultServerBase and override class attributes:

from sglang.test.server_fixtures.default_fixture import DefaultServerBase

class TestMyFeature(DefaultServerBase):
    model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
    other_args = ["--enable-my-feature"]

    def test_something(self):
        ...

Available fixtures in python/sglang/test/server_fixtures/:

CI Registration

Every CI-discovered test file must call a registration function at module level:

from sglang.test.ci.ci_register import (
    register_cuda_ci,
    register_amd_ci,
    register_cpu_ci,
    register_npu_ci,
)

# Per-commit test (small 1-gpu, runs on 5090)
register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small")

# Per-commit test (large 1-gpu, runs on H100)
register_cuda_ci(est_time=120, suite="base-b-test-1-gpu-large")

# Nightly-only test (same shape as per-commit, stage is just "nightly")
register_cuda_ci(est_time=200, stage="nightly", runner_config="1-gpu-large")

# Multi-backend test (only when testing backend-specific code paths)
register_cuda_ci(est_time=80, suite="base-a-test-1-gpu-small")
register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd")
register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True)

# Temporarily disabled test
register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small", disabled="flaky - see #12345")

Parameters:

• est_time: estimated runtime in seconds (used for CI partitioning)
• suite: which CI suite to run in (see suite tables above)
• nightly=True: legacy cadence flag, for non-CUDA nightly suites only. CUDA nightly uses stage="nightly" and must leave this unset
• disabled="reason": temporarily disable with explanation

Key principle: Only add register_amd_ci / register_npu_ci when the test exercises backend-specific code paths. Common E2E tests just need register_cuda_ci — duplicating across backends wastes CI time.

JIT Kernel Registration

JIT kernel files live outside test/registered/ but still use registration:

from sglang.test.ci.ci_register import register_cuda_ci

# Correctness tests in test/registered/jit/
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
register_cuda_ci(est_time=120, stage="base-b-kernel-unit", runner_config="8-gpu-h200")

# Benchmarks in test/registered/jit/benchmark/
register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")

# Optional nightly registration — same form, stage is just "nightly"
register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
register_cuda_ci(est_time=120, stage="nightly", runner_config="8-gpu-h200")

Every call generates a suite named {stage}-test-{runner_config}, e.g. base-b-kernel-unit-test-1-gpu-large and nightly-test-1-gpu-large. Keep est_time, stage, runner_config, and suite as literal values — run_suite.py collects them by AST parsing.

Test Placement

test/
├── registered/          # CI tests (auto-discovered by run_suite.py)
│   ├── unit/<subsystem>/      # CPU-only; no server or model weights
│   ├── kernel/<group>/        # accelerator operator correctness/benchmarks
│   ├── e2e/<subsystem>/       # engine/server integration
│   ├── accuracy/<family>/     # scheduled eval floors
│   ├── perf/<family>/         # scheduled latency/throughput contracts
│   └── stress/<subsystem>/    # stress/weekly coverage
├── manual/              # Non-CI: debugging, one-off, manual verification
└── run_suite.py         # CI runner (scans registered/ plus jit_kernel test/benchmark files)

python/sglang/kernels/jit/
├── tests/               # JIT kernel correctness tests (CI-discovered by test/run_suite.py)
└── benchmark/           # JIT kernel benchmarks (CI-discovered by test/run_suite.py)

Decision rule (see also test/registered/README.md):

• CPU component logic, no server → registered/unit/<subsystem>/
• JIT kernel correctness / benchmarks → registered/kernel/jit/
• Other accelerator operator correctness → registered/kernel/<group>/
• Server needed → registered/e2e/<subsystem>/
• Eval floor / performance contract → registered/{accuracy,perf}/<family>/
• Local debugging → manual/

Eval Accuracy Mixins

Design philosophy: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates what to test (threshold) from how to test (run_eval, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the test_* method. Override when you need extra assertions (e.g. EAGLE accept length).

Available mixins in python/sglang/test/kits/eval_accuracy_kit.py: MMLUMixin, HumanEvalMixin, MGSMEnMixin, GSM8KMixin. Can be combined freely. Read the source for attrs and defaults.

class TestMyFeature(CustomTestCase, MMLUMixin):
    mmlu_score_threshold = 0.65
    mmlu_num_examples = 64
    mmlu_num_threads = 32
    # test_mmlu is inherited — no code needed

Key Utilities

from sglang.test.test_utils import (
    CustomTestCase,              # base class with retry logic
    popen_launch_server,         # launch server subprocess
    DEFAULT_URL_FOR_TEST,        # auto-configured base URL
    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,  # 600s default
    run_bench_serving,           # benchmark helper (launch + bench)
)
from sglang.srt.utils import kill_process_tree  # cleanup server

Checklist

Before submitting a test:

• Inherits from CustomTestCase (not unittest.TestCase)
• Has register_*_ci(...) call at module level
• Placed in test/registered/<kind>/<subsystem>/
• JIT kernel work: test files live in test/registered/kernel/jit/; only test-only helpers stay under python/sglang/kernels/jit/
• Backend-independent tests: register_cuda_ci only + smallest model
• Logic that doesn't need a server / engine launch → unit test in registered/unit/ (see Unit Tests section)
• setUpClass launches server, tearDownClass kills it (if server-based)
• tearDownClass is defensive — uses hasattr/null checks before accessing resources that may not have been allocated
• Has if __name__ == "__main__": unittest.main()
• est_time is reasonable (measure locally)