# Testing

> Testing conventions for NeMo-RL. Covers Ray actor coverage pragmas, nightly test requirements, and recipe naming rules.

- Skill: `nvidia-nemo-rl/testing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nvidia-nemo-rl/testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-nemo-rl/testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: NVIDIA-NeMo (https://skillmd.com/u/nvidia-nemo-rl)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/nvidia-nemo-rl/testing

---


# Testing Conventions

## Coverage and Ray Actors

For any source file under `nemo_rl/*.py` that defines a class or function decorated with `@ray.remote`, add a coverage pragma because these run in separate Ray processes and are not reliably tracked by coverage.

Place `# pragma: no cover` on the `class` or `def` line:

```python
import ray

@ray.remote  # pragma: no cover
class RolloutActor:
    def run(self) -> None:
        ...

@ray.remote  # pragma: no cover
def remote_eval(batch):
    ...
```

## Nightly Tests for New Model Support

When adding support for a new model, add a corresponding nightly test consisting of:

### 1. Recipe YAML under `examples/configs/recipes/`

Place in the appropriate domain subdirectory (`examples/configs/recipes/llm/` or `examples/configs/recipes/vlm/`). Name it following the recipe naming rules below.

### 2. Driver script under `tests/test_suites/`

Create a shell script in the matching domain (`tests/test_suites/llm/` or `tests/test_suites/vlm/`). Source any common environment (e.g., `common.env`) and invoke the training entrypoint with `uv run ... --config <path-to-yaml>`. Match the driver script filename to the YAML base name with `.sh`.

### 3. Add to nightly list

Append the driver script path (relative to `tests/test_suites/`) to @tests/test_suites/nightly.txt.

## Recipe Naming Rules

### LLM Pattern

```
<algo>-<model>-<nodes>n<gpus>g-<strategy-and-params>[-modifiers][-long][.vN].(yaml|sh)
```

- **algo**: `sft`, `dpo`, `grpo`, etc.
- **model**: `llama3.1-8b-instruct`, `qwen2.5-7b-instruct`, etc.
- **nodes/gpus**: `1n8g`, `4n8g`, `8n8g`
- **strategy-and-params**: `fsdp2tp1`, `tp4pp2`, `megatron`, `dtensor2tp1`
- **modifiers** (optional): `sp`, `actckpt`, `fp8`, `noncolocated`, `quick`
- **-long** (optional): long-running recipe
- **.vN** (optional): version suffix for convergence-impacting changes

Examples:
```
sft-llama3.1-8b-1n8g-fsdp2tp1-long.yaml
grpo-llama3.1-8b-instruct-1n8g-megatron-fp8.yaml
grpo-qwen2.5-7b-instruct-4n8g-fsdp2tp4sp.v3.yaml
```

### VLM Pattern

```
vlm_<algo>-<model>-<nodes>n<gpus>g-<strategy>[-modifiers][.vN].(yaml|sh)
```

### Directory Placement

```
examples/configs/recipes/
  llm/<name>.yaml
  vlm/<name>.yaml

tests/test_suites/
  llm/<name>.sh
  vlm/<name>.sh
  nightly.txt
```