Retail Test Runner

Retail Test Runner

Use this skill when the user asks to run, validate, debug, or explain the Retail Shopping Assistant tests under this repository's tests/ directory.

Default Runner

Use the deterministic runner from the repo root:

python skills/retail-test-runner/scripts/run_retail_tests.py all --test-path shopping

Common invocations:

python skills/retail-test-runner/scripts/run_retail_tests.py unit
python skills/retail-test-runner/scripts/run_retail_tests.py integration --test-path shopping
python skills/retail-test-runner/scripts/run_retail_tests.py integration --test-path rails --skip-quality

The runner:

• Loads repo-root .env before checking integration credentials.
• Uses .local-run/dev-venv/bin/python when it exists, then .venv-tests/bin/python, then the current Python.
• Runs unit tests from tests/ so tests/pytest.ini is applied.
• Runs integration scripts from tests/integration/ so their relative conversations/<TEST_PATH> paths resolve correctly.
• Targets the chain-server timing endpoint at http://localhost:8009/query/timing by default.
• Sets TEST_PATH for integration runs.

Unit Tests

The unit suite lives under tests/unit and is offline. It should not require Docker, Milvus, NIMs, guardrails services, or network calls.

If Python dev dependencies are missing, install them into the repo-local dev venv:

python skills/retail-local-runner/scripts/local_runner.py install-dev

Targeted unit examples:

cd tests
../.local-run/dev-venv/bin/python -m pytest -q unit/chain_server
../.local-run/dev-venv/bin/python -m pytest -q unit/chain_server/test_cart.py
../.local-run/dev-venv/bin/python -m pytest -q -k test_add_to_cart

Integration Tests

The integration suite lives under tests/integration and drives live HTTP endpoints. Before running it:

• Ensure the chain server is running and reachable at http://localhost:8009.
• Choose an existing scenario directory under tests/integration/conversations/, usually shopping or rails.
• Put NVIDIA_API_KEY in the repo-root .env or export it in the launching shell, unless using --skip-quality; response_quality.py requires it for LLM-as-judge scoring.

The runner preflights the selected conversation directory and service URL. Use --no-preflight only when intentionally testing a nonstandard setup.

Integration outputs are written under:

• tests/integration/conversations/<TEST_PATH>/results/
• tests/integration/conversations/<TEST_PATH>/judge/

Do not delete or overwrite these outputs unless the user asks for a clean run.

Validation

After editing this skill, validate both skill folders:

uv run --with pyyaml python "$CODEX_HOME/skills/.system/skill-creator/scripts/quick_validate.py" skills/retail-test-runner
uv run --with pyyaml python "$CODEX_HOME/skills/.system/skill-creator/scripts/quick_validate.py" .agents/skills/retail-test-runner