Datadog Python Test Optimization Onboarding

Datadog Python Test Optimization onboarding

Use this skill when you need to instrument a Bazel Python repository with Datadog Test Optimization. The skill is intentionally project-neutral: it is stored in this repository as a Codex-compatible skill, but any agent can read it as a normal implementation guide.

Non-negotiable contract

Keep the RFC contract intact:

• Tests write JSON payloads to TEST_UNDECLARED_OUTPUTS_DIR.
• Bazel collects those files under bazel-testlogs/<target>/test.outputs/.
• The doctor validates local files after bazel test.
• The uploader runs after the doctor with bazel run.
• Use the default Python 3.10+ uploader unless a temporary rollback explicitly requires use_python_uploader = False. Its coordinator prepares shared CODEOWNERS, contexts, schemas, freshness, and telemetry once, then starts up to eight independent file workers by default. Each worker owns enrichment, validation, preventive splitting, retries, and cleanup for one test, coverage, or telemetry source file.
• Run one uploader process. Use --dry-run --validate-enrichment to prepare requests without HTTP or deletion, --debug only for verbose redacted diagnostics, and review the final file/type/split/request/cleanup totals.
• Test bodies are split before HTTP when they exceed 4_718_592 bytes. HTTP 413 is terminal and must not trigger a retry or adaptive split; coverage and telemetry are not split.
• Do not add payload proxies or upload-from-test-sandbox paths.
• Do not pass DD_GIT_* through --test_env; use --repo_env for sync metadata.
• Do not pass uploader credentials or upload endpoints into the test sandbox.
• Put --remote_download_minimal, --remote_download_regex=.*test[.]outputs.*, and --zip_undeclared_test_outputs in the active test .bazelrc config when remote execution or remote cache can leave test outputs remote-only.
• Configure doctor/uploader with repeatable --bep-json=<path> flags, --freshness-source=bep, --freshness-mode=required, --artifact-source=bep, and --artifact-staging-dir=<temp-dir>. If BEP still points at HTTP/HTTPS outputs.zip artifacts, use --remote-artifacts=download or required without a downloader. Use a downloader only for bytestream/CAS/custom-auth artifact providers.
• Run pilot tests with a fresh --build_event_json_file path per Bazel test invocation; pass the same paths to doctor/uploader with --bep-json.
• In CI, keep a per-job diagnostic report directory with DD_TEST_OPTIMIZATION_REPORT_DIR or wrapper --report-dir, and configure wrapper --support-bundle or DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE for complete escalation artifacts. For first-pass customer troubleshooting after tests have run, ask for bazel run --config=test-optimization //<topt-package>:dd_test_optimization_doctor -- --support-bundle=<path> with any matching BEP/artifact flags. Replace <topt-package> with the package that owns the logical doctor/uploader pair; use //: only when a small repository intentionally keeps the targets at the root. For bundle triage, inspect summary.md, diagnostics.json, reports/doctor-report.json, optional uploader reports, and command/flags.json in that order.

First actions

1. Read the consumer repository's Bazel shape before editing:
   • Does it use MODULE.bazel, WORKSPACE, or both?
   • What command does the repository use for Bazel: bazel, bazelw, bzl, or a repo-local wrapper?
   • What is the Bazel repository name for rules_python?
   • What Python version and toolchain does Bazel use?
   • Which repository owns Python dependencies and lockfiles?
   • Does the repository already have a pytest wrapper macro?
   • Which lightweight package should own the logical doctor/uploader pair (for example //tools/test_optimization)?
   • Does fetching this rules repository require SSH git or authenticated archive access?
   • Which runtime test targets should emit payloads?
   • Which build-only or analysis-only targets should not be expected to emit payloads?
   • Is FETCH_SALT absent from the normal test, doctor, and uploader flow?
2. Read this repository's current docs when details are needed:
   • README.md for quickstart and current command flow.
   • docs/Language_Onboarding.md for language-specific Python guidance.
   • docs/Installation_Reference.md for helper APIs and pinning.
   • docs/Uploader_Reference.md for doctor, dry-run, and upload behavior.
   • docs/Troubleshooting.md for failure diagnosis.
3. Pick the correct path:
   • Bzlmod repo: follow bzlmod-onboarding.md.
   • WORKSPACE repo: follow workspace-onboarding.md.
   • Existing pytest wrapper: also follow consumer-runner.md.
   • Consumer-owned managed monorepo command: use manifest sync only when the command expands exact Go/Python labels and derives runtime contexts. Do not add a checked-in target/service map.
   • Validation and debugging: follow validation-checklist.md and troubleshooting.md.

Universal shape

Every successful Python onboarding should end with these pieces:

• Repository or module resolution fetches Test Optimization metadata.
• The consumer repository owns rules_python, Python toolchains, pip_parse, pytest, ddtrace, and lockfiles.
• Python tests use dd_topt_py_test directly or through a repo-local wrapper.
• Managed pytest mode is used when the repository does not already own a pytest runner.
• consumer_runner mode is used when the repository must keep an existing pytest wrapper, custom launcher, or import policy.
• The workspace has exactly one logical doctor/uploader pair. In monorepos, place it in a lightweight package such as //tools/test_optimization; root labels are still fine for small repositories.
• .bazelrc or CLI commands provide sync metadata with --repo_env.
• Test commands use a named config such as --config=test-optimization.
• Validation first runs the ordinary public Python test without that config, then reruns it with the config on the same fresh Bazel output root. Disabled mode must keep the consumer runner intact while omitting metadata requests, selectors, Bazel metadata, and payload generation.
• Remote-output-sensitive test configs include --remote_download_minimal --remote_download_regex=.*test[.]outputs.* and --zip_undeclared_test_outputs.
• Validation commands pass each matching BEP file with repeatable --bep-json flags and required BEP freshness/artifact flags. Use DD_TEST_OPTIMIZATION_* environment variables only for single-invocation manual flows where one BEP file is sufficient.
• CI wrappers write doctor-report.json, one selected uploader report (uploader-dry-run-report.json or uploader-upload-report.json), and dd-test-optimization-support.zip under a per-job report directory. Prefer the wrapper support bundle for full CI escalation; use the doctor-only support bundle for the simplest initial customer request. Keep individual reports for local inspection and manual fallback flows.
• FETCH_SALT is used only for a separate, explicit bazel sync --config=test-optimization --only=<repo> --repo_env=FETCH_SALT="$(date +%s)" refresh, never as part of normal test, doctor, or uploader commands.
• A real upload processes every available fresh valid payload after validation attempts. The wrapper preserves the earliest test, doctor, or uploader exit code; uploader errors never replace an earlier test result.

For automatic managed Go/Python monorepos:

• declare one manifest aggregate repository, separate from static multi-sync;
• load topt_data_by_target in the central Python wrapper;
• preserve the consumer's comparison-base Python path when the current full label is absent;
• preserve consumer_runner behavior and existing pytest/JUnit policy for selected targets;
• wire doctor to aggregate contexts and generated exact targets;
• keep the invocation manifest private to the consumer command;
• do not describe Java or other runtimes as automatically enrolled.

Use the consumer's existing Bazel entrypoint in all commands. Do not switch a repository from bzl or bazelw to raw bazel just because examples use the generic binary name.

Branch and PR hygiene

Before making changes in a real repository, confirm whether to use the current branch or create a new branch from the latest default branch. Keep onboarding changes reviewable:

• Put reusable rule changes in rules_test_optimization, not in a consumer repository workaround.
• Put consumer-specific scheduling, Docker, tag, flaky, and wrapper policy in the consumer repository.
• Keep automatic target expansion and service naming in the consumer's managed command, not in the Rule, BUILD files, or Gazelle.
• If an issue requires changing this rule repository, add matching fixture coverage in rules_test_optimization_tests before declaring it solved.

Stop conditions

Stop and escalate instead of guessing when:

• The repository requires a new public rule behavior not covered by current docs.
• A target produces no JSON payloads after the pytest process ran.
• The doctor reports missing Git metadata after sync metadata was configured.
• The doctor reports missing Bazel metadata.
• The only available fix would put DD_GIT_*, credentials, or upload endpoints into the test sandbox.
• The only tried doctor/uploader placement is the root package in a large monorepo and no lightweight package placement has been attempted.
• A private repository fetch returns 404 and SSH/authenticated archive mode has not been confirmed.
• Validation requires secrets that are not already available in the environment.