datadog-python-test-optimization-onboarding

<!-- Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2.0 License. This product includes software developed at Datadog (https://www.datadoghq.com/) Copyright 2025-Present Datadog, Inc. --> # Datadog Python Test Optimization onboarding Use this skill when you need to instrument a Bazel Python repository with Datadog Test Optimization. The skill is intentionally project-neutral: it is stored in this repository as a Codex-compatible skill, but any agent can read it as a normal implementation guide. ## Non-negotiable contract Keep the RFC contract intact: - Tests write JSON payloads to `TEST_UNDECLARED_OUTPUTS_DIR`. - Bazel collects those files under `bazel-testlogs/<target>/test.outputs/`. - The doctor validates local files after `bazel test`. - The uploader runs after the doctor with `bazel run`. - Use the default Python 3.10+ uploader unless a temporary rollback explicitly requires `use_python_uploader = False`. Its coordinator prepares shared CODEOWNERS, contexts, schemas, freshness, and telemetry once, then starts up to eight independent file workers by default. Each worker owns enrichment, validation, preventive splitting, retries, and cleanup for one test, coverage, or telemetry source file. - Run one uploader process. Use `--dry-run --validate-enrichment` to prepare requests without HTTP or deletion, `--debug` only for verbose redacted diagnostics, and review the final file/type/split/request/cleanup totals. - Test bodies are split before HTTP when they exceed `4_718_592` bytes. HTTP `413` is terminal and must not trigger a retry or adaptive split; coverage and telemetry are not split. - Do not add payload proxies or upload-from-test-sandbox paths. - Do not pass `DD_GIT_*` through `--test_env`; use `--repo_env` for sync metadata. - Do not pass uploader credentials or upload endpoints into the test sandbox. - Put `--remote_download_minimal`, `--remote_download_regex=.*test[.]outputs.*`, and `--zip_undeclared_test_outputs` in the active test `.bazelrc` config when remote execution or remote cache can leave test outputs remote-only. - Configure doctor/uploader with repeatable `--bep-json=<path>` flags, `--freshness-source=bep`, `--freshness-mode=required`, `--artifact-source=bep`, and `--artifact-staging-dir=<temp-dir>`. If BEP still points at HTTP/HTTPS `outputs.zip` artifacts, use `--remote-artifacts=download` or `required` without a downloader. Use a downloader only for bytestream/CAS/custom-auth artifact providers. - Run pilot tests with a fresh `--build_event_json_file` path per Bazel test invocation; pass the same paths to doctor/uploader with `--bep-json`. - In CI, keep a per-job diagnostic report directory with `DD_TEST_OPTIMIZATION_REPORT_DIR` or wrapper `--report-dir`, and configure wrapper `--support-bundle` or `DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE` for complete escalation artifacts. For first-pass customer troubleshooting after tests have run, ask for `bazel run --config=test-optimization //<topt-package>:dd_test_optimization_doctor -- --support-bundle=<path>` with any matching BEP/artifact flags. Replace `<topt-package>` with the package that owns the logical doctor/uploader pair; use `//:` only when a small repository intentionally keeps the targets at the root. For bundle triage, inspect `summary.md`, `diagnostics.json`, `reports/doctor-report.json`, optional uploader reports, and `command/flags.json` in that order. ## First actions 1. Read the consumer repository's Bazel shape before editing: - Does it use `MODULE.bazel`, `WORKSPACE`, or both? - What command does the repository use for Bazel: `bazel`, `bazelw`, `bzl`, or a repo-local wrapper? - What is the Bazel repository name for `rules_python`? - What Python version and toolchain does Bazel use? - Which repository owns Python dependencies and lockfiles? - Does the repository already have a pytest wrapper macro? - Which lightweight package should own the logical doctor/uploader pair (for example `//tools/test_optimization`)? - Does fetching this rules repository require SSH git or authenticated archive access? - Which runtime test targets should emit payloads? - Which build-only or analysis-only targets should not be expected to emit payloads? - Is `FETCH_SALT` absent from the normal test, doctor, and uploader flow? 2. Read this repository's current docs when details are needed: - `README.md` for quickstart and current command flow. - `docs/Language_Onboarding.md` for language-specific Python guidance. - `docs/Installation_Reference.md` for helper APIs and pinning. - `docs/Uploader_Reference.md` for doctor, dry-run, and upload behavior. - `docs/Troubleshooting.md` for failure diagnosis. 3. Pick the correct path: - Bzlmod repo: follow [bzlmod-onboarding.md](references/bzlmod-onboarding.md). - WORKSPACE repo: follow [workspace-onboarding.md](references/workspace-onboarding.md). - Existing pytest wrapper: also follow [consumer-runner.md](references/consumer-runner.md). - Consumer-owned managed monorepo command: use manifest sync only when the command expands exact Go/Python labels and derives runtime contexts. Do not add a checked-in target/service map. - Validation and debugging: follow [validation-checklist.md](references/validation-checklist.md) and [troubleshooting.md](references/troubleshooting.md). ## Universal shape Every successful Python onboarding should end with these pieces: - Repository or module resolution fetches Test Optimization metadata. - The consumer repository owns `rules_python`, Python toolchains, `pip_parse`, `pytest`, `ddtrace`, and lockfiles. - Python tests use `dd_topt_py_test` directly or through a repo-local wrapper. - Managed pytest mode is used when the repository does not already own a pytest runner. - `consumer_runner` mode is used when the repository must keep an existing pytest wrapper, custom launcher, or import policy. - The workspace has exactly one logical doctor/uploader pair. In monorepos, place it in a lightweight package such as `//tools/test_optimization`; root labels are still fine for small repositories. - `.bazelrc` or CLI commands provide sync metadata with `--repo_env`. - Test commands use a named config such as `--config=test-optimization`. - Validation first runs the ordinary public Python test without that config, then reruns it with the config on the same fresh Bazel output root. Disabled mode must keep the consumer runner intact while omitting metadata requests, selectors, Bazel metadata, and payload generation. - Remote-output-sensitive test configs include `--remote_download_minimal --remote_download_regex=.*test[.]outputs.*` and `--zip_undeclared_test_outputs`. - Validation commands pass each matching BEP file with repeatable `--bep-json` flags and required BEP freshness/artifact flags. Use `DD_TEST_OPTIMIZATION_*` environment variables only for single-invocation manual flows where one BEP file is sufficient. - CI wrappers write `doctor-report.json`, one selected uploader report (`uploader-dry-run-report.json` or `uploader-upload-report.json`), and `dd-test-optimization-support.zip` under a per-job report directory. Prefer the wrapper support bundle for full CI escalation; use the doctor-only support bundle for the simplest initial customer request. Keep individual reports for local inspection and manual fallback flows. - `FETCH_SALT` is used only for a separate, explicit `bazel sync --config=test-optimization --only=<repo> --repo_env=FETCH_SALT="$(date +%s)"` refresh, never as part of normal test, doctor, or uploader commands. - A real upload processes every available fresh valid payload after validation attempts. The wrapper preserves the earliest test, doctor, or uploader exit code; uploader errors never replace an earlier test result. For automatic managed Go/Python monorepos: - declare one manifest aggregate repository, separate from static multi-sync; - load `topt_data_by_target` in the central Python wrapper; - preserve the consumer's comparison-base Python path when the current full label is absent; - preserve `consumer_runner` behavior and existing pytest/JUnit policy for selected targets; - wire doctor to aggregate contexts and generated exact targets; - keep the invocation manifest private to the consumer command; - do not describe Java or other runtimes as automatically enrolled. Use the consumer's existing Bazel entrypoint in all commands. Do not switch a repository from `bzl` or `bazelw` to raw `bazel` just because examples use the generic binary name. ## Branch and PR hygiene Before making changes in a real repository, confirm whether to use the current branch or create a new branch from the latest default branch. Keep onboarding changes reviewable: - Put reusable rule changes in `rules_test_optimization`, not in a consumer repository workaround. - Put consumer-specific scheduling, Docker, tag, flaky, and wrapper policy in the consumer repository. - Keep automatic target expansion and service naming in the consumer's managed command, not in the Rule, BUILD files, or Gazelle. - If an issue requires changing this rule repository, add matching fixture coverage in `rules_test_optimization_tests` before declaring it solved. ## Stop conditions Stop and escalate instead of guessing when: - The repository requires a new public rule behavior not covered by current docs. - A target produces no JSON payloads after the pytest process ran. - The doctor reports missing Git metadata after sync metadata was configured. - The doctor reports missing Bazel metadata. - The only available fix would put `DD_GIT_*`, credentials, or upload endpoints into the test sandbox. - The only tried doctor/uploader placement is the root package in a large monorepo and no lightweight package placement has been attempted. - A private repository fetch returns `404` and SSH/authenticated archive mode has not been confirmed. - Validation requires secrets that are not already available in the environment.