datadog-java-test-optimization-onboarding

<!-- Unless explicitly stated otherwise all files in this repository are licensed under the Apache 2.0 License. This product includes software developed at Datadog (https://www.datadoghq.com/) Copyright 2025-Present Datadog, Inc. --> # Datadog Java Test Optimization onboarding Use this skill when you need to instrument a Bazel Java repository with Datadog Test Optimization. The skill is intentionally project-neutral: it is stored in this repository as a Codex-compatible skill, but any agent can read it as a normal implementation guide. Automatic invocation-scoped manifest onboarding currently supports Go and Python only. Java continues to use the static single-service or static multi-service contracts in this skill. Do not enroll Java targets in `test_optimization_manifest_sync` or describe the managed Go/Python command as a Java onboarding path. ## Non-negotiable contract Keep the RFC contract intact: - Tests write JSON payloads to `TEST_UNDECLARED_OUTPUTS_DIR`. - Bazel collects those files under `bazel-testlogs/<target>/test.outputs/`. - The doctor validates local files after `bazel test`. - The uploader runs after the doctor with `bazel run`. - Use the default Python 3.10+ uploader unless a temporary rollback explicitly requires `use_python_uploader = False`. Its coordinator prepares shared CODEOWNERS, contexts, schemas, freshness, and telemetry once, then starts up to eight independent file workers by default. Each worker owns enrichment, validation, preventive splitting, retries, and cleanup for one test, coverage, or telemetry source file. - Run one uploader process. Use `--dry-run --validate-enrichment` to prepare requests without HTTP or deletion, `--debug` only for verbose redacted diagnostics, and review the final file/type/split/request/cleanup totals. - Test bodies are split before HTTP when they exceed `4_718_592` bytes. HTTP `413` is terminal and must not trigger a retry or adaptive split; coverage and telemetry are not split. - Do not add payload proxies or upload-from-test-sandbox paths. - Do not manually set manifest or payload-in-files environment variables in consumer test rules; `dd_topt_java_test` owns that wiring. - Do not pass `DD_GIT_*` through `--test_env`; use `--repo_env` for sync metadata. - Do not pass uploader credentials or upload endpoints into the test sandbox. - Put `--remote_download_minimal`, `--remote_download_regex=.*test[.]outputs.*`, and `--zip_undeclared_test_outputs` in the active test `.bazelrc` config when remote execution or remote cache can leave test outputs remote-only. - Configure doctor/uploader with repeatable `--bep-json=<path>` flags, `--freshness-source=bep`, `--freshness-mode=required`, `--artifact-source=bep`, and `--artifact-staging-dir=<temp-dir>`. If BEP still points at HTTP/HTTPS `outputs.zip` artifacts, use `--remote-artifacts=download` or `required` without a downloader. Use a downloader only for bytestream/CAS/custom-auth artifact providers. - Run pilot tests with a fresh `--build_event_json_file` path per Bazel test invocation; pass the same paths to doctor/uploader with `--bep-json`. - In CI, keep a per-job diagnostic report directory with `DD_TEST_OPTIMIZATION_REPORT_DIR` or wrapper `--report-dir`, and configure wrapper `--support-bundle` or `DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE` for complete escalation artifacts. For first-pass customer troubleshooting after tests have run, ask for `bazel run //<topt-package>:dd_test_optimization_doctor -- --support-bundle=<path>` with any matching BEP/artifact flags. Replace `<topt-package>` with the package that owns the logical doctor/uploader pair; use `//:` only when a small repository intentionally keeps the targets at the root. For bundle triage, inspect `summary.md`, `diagnostics.json`, `reports/doctor-report.json`, optional uploader reports, and `command/flags.json` in that order. ## First actions 1. Read the consumer repository's Bazel shape before editing: - Does it use `MODULE.bazel`, `WORKSPACE`, or both? - What command does the repository use for Bazel: `bazel`, `bazelw`, `bzl`, or a repo-local wrapper? - What is the Bazel repository name for `rules_java`? - What Java version and toolchain does Bazel use? - Which repository owns Java dependencies, Maven artifacts, and lockfiles? - Where is the dd-java-agent JAR already declared, or how should the consumer repository source it? - Does the repository already have a Java/JUnit test wrapper macro? - Which lightweight package should own the logical doctor/uploader pair (for example `//tools/test_optimization`)? - Does fetching this rules repository require SSH git or authenticated archive access? - Which runtime test targets should emit payloads? - Which build-only or analysis-only targets should not be expected to emit payloads? - Is `FETCH_SALT` absent from the normal test, doctor, and uploader flow? 2. Read this repository's current docs when details are needed: - `README.md` for quickstart and current command flow. - `docs/Language_Onboarding.md` for language-specific Java guidance. - `docs/Installation_Reference.md` for helper APIs and pinning. - `docs/Uploader_Reference.md` for doctor, dry-run, and upload behavior. - `docs/Troubleshooting.md` for failure diagnosis. 3. Pick the correct path: - Bzlmod repo: follow [bzlmod-onboarding.md](references/bzlmod-onboarding.md). - WORKSPACE repo: follow [workspace-onboarding.md](references/workspace-onboarding.md). - Validation and debugging: follow [validation-checklist.md](references/validation-checklist.md) and [troubleshooting.md](references/troubleshooting.md). ## Universal shape Every successful Java onboarding should end with these pieces: - Repository or module resolution fetches Test Optimization metadata. - The consumer repository owns `rules_java`, Java toolchains, test framework dependencies, and the dd-java-agent artifact. - Java tests use `dd_topt_java_test` directly or through a repo-local wrapper. - Existing repository wrapper policy stays in the consumer repository; the Datadog macro wraps the raw Java test rule or wrapper rule and injects the Java agent plus Test Optimization runtime files. - Java onboarding explicitly sets `stage_sources = True` in the direct `dd_topt_java_test` call or in the repo-local wrapper's call into `dd_topt_java_test`, unless the repository intentionally opts out after accepting that source location tags may be missing. - The workspace has exactly one logical doctor/uploader pair. In monorepos, place it in a lightweight package such as `//tools/test_optimization`; root labels are still fine for small repositories. - `.bazelrc` or CLI commands provide sync metadata with `--repo_env`. - Test commands use a named config such as `--config=test-optimization`. - Remote-output-sensitive test configs include `--remote_download_minimal --remote_download_regex=.*test[.]outputs.*` and `--zip_undeclared_test_outputs`. - Validation commands pass each matching BEP file with repeatable `--bep-json` flags and required BEP freshness/artifact flags. Use `DD_TEST_OPTIMIZATION_*` environment variables only for single-invocation manual flows where one BEP file is sufficient. - CI wrappers write `doctor-report.json`, one selected uploader report (`uploader-dry-run-report.json` or `uploader-upload-report.json`), and `dd-test-optimization-support.zip` under a per-job report directory. Prefer the wrapper support bundle for full CI escalation; use the doctor-only support bundle for the simplest initial customer request. Keep individual reports for local inspection and manual fallback flows. - `FETCH_SALT` is used only for a separate, explicit `bazel sync --only=<repo> --repo_env=FETCH_SALT="$(date +%s)"` refresh, never as part of normal test, doctor, or uploader commands. - A real upload processes every available fresh valid payload after validation attempts. The wrapper preserves the earliest test, doctor, or uploader exit code; uploader errors never replace an earlier test result. Use the consumer's existing Bazel entrypoint in all commands. Do not switch a repository from `bzl` or `bazelw` to raw `bazel` just because examples use the generic binary name. ## Branch and PR hygiene Before making changes in a real repository, confirm whether to use the current branch or create a new branch from the latest default branch. Keep onboarding changes reviewable: - Put reusable rule changes in `rules_test_optimization`, not in a consumer repository workaround. - Put consumer-specific scheduling, Docker, tag, flaky, and wrapper policy in the consumer repository. - If an issue requires changing this rule repository, add matching fixture coverage in `rules_test_optimization_tests` before declaring it solved. ## Stop conditions Stop and escalate instead of guessing when: - The repository requires a new public rule behavior not covered by current docs. - A target produces no JSON payloads after the Java test process ran. - The doctor reports missing Git metadata after sync metadata was configured. - The doctor reports missing Bazel metadata. - The only available fix would manually set manifest or payload-in-files env vars in consumer tests. - The only available fix would put `DD_GIT_*`, credentials, or upload endpoints into the test sandbox. - The only tried doctor/uploader placement is the root package in a large monorepo and no lightweight package placement has been attempted. - A private repository fetch returns `404` and SSH/authenticated archive mode has not been confirmed. - Validation requires secrets that are not already available in the environment.