developing-rally

name	developing-rally
description	Work on Rally's own codebase, not running benchmarks with it. Use when setting up the dev environment, running Rally's tests or linters, navigating its source, debugging Rally's own code, or making changes to Rally itself.

Developing Rally

For working on Rally, not just running it. The docs under References are the source of truth — prefer them over guessing.

Setup

Development uses uv and needs jq. Rally runs on Linux/macOS only (no Windows) and targets Python 3.10–3.13. From a clone:

make install            # create .venv and editable-install with dev extras
source .venv/bin/activate
esrally --help          # runs your working-tree code (editable install)

make install editable-installs the workspace, so esrally / esrallyd (or uv run esrally …) run your working-tree code directly. Avoid the ./rally / ./rallyd wrappers while developing: they auto-update from git (via run.sh) and only run on a clean master, otherwise erroring There are uncommitted changes … — pass --skip-update if you must use them.

Everyday make targets

• make lint / make format — run/apply pre-commit linters (black, isort, …).
• make test — unit tests. Scope with make test ARGS=tests/foo_test.py; make test-all runs across Python 3.10–3.13.
• make it — integration tests (slow, CI-oriented). make benchmark runs the micro-benchmarks.
• make docs / make serve-docs — build/serve the docs/ site.

Before committing run make lint test (unit tests + lint; integration tests are too slow for the inner loop). Optionally make install-pre-commit to run lint automatically on each commit.

Testing & conventions

• Unit tests live in tests/ mirroring esrally/, named *_test.py; integration tests in it/, micro-benchmarks in benchmarks/.
• asyncio_mode = strict, so async tests need an explicit @pytest.mark.asyncio.
• xfail_strict = true, so an xfail test that unexpectedly passes fails the suite.
• Type hints are not used broadly — they're added opportunistically per module, with the strictly-checked modules listed under [tool.mypy].overrides in pyproject.toml. Match the surrounding code rather than typing everything.
• Keep the load-driver hot path tight. Code in driver/ runners, schedulers, and parameter sources runs per operation during measurement, so extra overhead (logging, allocations, blocking I/O, locks) perturbs the throughput/latency/service_time numbers Rally exists to report. Do expensive work outside the measured loop.
• For a big or risky hot-path change, consider quantifying its overhead with a pytest-benchmark micro-benchmark under benchmarks/ — coverage there is sparse, so treat this as a tool to reach for, not a standing requirement. Take the benchmark fixture, call benchmark(fn), decorate with @pytest.mark.benchmark(group="...", warmup="on", disable_gc=True), and compare against a baseline variant. See benchmarks/driver/runner_test.py.

Source organization (esrally/)

• rally.py — CLI entry point: argument parsing and subcommand dispatch.
• racecontrol.py — orchestrates a race end-to-end via the pipelines registry.
• mechanic/ — provisions, launches, and tears down Elasticsearch (only for from-* pipelines).
• driver/ — generates load: schedules tasks and runs runner.py operations against the cluster.
• track/ — loads and parses tracks/challenges; tracker/ generates a track from an existing cluster.
• metrics.py — the metrics store (in-memory or Elasticsearch); reporter.py — the summary report.
• client/ — Elasticsearch client factory; telemetry.py — telemetry devices; config.py / types.py — config.

Actor system

Rally is a distributed system built on Thespian; even a single-machine race runs as actors. Three logical roles (docs/rally_daemon.rst) that, by default, all run on the one machine where you invoke esrally:

• benchmark coordinator — drives the whole race and shows results.
• load driver — interprets and runs the track.
• provisioner — configures and starts Elasticsearch.

--load-driver-hosts and --target-hosts split the load driver and provisioner onto other machines; multi-machine runs use the esrallyd daemon. esrally/actor.py holds the shared RallyActor base and the bootstrap logic; docs/architecture/actor_system.md walks through the actors and their message flow with sequence diagrams. If actor-system startup fails (common on a VPN), set THESPIAN_BASE_IPADDR to a routable address.

Debugging

• Rally logs to ~/.rally/logs/rally.log (human-readable) and rally.json (structured); profiling output goes to profile.log. Console output is only a summary — the full stack trace of a failure is usually in the log file, not on stdout.

• Each log line carries the emitting actor's address (%(actorAddress)s), so you can trace a failure across the actor system. The actor framework's own internal log is separate: ~/.rally/logs/actor-system-internal.log (path via THESPLOG_FILE) — check it when actors fail to start or communicate, and set THESPLOG_THRESHOLD (default INFO) to DEBUG for more detail on the actor system itself.

• Raise the log level to see more: edit ~/.rally/logging.json (created on first run from esrally/resources/logging.json) and set "level": "DEBUG" on the root logger for everything, or add a per-module entry under loggers to scope it (matching the existing entries, which just set a level and inherit the root handlers), e.g.:

  "loggers": {
    "esrally.driver": {"level": "DEBUG"}
  }
  

  Then re-run. Revert to INFO when done — DEBUG is verbose.

Reproduce a failing integration test

Integration tests in it/ drive real races, so a failure usually reflects a real bug. To debug any IT failure:

• Re-run the single failing test in isolation: make it ARGS="it/<file>.py -k <pattern>".

• Reproduce outside the test harness by running the equivalent race yourself — fast, against a throwaway local cluster — then read the logs (see Debugging, raising the level to DEBUG as needed):

  esrally race --distribution-version=9.4.0 --track=geonames --test-mode --on-error=abort
  

• If the failure is in the actor system, add THESPLOG_THRESHOLD=DEBUG and inspect a running system with the Thespian shell: python3 -m thespian.shell (see docs/rally_daemon.rst).

References

• Developer setup & key components: docs/developing.rst
• Contribution workflow (PRs, license headers, CLA): CONTRIBUTING.md
• Distributed setup & the actor-system roles: docs/rally_daemon.rst
• Actor message flow (sequence diagrams): docs/architecture/actor_system.md
• Thespian actor framework: https://github.com/kquick/Thespian
• All flags (do not invent flags; verify here or via esrally <subcommand> --help): docs/command_line_reference.rst
• Running benchmarks against a cluster: the running-benchmarks skill