| name | before-after-comparison |
| description | Measure whether a change actually moved throughput or latency, by running the same workflow at the same concurrency on both sides and saying which differences the numbers support. |
| invocation | ["compare-runs","before-and-after","did-it-help","regression-check"] |
| tools | ["list_workflows","validate_workflow","test_connection","start_load_test","get_load_test_metrics","get_step_latencies","get_run_summary","list_load_tests","stop_load_test"] |
| instructions | ["load-test-safety","reading-latency"] |
Comparing a run before and after a change
When to use
Someone changed something — a region setting, a JVM flag, more CICS
tasks, a code deployment, different hardware — and wants to know whether it
helped. Also for the reverse question: whether a change that was supposed to
be neutral cost anything.
Not for "how much load can this take". That is find-concurrency-knee.
The comparison is only worth as much as what was held still
Everything below is one paragraph long because each one has, on its own,
produced a confident wrong answer:
- The same workflow document. A workflow edited between runs measures the
edit. Take the path from
list_workflows and use it for both sides.
- The same concurrency and the same runtime. Throughput is not linear in
either, so a 10-user run and a 20-user run are not comparable by scaling.
- The same host, and nothing else on it. Another test running against the
same region is not an experiment, it is two experiments sharing a host.
- The same time of day, as far as possible. A batch window is a change
you did not make.
If any of these could not be held still, say which, and say it before the
numbers rather than after them.
Steps
list_workflows — pick one document and use its path for both sides.validate_workflow and test_connection once, before either run.- Run the before side:
start_load_test with the chosen concurrency and
runtime. Record the pid.
- Poll
get_load_test_metrics until it finishes. Collect
get_run_summary, and get_step_latencies if the run had -promListen.
- Ask the user to apply the change, and wait. Do not attempt it — this tool
drives a terminal, it does not administer a host.
- Run the after side with identical parameters.
- Collect the same three artefacts.
Prefer at least two runs per side when the change is expected to be small.
One run per side can only distinguish differences larger than the run-to-run
variation, and nothing in a single pair says what that variation is.
Reading the difference
Report each side's counters separately before any delta, so the reader can
see the sample sizes the comparison rests on.
- Completed workflows is the throughput figure. Compare the rate, not the
total, unless both runtimes were identical to the second.
- p50 moving is a change in the typical transaction. p95 moving while
p50 holds is a change in the tail — often queueing or contention rather
than the transaction getting faster, and worth saying in those terms.
- Per-step latencies are where a real explanation lives. "p95 improved by
600ms and the whole 600ms is in step 4" is an answer; "p95 improved" is a
measurement.
- Failures and connect failures must be compared before anything else. A
faster run with more failures is usually faster because of them: a
workflow that failed at step two did not do the work the other one did.
Say how confident the numbers let you be
State the sample count on both sides — the durations are a rolling window,
not the whole run, so get_load_test_metrics reports how many. A difference
smaller than the spread within either side is not a result; say that plainly
rather than reporting a percentage that reads like one.
If the two sides disagree in direction between p50 and p95, or between
throughput and latency, report both and do not pick the flattering one.
Anti-patterns
- Reporting a percentage improvement with no sample size behind it.
- Comparing runs at different concurrency and calling the difference a change
in the host.
- Quietly dropping the failure counts because they complicate the story.
- Attributing the improvement to the change. This measures from outside the
host: it can say the transaction got faster and when, never why. Name what
moved and hand the cause to someone who can see inside.
- Running the after side while the before side is still finishing.
