| name | measuring-well |
| description | Arithmetic on measurements, and the traps that make a number look right and be wrong. Use when computing a rate, a percentile or a drift, when subtracting timestamps, when comparing two runs or two arms, when deciding whether an outlier matters, or when a measurement disagrees with a prediction. Also use before quoting any measured figure as evidence for a decision. |
Measuring well
A wrong number is worse than no number, because a decision gets built on it. This is a
collection of traps that each produced a confident, plausible, wrong figure, and the
check that catches each one.
Every example is real and the numbers are the ones that were printed.
Never subtract across clock bases
Two instants are only comparable if they came from the same clock. This sounds obvious
and is violated constantly, because the violation looks like ordinary arithmetic.
Across machines. There is no synchronisation good enough for a sub-millisecond
segment. When a figure would need to cross, name the two hops and leave them unmeasured
rather than interpolating - a chain of intervals each local to one machine, with the
network hops written as "not measured", is worth more than a total that fabricates the
part nobody can see.
Where the code already refuses this, it is deliberate: a frame age computed from a birth
mark on another machine reports as absent rather than as a number.
Within a machine. mach_absolute_time and mach_continuous_time diverge by time
spent asleep, so Core Animation's timestamps and this project's monotonic marks agree
until the machine sleeps once and then disagree by however long that was. An audio
device's clock is neither of them. QPC is neither.
If two bases must be related, measure the offset once, and say in the comment that it is
an offset between two bases rather than a shared clock.
A count and a span must describe the same interval
The trap, and the arithmetic:
frames captured 2880480
qpc span 60.0009 s
2880480 / 60.0009 = 48007 Hz → +151 ppm against 48000 nominal
That +151 ppm is entirely an artefact. Both the device position and the QPC position
timestamp the first frame of their packet, so the span runs first-frame to
first-frame and excludes the last packet's audio that the count includes. One 480-frame
packet over a minute is exactly 150 ppm.
device position advanced 2880000 frames between the two instants
2880000 / 60.0009 = 47999.27 Hz → -15 ppm
The honest figure was −15 ppm. The wrong one was about to become the largest term in a
gate whose entire subject was drift measured in parts per million.
The check: measure the elapsed quantity between the two instants the span was taken
at, not a total that includes something outside them.
A maximum is not a frequency
recv to injected p50 123.90 us p99 292.86 us max 4321.28 us
A maximum twelve times the p99 reads like an argument for building something to avoid it.
Counted:
over 0.5 ms 13 0.092 %
over 1.0 ms 1 0.007 %
over 2.0 ms 1 0.007 %
over 5.0 ms 0 0.000 %
One event in 14193. The argument evaporated, and a virtual HID device did not get built.
The check: count crossings of a threshold. Report the maximum beside them, never
instead of them.
Comparing two runs: what moves between them on its own?
alignment off presentation wait p50 5.51 ms
alignment on presentation wait p50 2.38 ms
That looks like a 3.13 ms improvement from a mechanism. It was not. The quantity being
aligned drifts on its own at 0.02 ms per second, crossing its period every two hundred
seconds, and the arms were forty seconds each. The "aligned" arm had started at 1.89 ms
against a 2.00 ms target and held twenty-five times out of forty-five: it was a
favourable draw, not a mechanism.
The check: before comparing arms, ask what changes between them without anybody
touching it, at what rate, and whether the arm is long enough to average it out. If it is
not, do not compare arms - find the step inside one run instead.
Finding a step instead of comparing ends
When a known change is applied part-way through a run, do not compare the first sample
against the last. The drift between them can be larger than the change:
phase 7.43 ms -> 1.03 ms after a 3.00 ms shift
That looked like a −6.40 ms movement and was mostly drift, which sweeps a whole period
every few hundred seconds.
The method: difference consecutive samples, take the differences the short way round
if the quantity is a phase, and look for one movement far larger than the per-sample
drift. A 3 ms step against 0.125 ms of drift per sample is twenty-four times the largest
innocent movement and stands out on its own. Then check there is exactly one such
step: two means something else moved too and neither can be attributed.
This also removes the need to align a shell's clock with a program's, which would have
been another base to cross.
The same numbers can fit two different stories
Six sessions measured a phase and produced:
0.98, 2.21, 3.26, 4.93, 5.77, 6.84 ms
A gate checked that each session's phase barely moved within itself and that the sessions
covered much of the period between them, and passed. Both of those are also true of a
single slowly drifting clock, which is what it actually was - and independent draws come
out strictly ordered about once in seven hundred and twenty times.
The check: when a criterion would pass under two different mechanisms, it does not
distinguish them. Find the quantity that differs. Here it was the gap between sessions:
predicted from elapsed time times the drift the sessions agreed on, every boundary matched
to within 0.28 ms of an 8.33 ms period, and the lottery story died.
Prove the traffic went where you think
A datagram addressed to this machine's own routable address never leaves it. Measured:
1000 packets to 192.168.1.108 lo0 +1016 en0 +138
1000 packets to 192.168.1.1 lo0 +16 en0 +1091
A gate's "radio arm" was the loopback path measured twice.
The check: interface counters, not the address you typed. And reordering is intrinsic
evidence in the other direction - loopback delivers strictly in order, so a run reporting
22557 reordered packets did cross a real network, whatever else is in doubt.
An instrument nobody configured deliberately is not evidence
A smoke test of an audio encoder reported 126 bytes per frame, about 201 kbps against a
128 kbps target, and that discrepancy was written into a work brief as something to
explain. It did not need explaining: the smoke test had left every setting at its
default. Configured deliberately, the same encoder produced 81 bytes and 129.6 kbps.
The check: before treating a number as a finding, establish that whatever produced it
was configured on purpose. Especially when the number is your own.
Derive before building
Three mechanisms in this project were built and then measured to do nothing. One was
neutral by derivation and could have been ruled out on paper in five lines:
period T = 8.33 ms, pipeline work P = 3 ms, content drawn at t = 0
capture at once ready 3 shown 8.33 age 8.33
capture at +4 ready 7 shown 8.33 age 8.33 ← neutral
capture at +6 ready 9 shown 16.67 age 16.67 ← worse
source at +4 ready 7 shown 8.33 age 4.33 ← the lever
The check: write the arithmetic before writing the code, and prefer a cheap experiment
to a confident implementation. A mechanism measured to do nothing must then be deleted
rather than left wired, because one that looks connected will be believed.
