Specifies what analytics events to track, when they fire, and what properties to include, as a contract between product and engineering that prevents undertracked features. Use before engineering builds a feature or when auditing existing tracking for gaps. For the dashboard built on top of these events, use measure-dashboard-requirements instead.
Specifies what analytics events to track, when they fire, and what properties to include, as a contract between product and engineering that prevents undertracked features. Use before engineering builds a feature or when auditing existing tracking for gaps. For the dashboard built on top of these events, use measure-dashboard-requirements instead.
An instrumentation spec defines what analytics events to track, when to fire them, and what properties to include. It serves as a contract between product and engineering, ensuring consistent data collection that enables accurate measurement. Good instrumentation specs prevent the "we can't answer that question because we didn't track it" problem.
When to Use
Before engineering implements a new feature
When defining analytics requirements for experiments
When auditing existing tracking for gaps or inconsistencies
When onboarding a new analytics tool
Before launch to ensure measurement is in place
When NOT to Use
You are specifying the dashboard built on top of the events -> use measure-dashboard-requirements
You need experiment-specific metrics and variants, not product-wide tracking -> use measure-experiment-design
The feature itself is not yet specified (no flows to instrument) -> use deliver-prd first
You are analyzing data you already collect -> use measure-experiment-results or measure-survey-analysis
Instructions
When asked to create an instrumentation spec, follow these steps:
Define Analytics Goals
Start with the questions you need to answer. What will you measure? What decisions will this data inform? This prevents over-instrumentation while ensuring nothing important is missed.
Identify Events to Track
List each user action or system event that should be tracked. Follow consistent naming conventions (typically noun_verb or verb_noun in snake_case). Each event should represent a distinct, meaningful action.
Specify Event Triggers
For each event, describe exactly when it fires. Be precise: "When user clicks Submit button" vs. "When form is submitted successfully." These are different events with different meanings.
Define Event Properties
List the properties (attributes) attached to each event. Include property name, data type, description, and example values. Properties provide context that makes events useful.
Document User Properties
Identify persistent user-level attributes that should be associated with all events (e.g., subscription tier, account creation date). These enable segmentation in analysis.
Address PII and Privacy
Flag any properties that contain personally identifiable information. Document how PII should be handled - hashing, encryption, or exclusion.
When any model input, output, retrieval context, or tool call is captured, extend this section to cover the trace as well. The trigger is capture, not authorship: it fires whether the sensitive text was typed by a user, retrieved from a tenant's documents, carried in a system prompt, passed to or returned from a tool, read out of an uploaded file or a batch job, or generated by the model itself. A feature with no direct user input can still write a customer's contract text into a trace store.
A trace is not an event: an event carries properties you chose in advance, while a trace carries free text that can contain anything, including data no property schema anticipated. Decide and record:
Data classes captured. Name them (user text, retrieved documents, system prompt, tool arguments and results, file contents, model output). "The trace" is not an answer.
Minimization, at both boundaries. What is dropped or redacted before the trace leaves the process, and separately what is dropped before it is stored. A redactor that runs only at the storage layer has already sent the raw text over the wire.
Access. Who can read traces, and whether reads are logged. Trace stores are routinely the least-governed copy of the most sensitive data a feature handles.
Output Format
Use the template in references/TEMPLATE.md to structure the output. A complete spec fills every template section: Overview; Event Inventory; User Properties; PII & Privacy Considerations; Implementation Notes; and Testing Checklist.
Quality Checklist
Before finalizing, verify:
Event names follow consistent naming convention
Each event has a clear, unambiguous trigger
Properties include data types and example values
PII is identified and handling is documented
Events map to the analytics questions you need to answer
Testing checklist enables QA verification
If any model input, output, retrieval context, or tool call is captured: the data classes are named, minimization is decided at both the egress and the storage boundary, access and read-logging, retention and deletion, consent, and sampling are all decided, not deferred
Examples
See references/EXAMPLE.md for a completed example.
Retention and deletion. How long, what deletes them, and how a deletion request reaches a trace that was copied into an evaluation set.
Consent and opt-out. Whether the subject agreed, and what the feature does when they decline.
Sampling. What fraction is captured and how that sample is chosen. A uniform sample is the wrong instrument for finding rare failures; if traces exist to diagnose bad output, oversample the flagged cases and say so.
Create Testing Checklist
Define how QA should verify that tracking is implemented correctly. Include steps to validate events fire at the right times with correct properties.
