Use this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement. Trigger on "is this answer reliable", "flag low-confidence answers", "cheap hallucination check", "confidence score without a ground truth", "self-consistency check". Ships a runnable, tested scorer you can put inline or on sampled traffic.
Use this to estimate what an LLM call or feature will cost, and to compare models on price, before or after shipping. Trigger on "how much will this cost", "estimate my OpenAI/Anthropic bill", "is a cheaper model worth it", "cost of this prompt", "project my LLM spend". Ships a runnable, tested calculator so the numbers are real, not hand-waved.
Use this to strip PII (emails, credit cards, SSNs, phone numbers, IPs, IBANs) out of text before it is logged to a tracing backend or sent to a third-party model. Trigger on "redact PII", "scrub sensitive data", "mask PII before logging", "don't send customer data to the tracing tool", especially for finance/healthcare/regulated apps. Ships a runnable, tested scrubber with a Luhn check to cut false positives.
Use this to check that your LLM tracing actually emits complete, spec-compliant spans, so cost/latency/model dashboards downstream are not full of holes. Trigger on "are my traces complete", "validate my instrumentation", "my spans are missing fields", "lint my OTel GenAI spans", "test my tracing". Ships a runnable, tested validator you can drop into your instrumentation tests.
Use this to set up human review and annotation of LLM traces, so people (often domain experts) can label outputs, do error analysis, and build a trustworthy golden dataset. Trigger on "review my LLM outputs", "have an expert label these", "error analysis", "annotate traces", "build a golden dataset", or when automated evals are not enough for a high-stakes or specialized domain. Looking at your data is the highest-ROI thing you can do.
Use this to measure whether an AI agent actually completed its task end to end, not just whether individual LLM calls looked fine. Trigger on "is my agent working", "measure agent success rate", "evaluate my agent", "how good is my agent", "agent completion rate", or evaluating a multi-step/tool-using agent. Score the outcome of the whole task, plus the path it took.
Use this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone. Trigger on "A/B test my prompt", "roll out a new model safely", "compare two prompts in production", "canary this change", "does this actually improve things for real users". Measure impact on real users, gated, before a full switch.
Use this to make an LLM app resilient to provider failures, rate limits, timeouts, and outages. Trigger on "handle LLM API errors", "add retries/fallbacks", "the app breaks when OpenAI is down", "rate limit errors", "make my LLM calls reliable", "timeout handling". Add retries, timeouts, and model/provider fallbacks, and observe them so failures are visible.