| name | collect-user-feedback |
| description | Use this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals. Trigger on "add thumbs up/down", "collect feedback on responses", "how do I know if users like the answers", "improve from real usage", "human feedback loop". Turn real user signal into your best source of eval data. |
| license | CC0-1.0 |
Collect user feedback
Your users are running the best eval you have, for free, every day. Capturing their reactions and attaching them to the trace turns production into a continuous source of labeled data. Most teams either skip this or collect it and never use it.
Capture both explicit and implicit signals
Explicit (ask directly):
- Thumbs up/down on a response (the simplest, highest-signal control).
- A short reason on thumbs-down (dropdown: wrong / unhelpful / unsafe / other).
- Accept / edit / reject on a suggested output (great for copilot-style apps).
Implicit (infer from behavior, no extra UI):
- Did the user retry, rephrase, or immediately give up? (dissatisfaction).
- Did they copy/use the output, or continue the task? (satisfaction).
- Did a downstream action succeed (the code ran, the ticket resolved)?
Implicit signals are noisier but you get them on 100% of traffic instead of the small % who click thumbs.
Wire it to the trace (this is the key step)
Attach every feedback signal to the trace/span it is about (by trace ID), with the user, timestamp, and the reason. Most observability platforms (Langfuse, Phoenix, Opik, LangSmith) have a feedback/scores API for exactly this. Without the trace link, feedback is a number with no context; with it, a thumbs-down is a fully debuggable example.
Use it (do not just collect it)
- Monitor feedback rate + thumbs-down rate over time; alert on spikes (see
set-up-drift-alerts).
- Triage thumbs-down traces into candidate test cases (see
trace-based-testing) and your eval set (build-eval-dataset).
- Correlate feedback with your automated eval scores to check your judge actually agrees with humans (calibration).
- Close the loop: the corrections/edits users make are gold-standard reference outputs, use them.
Handle it responsibly
- PII / privacy: feedback text can contain sensitive data, redact before it hits a third-party backend (
redact-pii-for-tracing).
- Bias: thumbs are a biased sample (angry and delighted users click most). Do not treat thumbs-up rate as ground-truth quality; use it as signal, validate with evals.
Verify
- A thumbs-down in the UI shows up attached to the right trace in your backend.
- Thumbs-down traces are being triaged into the eval/regression set, not just counted.
- A dashboard shows feedback trends with an alert on a drop.
Anti-patterns
- Collecting feedback that never links to a trace (a number you cannot act on).
- Collecting it and never feeding it back into evals or fixes.
- Treating thumbs-up rate as objective quality (selection bias).
- Logging raw feedback with PII to a SaaS backend without redaction.
Grounding
Human feedback as the signal for LLM quality underpins RLHF and preference modeling (Ouyang et al., InstructGPT, arXiv:2203.02155) and human-aligned evaluation (Zheng et al. 2023, arXiv:2306.05685). Here it feeds the lighter-weight eval + observability loop (trace-based-testing, build-eval-dataset).
