04c-end-user-feedback

# End-user feedback collection (production) Canonical reference for **collecting end-user feedback in production** and writing it back as **MLflow Assessments** on the originating trace. Pair with [Track A 02-agent-framework](../../tracks/A-custom-agent-apps/02-agent-framework/SKILL.md) (which owns the tracing setup that produces the `trace_id`) and [04-evaluation-runs](../04-evaluation-runs/SKILL.md) (which owns *expert / labeling-session* feedback). ## Upstream Lineage This skill references Databricks Agent Skills' `databricks-mlflow-evaluation` skill for feedback-to-dataset, production trace analysis, and evaluation-loop guidance. If collected feedback needs to become labeled eval data or monitoring signal, consult the upstream skill first, then apply this skill's production feedback correlation and assessment write-path contracts. This skill is the **production user-feedback** counterpart to [04-evaluation-runs](../04-evaluation-runs/SKILL.md): | Concern | Owner skill | |---|---| | End-user thumbs/rating from a live app → `log_feedback` on the trace | **This skill (04c)** | | Expert labeling sessions, label schemas, Review App, `session.sync()` | [04-evaluation-runs](../04-evaluation-runs/SKILL.md) (Human feedback section) | | Stakeholder business sign-off gate before deploy | [04b-stakeholder-signoff](../04b-stakeholder-signoff/SKILL.md) | | Periodic resampling of production traces into labeling sessions | [07-production-monitoring](../07-production-monitoring/SKILL.md) | | AppKit-side UI, Vote table, REST POST plumbing | [`apps_lakebase/skills/08-appkit-feedback`](../../../apps_lakebase/skills/08-appkit-feedback/SKILL.md) | > **Canonical write-path (always):** `mlflow.log_feedback(trace_id=..., name="user_feedback", value=..., source=AssessmentSource(HUMAN, source_id=user_id), rationale=...)`. Whether that runs in your Track A Agent App's `@invoke` handler, a sidecar FastAPI route, or the AppKit `08-appkit-feedback` REST proxy, the **API surface is the same** and the assessments land on the same trace. --- ## When to Use - You have a **deployed agent** (Track A on Databricks Apps, or Model Serving) that already emits MLflow traces. - You want **end users** of a frontend (AppKit dashboard, template chat UI, Slack bot, etc.) to be able to give feedback (👍/👎, 1–5 stars, free-form comment) that lands on the originating trace. - You need to **return `trace_id`** to the frontend so the user can later attach feedback to the same trace. - You need **streaming** support — the trace id is only available **after** the SSE stream finishes; the UI must wait. - You want to **update or delete** an assessment (user changed their mind, accidental thumb). - You want to **analyze** collected feedback (positive rate, dimensional ratings, traces with feedback) for monitoring or to seed an evaluation dataset. Skip if you only need **expert / SME labeling** — that is `04-evaluation-runs` → Human feedback. Skip if you have **no live users** yet — instrument tracing first ([Track A 02](../../tracks/A-custom-agent-apps/02-agent-framework/SKILL.md)) and come back. --- ## Architecture ``` End user clicks 👍 / 👎 / star rating / writes comment │ ▼ Frontend calls POST /feedback body: { trace_id, value, rationale?, dimension? } auth: x-forwarded-access-token (OBO) — see Track A 04-authentication │ ▼ Backend route (Track A Agent App or AppKit server) resolves user_id from x-app-user-email / x-forwarded-email first, then falls back to OBO current_user.me() only when the inbound Bearer is the user's own OBO token. │ ▼ mlflow.log_feedback( trace_id=..., name="user_feedback", value=value, rationale=rationale, source=AssessmentSource(HUMAN, source_id=user_id), ) │ ▼ Assessment attached to the trace in Unity Catalog (OTeL trace location) │ ▼ Visible in: - MLflow Trace UI → Assessments panel - SQL over UC trace tables (analytics) - mlflow.search_traces() (dataset construction) - Production monitoring dashboards (Step 07) ``` The two **correlation IDs** you can use: | ID | Source | When to pick | |---|---|---| | `trace_id` | `_resolve_active_trace_id()` — wraps `mlflow.get_current_active_span()` with `mlflow.tracing.fluent.get_last_active_trace_id()` fallback | **Default.** Simplest; no extra plumbing. Works for non-streaming responses. | | `client_request_id` | Frontend-generated UUID, passed in the request, attached to the trace via `mlflow.update_current_trace(client_request_id=...)` | Pick when you can't return `trace_id` synchronously (deeply async pipelines, WebSockets where the UI generates IDs first). | Both end up on the same trace; you choose which one the **feedback POST** carries. --- ## Trace ID Contract: Two Forms A `trace_id` you read off the wire and a `trace_id` you pass to the MLflow Assessments backend are **not always the same string**. Skills that confuse the two will quietly drop assessments. Document and exchange both forms explicitly. ### Form 1 — Client / UI form (UC v4) What the agent returns to the frontend (in the JSON body or the SSE `done` event) is the **Unity Catalog v4 trace URI**: ``` trace:/<catalog>.<schema>.<prefix>/<bare_id> ``` Example: `trace:/main.skyloyalty_ops.agent_traces/0a1b2c3d4e5f...`. This is the canonical, fully qualified handle the UI stores per assistant message and replays into `POST /feedback`. Storing the bare id alone is fragile because the same bare id can collide across catalogs / schemas. Always round-trip the full `trace:/...` URI. ### Form 2 — Assessments backend form The MLflow Assessments API (`mlflow.log_feedback`, `mlflow.override_feedback`, `mlflow.delete_assessment`) accepts whatever the **target runtime** requires. Concretely: - On Databricks workspaces running MLflow 3.1+, the backend accepts the UC v4 URI directly. - On older runtimes or self-hosted MLflow, the backend wants the bare id (`<bare_id>`) and resolves the trace from the configured experiment. The feedback route MUST normalize the inbound id once, at the edge, before calling `log_feedback`. Centralize the conversion so the agent code never branches on runtime version inline: ```python def to_assessments_id(trace_uri_or_id: str) -> str: """Convert UC v4 client form to whatever the Assessments backend expects.""" if trace_uri_or_id.startswith("trace:/"): # Modern Databricks runtimes accept the URI as-is. # If your runtime requires a bare id, strip to the segment after the last '/'. return trace_uri_or_id return trace_uri_or_id # already bare ``` Document at the top of the feedback route which form your runtime accepts. Cross-reference [Track A 02-agent-framework](../../tracks/A-custom-agent-apps/02-agent-framework/SKILL.md) for the producer side that emits the URI. --- ## Trace assessment round-trip gate Before declaring feedback wired-up, exercise the full assessment lifecycle against a **single trace** and verify the result reads back from the SQL warehouse. Run the gate against a known trace from a real `/chat` round trip: ```bash python genai-agents/sdlc/04c-end-user-feedback/scripts/feedback_round_trip.py \ --trace-id "$KNOWN_GOOD_TRACE_ID" \ --user-id "$EXPECTED_USER_EMAIL" \ --assessments-table "$MLFLOW_TRACING_TABLE_PREFIX"_assessments \ --warehouse-id "$MLFLOW_TRACING_SQL_WAREHOUSE_ID" ``` The script exercises `log_feedback → override_feedback → delete_assessment → re-log` and verifies via the SQL warehouse that: 1. `log_feedback` returns a non-empty `assessment_id`. 2. `override_feedback` preserves `assessment_id`. 3. `delete_assessment` succeeds. 4. The re-log mints a fresh `assessment_id` distinct from the deleted one. 5. The latest warehouse row matches the re-log id and the deleted id does not leak back. Wire this into the same CI step that runs the dataset / scorer smoke tests. Required env for the warehouse verify: `MLFLOW_TRACING_SQL_WAREHOUSE_ID`, `MLFLOW_TRACING_TABLE_PREFIX`, `DATABRICKS_HOST`, `DATABRICKS_TOKEN`. Missing warehouse inputs fail by default. Use `--api-only` only for a local smoke test that intentionally does not claim end-to-end verification. If any step fails — `log_feedback` rejects the id, `delete_assessment` 404s, the re-log mints the same id, or the SQL warehouse omits or duplicates rows — the deployment is blocked. The two most common failures this gate catches: 1. The frontend stored the bare id instead of the UC v4 URI, so the backend can't resolve the trace. 2. The runtime accepts `log_feedback` but the warehouse lag means the row isn't queryable yet — the script's `--warehouse-wait-seconds` flag (default 10s) absorbs replication lag instead of debugging it post-launch. --- ## Step 1 — Return `trace_id` from the agent response The trace id must reach the frontend *somehow*. Pick **one** of three patterns based on your transport. ### Defensive `trace_id` capture Both `mlflow.get_current_active_span()` and `mlflow.tracing.fluent.get_last_active_trace_id()` are valid sources, but each can return `None` depending on **when** in the request lifecycle you call it. Use this helper everywhere instead of either bare call: ```python import mlflow def _resolve_active_trace_id() -> str | None: """Return the current trace id, defensively. Tries the active span first (works inside a traced handler before the span closes). Falls back to the most recently completed trace on this thread (works after the function returns and the span auto-closes). Returns None only when there is genuinely no trace on this thread — in which case feedback should be disabled in the UI for this turn. """ span = mlflow.get_current_active_span() if span is not None: return span.trace_id try: from mlflow.tracing.fluent import get_last_active_trace_id return get_last_active_trace_id() except Exception: return None ``` ### Pattern A — Non-streaming `@invoke` (canonical, easiest) Track A Agent App handlers built with `@mlflow.genai.agent_server.invoke`: ```python import mlflow from mlflow.genai import agent_server @agent_server.invoke async def invoke(request, context): response_text = await run_agent(request) trace_id = _resolve_active_trace_id() if trace_id is None: raise RuntimeError("no active or recent trace; tracing not enabled") return { "output_text": response_text, "trace_id": trace_id, } ``` The frontend reads `data.trace_id` from the JSON response and stores it alongside the assistant message. ### Pattern B — Streaming `@stream` (SSE) The trace id is only complete **after** the stream finishes. Send it as the **last** SSE event, with a distinct event `type`: ```python import json import mlflow from mlflow.genai import agent_server @agent_server.stream async def stream(request, context): with mlflow.start_span(name="agent_turn") as span: async for token in run_agent_streaming(request): yield f"data: {json.dumps({'type': 'token', 'content': token})}\n\n" trace_id = span.trace_id or _resolve_active_trace_id() yield f"data: {json.dumps({'type': 'done', 'trace_id': trace_id})}\n\n" ``` Frontend handling (sketch): ```javascript for (const line of lines) { if (!line.startsWith("data: ")) continue; const event = JSON.parse(line.slice(6)); if (event.type === "token") appendToken(event.content); else if (event.type === "done") setTraceId(event.trace_id); // enables 👍/👎 buttons else if (event.type === "error") showError(event.error); } ``` The feedback buttons stay disabled until `trace_id` arrives. See [trace-context-patterns.md](../../tracks/A-custom-agent-apps/02-agent-framework/references/trace-context-patterns.md) for the full streaming pattern. ### Pattern C — `client_request_id` correlation Frontend generates a UUID per request, sends it on **both** `/chat` and `/feedback`. Backend tags the trace: ```python import uuid, mlflow @agent_server.invoke async def invoke(request, context): client_request_id = request.get("client_request_id") or str(uuid.uuid4()) mlflow.update_current_trace(client_request_id=client_request_id) response_text = await run_agent(request) return {"output_text": response_text, "client_request_id": client_request_id} ``` On the feedback side, look up the trace by `client_request_id` first: ```python traces = mlflow.search_traces( filter_string=f"attributes.client_request_id = '{client_request_id}'", max_results=1, ) trace_id = traces[0].info.trace_id ``` Use this pattern when the frontend has its own request-id discipline (typed analytics, Sentry breadcrumbs, etc.) that should be the system of record. --- ## Step 2 — Backend feedback route (Python) This is the **canonical write-path**. Same code regardless of where it runs (Track A Agent App, AppKit Python sidecar, FastAPI on Model Serving sidecar): ```python from typing import Optional from fastapi import APIRouter, Header, Query, Request from pydantic import BaseModel import mlflow from mlflow.entities import AssessmentSource router = APIRouter() class FeedbackBody(BaseModel): is_positive: bool # True = thumbs up rationale: Optional[str] = None # free-form comment @router.post("/feedback") def submit_feedback( body: FeedbackBody, trace_id: str = Query(..., description="Trace id returned by /chat"), request: Request, ): user_id = _resolve_user_id(dict(request.headers)) mlflow.log_feedback( trace_id=trace_id, name="user_feedback", value=body.is_positive, rationale=body.rationale, source=AssessmentSource( source_type="HUMAN", source_id=user_id, ), ) return {"status": "ok", "trace_id": trace_id} ``` `_resolve_user_id` resolves the originating end-user identity from inbound headers, with a clear priority order so 2-Apps Pathway-C deployments do not attribute feedback to the AppKit service principal: ```python from databricks.sdk import WorkspaceClient from databricks_app.utils import get_user_workspace_client def _resolve_user_id(headers: dict[str, str]) -> str: """Resolve the originating end-user's identity. Priority order (high → low): 1. x-app-user-email — set by an AppKit Pathway-C proxy (skill 06d) when the inbound Bearer is the AppKit SP. 2. x-forwarded-email — set by the Apps platform on direct end-user requests (1-App pathway). 3. x-forwarded-preferred-username — fallback when email is missing. 4. OBO -> current_user.me() — works only when the inbound Bearer is the user's own OBO token, NOT an SP. 5. "anonymous" — last resort, breaks per-user dashboards. """