04c-end-user-feedback

name	04c-end-user-feedback
description	>

End-user feedback collection (production)

Canonical reference for collecting end-user feedback in production and writing it back as MLflow Assessments on the originating trace. Pair with Track A 02-agent-framework (which owns the tracing setup that produces the trace_id) and 04-evaluation-runs (which owns expert / labeling-session feedback).

Upstream Lineage

This skill references Databricks Agent Skills' databricks-mlflow-evaluation skill for feedback-to-dataset, production trace analysis, and evaluation-loop guidance. If collected feedback needs to become labeled eval data or monitoring signal, consult the upstream skill first, then apply this skill's production feedback correlation and assessment write-path contracts.

This skill is the production user-feedback counterpart to 04-evaluation-runs:

Concern	Owner skill
End-user thumbs/rating from a live app → log_feedback on the trace	This skill (04c)
Expert labeling sessions, label schemas, Review App, session.sync()	04-evaluation-runs (Human feedback section)
Stakeholder business sign-off gate before deploy	04b-stakeholder-signoff
Periodic resampling of production traces into labeling sessions	07-production-monitoring
AppKit-side UI, Vote table, REST POST plumbing	apps_lakebase/skills/08-appkit-feedback

Canonical write-path (always): mlflow.log_feedback(trace_id=..., name="user_feedback", value=..., source=AssessmentSource(HUMAN, source_id=user_id), rationale=...). Whether that runs in your Track A Agent App's @invoke handler, a sidecar FastAPI route, or the AppKit 08-appkit-feedback REST proxy, the API surface is the same and the assessments land on the same trace.

---

When to Use

• You have a deployed agent (Track A on Databricks Apps, or Model Serving) that already emits MLflow traces.
• You want end users of a frontend (AppKit dashboard, template chat UI, Slack bot, etc.) to be able to give feedback (👍/👎, 1–5 stars, free-form comment) that lands on the originating trace.
• You need to return trace_id to the frontend so the user can later attach feedback to the same trace.
• You need streaming support — the trace id is only available after the SSE stream finishes; the UI must wait.
• You want to update or delete an assessment (user changed their mind, accidental thumb).
• You want to analyze collected feedback (positive rate, dimensional ratings, traces with feedback) for monitoring or to seed an evaluation dataset.

Skip if you only need expert / SME labeling — that is 04-evaluation-runs → Human feedback. Skip if you have no live users yet — instrument tracing first (Track A 02) and come back.

---

Architecture

End user clicks 👍 / 👎 / star rating / writes comment
        │
        ▼
Frontend calls POST /feedback
  body: { trace_id, value, rationale?, dimension? }
  auth: x-forwarded-access-token (OBO) — see Track A 04-authentication
        │
        ▼
Backend route (Track A Agent App or AppKit server)
  resolves user_id from x-app-user-email / x-forwarded-email first,
  then falls back to OBO current_user.me() only when the inbound Bearer is
  the user's own OBO token.
        │
        ▼
mlflow.log_feedback(
    trace_id=...,
    name="user_feedback",
    value=value,
    rationale=rationale,
    source=AssessmentSource(HUMAN, source_id=user_id),
)
        │
        ▼
Assessment attached to the trace in Unity Catalog (OTeL trace location)
        │
        ▼
Visible in:
  - MLflow Trace UI → Assessments panel
  - SQL over UC trace tables (analytics)
  - mlflow.search_traces() (dataset construction)
  - Production monitoring dashboards (Step 07)

The two correlation IDs you can use:

ID	Source	When to pick
trace_id	_resolve_active_trace_id() — wraps mlflow.get_current_active_span() with mlflow.tracing.fluent.get_last_active_trace_id() fallback	Default. Simplest; no extra plumbing. Works for non-streaming responses.
client_request_id	Frontend-generated UUID, passed in the request, attached to the trace via mlflow.update_current_trace(client_request_id=...)	Pick when you can't return trace_id synchronously (deeply async pipelines, WebSockets where the UI generates IDs first).

Both end up on the same trace; you choose which one the feedback POST carries.

---

Trace ID Contract: Two Forms

A trace_id you read off the wire and a trace_id you pass to the MLflow Assessments backend are not always the same string. Skills that confuse the two will quietly drop assessments. Document and exchange both forms explicitly.

Form 1 — Client / UI form (UC v4)

What the agent returns to the frontend (in the JSON body or the SSE done event) is the Unity Catalog v4 trace URI:

trace:/<catalog>.<schema>.<prefix>/<bare_id>

Example: trace:/main.skyloyalty_ops.agent_traces/0a1b2c3d4e5f.... This is the canonical, fully qualified handle the UI stores per assistant message and replays into POST /feedback. Storing the bare id alone is fragile because the same bare id can collide across catalogs / schemas. Always round-trip the full trace:/... URI.

Form 2 — Assessments backend form

The MLflow Assessments API (mlflow.log_feedback, mlflow.override_feedback, mlflow.delete_assessment) accepts whatever the target runtime requires. Concretely:

• On Databricks workspaces running MLflow 3.1+, the backend accepts the UC v4 URI directly.
• On older runtimes or self-hosted MLflow, the backend wants the bare id (<bare_id>) and resolves the trace from the configured experiment.

The feedback route MUST normalize the inbound id once, at the edge, before calling log_feedback. Centralize the conversion so the agent code never branches on runtime version inline:

def to_assessments_id(trace_uri_or_id: str) -> str:
    """Convert UC v4 client form to whatever the Assessments backend expects."""
    if trace_uri_or_id.startswith("trace:/"):
        # Modern Databricks runtimes accept the URI as-is.
        # If your runtime requires a bare id, strip to the segment after the last '/'.
        return trace_uri_or_id
    return trace_uri_or_id  # already bare

Document at the top of the feedback route which form your runtime accepts. Cross-reference Track A 02-agent-framework for the producer side that emits the URI.

---

Trace assessment round-trip gate

Before declaring feedback wired-up, exercise the full assessment lifecycle against a single trace and verify the result reads back from the SQL warehouse. Run the gate against a known trace from a real /chat round trip:

python genai-agents/sdlc/04c-end-user-feedback/scripts/feedback_round_trip.py \
    --trace-id "$KNOWN_GOOD_TRACE_ID" \
    --user-id "$EXPECTED_USER_EMAIL" \
    --assessments-table "$MLFLOW_TRACING_TABLE_PREFIX"_assessments \
    --warehouse-id "$MLFLOW_TRACING_SQL_WAREHOUSE_ID"

The script exercises log_feedback → override_feedback → delete_assessment → re-log and verifies via the SQL warehouse that:

1. log_feedback returns a non-empty assessment_id.
2. override_feedback preserves assessment_id.
3. delete_assessment succeeds.
4. The re-log mints a fresh assessment_id distinct from the deleted one.
5. The latest warehouse row matches the re-log id and the deleted id does not leak back.

Wire this into the same CI step that runs the dataset / scorer smoke tests. Required env for the warehouse verify: MLFLOW_TRACING_SQL_WAREHOUSE_ID, MLFLOW_TRACING_TABLE_PREFIX, DATABRICKS_HOST, DATABRICKS_TOKEN. Missing warehouse inputs fail by default. Use --api-only only for a local smoke test that intentionally does not claim end-to-end verification.

If any step fails — log_feedback rejects the id, delete_assessment 404s, the re-log mints the same id, or the SQL warehouse omits or duplicates rows — the deployment is blocked. The two most common failures this gate catches:

1. The frontend stored the bare id instead of the UC v4 URI, so the backend can't resolve the trace.
2. The runtime accepts log_feedback but the warehouse lag means the row isn't queryable yet — the script's --warehouse-wait-seconds flag (default 10s) absorbs replication lag instead of debugging it post-launch.

---

Step 1 — Return trace_id from the agent response

The trace id must reach the frontend somehow. Pick one of three patterns based on your transport.

Defensive trace_id capture

Both mlflow.get_current_active_span() and mlflow.tracing.fluent.get_last_active_trace_id() are valid sources, but each can return None depending on when in the request lifecycle you call it. Use this helper everywhere instead of either bare call:

import mlflow

def _resolve_active_trace_id() -> str | None:
    """Return the current trace id, defensively.

    Tries the active span first (works inside a traced handler before the
    span closes). Falls back to the most recently completed trace on this
    thread (works after the function returns and the span auto-closes).
    Returns None only when there is genuinely no trace on this thread —
    in which case feedback should be disabled in the UI for this turn.
    """
    span = mlflow.get_current_active_span()
    if span is not None:
        return span.trace_id
    try:
        from mlflow.tracing.fluent import get_last_active_trace_id
        return get_last_active_trace_id()
    except Exception:
        return None

Pattern A — Non-streaming @invoke (canonical, easiest)

Track A Agent App handlers built with @mlflow.genai.agent_server.invoke:

import mlflow
from mlflow.genai import agent_server

@agent_server.invoke
async def invoke(request, context):
    response_text = await run_agent(request)
    trace_id = _resolve_active_trace_id()
    if trace_id is None:
        raise RuntimeError("no active or recent trace; tracing not enabled")
    return {
        "output_text": response_text,
        "trace_id": trace_id,
    }

The frontend reads data.trace_id from the JSON response and stores it alongside the assistant message.

Pattern B — Streaming @stream (SSE)

The trace id is only complete after the stream finishes. Send it as the last SSE event, with a distinct event type:

import json
import mlflow
from mlflow.genai import agent_server

@agent_server.stream
async def stream(request, context):
    with mlflow.start_span(name="agent_turn") as span:
        async for token in run_agent_streaming(request):
            yield f"data: {json.dumps({'type': 'token', 'content': token})}\n\n"
        trace_id = span.trace_id or _resolve_active_trace_id()
        yield f"data: {json.dumps({'type': 'done', 'trace_id': trace_id})}\n\n"

Frontend handling (sketch):

for (const line of lines) {
  if (!line.startsWith("data: ")) continue;
  const event = JSON.parse(line.slice(6));
  if (event.type === "token") appendToken(event.content);
  else if (event.type === "done") setTraceId(event.trace_id);  // enables 👍/👎 buttons
  else if (event.type === "error") showError(event.error);
}

The feedback buttons stay disabled until trace_id arrives. See trace-context-patterns.md for the full streaming pattern.

Pattern C — client_request_id correlation

Frontend generates a UUID per request, sends it on both /chat and /feedback. Backend tags the trace:

import uuid, mlflow

@agent_server.invoke
async def invoke(request, context):
    client_request_id = request.get("client_request_id") or str(uuid.uuid4())
    mlflow.update_current_trace(client_request_id=client_request_id)
    response_text = await run_agent(request)
    return {"output_text": response_text, "client_request_id": client_request_id}

On the feedback side, look up the trace by client_request_id first:

traces = mlflow.search_traces(
    filter_string=f"attributes.client_request_id = '{client_request_id}'",
    max_results=1,
)
trace_id = traces[0].info.trace_id

Use this pattern when the frontend has its own request-id discipline (typed analytics, Sentry breadcrumbs, etc.) that should be the system of record.

---

Step 2 — Backend feedback route (Python)

This is the canonical write-path. Same code regardless of where it runs (Track A Agent App, AppKit Python sidecar, FastAPI on Model Serving sidecar):

from typing import Optional
from fastapi import APIRouter, Header, Query, Request
from pydantic import BaseModel
import mlflow
from mlflow.entities import AssessmentSource

router = APIRouter()

class FeedbackBody(BaseModel):
    is_positive: bool                  # True = thumbs up
    rationale: Optional[str] = None    # free-form comment

@router.post("/feedback")
def submit_feedback(
    body: FeedbackBody,
    trace_id: str = Query(..., description="Trace id returned by /chat"),
    request: Request,
):
    user_id = _resolve_user_id(dict(request.headers))
    mlflow.log_feedback(
        trace_id=trace_id,
        name="user_feedback",
        value=body.is_positive,
        rationale=body.rationale,
        source=AssessmentSource(
            source_type="HUMAN",
            source_id=user_id,
        ),
    )
    return {"status": "ok", "trace_id": trace_id}

_resolve_user_id resolves the originating end-user identity from inbound headers, with a clear priority order so 2-Apps Pathway-C deployments do not attribute feedback to the AppKit service principal:

from databricks.sdk import WorkspaceClient
from databricks_app.utils import get_user_workspace_client

def _resolve_user_id(headers: dict[str, str]) -> str:
    """Resolve the originating end-user's identity.

    Priority order (high → low):

      1. x-app-user-email          — set by an AppKit Pathway-C proxy (skill 06d)
                                     when the inbound Bearer is the AppKit SP.
      2. x-forwarded-email         — set by the Apps platform on direct
                                     end-user requests (1-App pathway).
      3. x-forwarded-preferred-username — fallback when email is missing.
      4. OBO -> current_user.me()  — works only when the inbound Bearer is
                                     the user's own OBO token, NOT an SP.
      5. "anonymous"               — last resort, breaks per-user dashboards.
    """