08-appkit-feedback

# Add User Feedback to an AppKit Chat Application Add thumbs up/down feedback on assistant responses, persisted in Lakebase and linked to MLflow traces via the Assessments REST API — all AppKit-native. Authentication uses `getExecutionContext().client.config.authenticate()` (the same pattern as [06-appkit-serving-wiring/references/custom-proxy-fallback.md](../06-appkit-serving-wiring/references/custom-proxy-fallback.md)), so there's no dependency on `process.env.DATABRICKS_TOKEN` or manual header parsing. > **Companion Python skill:** the canonical end-user feedback contract — what > `mlflow.log_feedback(...)` expects, trace-id vs `client_request_id`, > streaming, ratings, update/delete, and the negative-feedback → eval-dataset > loop — lives in > [`genai-agents/sdlc/04c-end-user-feedback`](../../../genai-agents/sdlc/04c-end-user-feedback/SKILL.md). > This AppKit skill is the **frontend / Node sidecar** wire-up; 04c is the > **Python / agent-side** contract. Naming, source format, and trace-id flow > here intentionally match 04c so dashboards aggregate cleanly. ## When to Use - Adding user feedback to an existing AppKit chat interface built from [07-appkit-chat-history](../07-appkit-chat-history/SKILL.md) - Connecting feedback to MLflow experiment traces for model evaluation - Building a feedback loop for agent quality monitoring **Prerequisites:** - Chat streaming + persistence from [07-appkit-chat-history](../07-appkit-chat-history/SKILL.md) (the `Vote` table, the `traceId` column on `chat.Message`, and the `messageMetaStore` ephemeral fallback are all produced there) - An MLflow experiment configured (optional — feedback still works without it, it just doesn't log to MLflow) - MLflow tracing enabled on the deployed agent endpoint (see Gotchas at end) ### Working in Genie Code (client routing) This skill is **source editing + server-side code** (feedback routes, MLflow assessment calls, UI). There is **no own DDL** — the `Vote` table comes from `07-appkit-chat-history` (server-side startup, SP-owned). The MLflow REST calls authenticate from the SP execution context, so they run identically on both clients. Only the CLI/local checks differ: | IDE/CLI (as written) | Genie Code substitution | |----------------------|--------------------------| | `databricks experiments create --name … --profile <PROFILE>` (Step 1) | run via `runDatabricksCli` (**omit `--profile`** — Genie Code injects the workspace + OAuth) | | `npm run build` gates | **IDE-only** convenience — no local Node toolchain; the platform builds **server-side** on deploy. Errors surface in `databricks apps logs <name>` | | local `curl http://localhost:8000/api/feedback …` tests (Step 4) | no local dev server — POST/GET the feedback routes on the **deployed** app via browser or the OAuth-session `requests.Session()` test in `03-appkit-deploy` | | `rg -n '(DATABRICKS_TOKEN\|…)' server/` (Step 4) | runs on both — Grep/`rg` over the cloned repo, no client difference | | `app.yaml` / `databricks.yml` `MLFLOW_EXPERIMENT_ID` edits (Step 1) | source edits — identical on both clients | | `databricks apps deploy …` | see the `03-appkit-deploy` deploy-routing contract (`runDatabricksCli`, else SDK `w.apps.deploy(... SNAPSHOT)`) | Paths are relative to `apps_lakebase/$APP_NAME` — inside your git-cloned workshop project (`artifact_root`) on Genie Code, never the read-only `.assistant/skills` copy and never `/tmp`. See `skills/genie-code-environment` for the full manifest. --- ## Header Contract (Canonical) The feedback handler reads `req.session.userId` (set by [07-appkit-chat-history](../07-appkit-chat-history/SKILL.md) Step 2 from the canonical Databricks Apps user headers) and submits it as `AssessmentSource.source_id`. Databricks Apps canonical user headers (inbound to AppKit): - `x-forwarded-email` - `x-forwarded-preferred-username` - `x-forwarded-user` - `x-forwarded-access-token` `x-forwarded-user-info` is **not** a canonical Databricks Apps header and must not be used. When this AppKit instance is the frontend half of a 2-Apps deployment (Pathway-C / Variant 4) and the agent runs in a separate Databricks App, the AppKit proxy must additionally set `x-app-user-email` on the outbound request to the Agent App so the agent can attribute its MLflow `log_feedback` calls to the originating user. The app-to-app `Authorization: Bearer` token represents the AppKit App service principal hop and **must not** be treated as the user identity. See [06d-appkit-agent-app-proxy](../06d-appkit-agent-app-proxy/SKILL.md) for the proxy-side contract. --- ## Architecture ``` User clicks 👍/👎 │ ├──► POST /api/feedback { chatId, messageId, isUpvoted } │ │ │ ├──► UPSERT to Lakebase chat.Vote table (always) │ │ │ ├──► Look up traceId from chat.Message (DB) or messageMetaStore (in-memory) │ │ │ └──► POST/PATCH Databricks MLflow Assessments REST API (if traceId available) │ /api/3.0/mlflow/traces/{traceId}/assessments │ Auth: AppKit getExecutionContext().client.config.authenticate() │ Logs: assessment_name="user_feedback", source={ HUMAN, userId }, feedback={ value } │ └──► UI updates thumbs button state ``` --- ## Step 1: Configure MLflow Experiment ### Environment Variables Add to `.env` for local development: ```env MLFLOW_EXPERIMENT_ID=your-experiment-id ``` Add to `app.yaml` for deployed apps: ```yaml env: - name: MLFLOW_EXPERIMENT_ID value: ${var.mlflow_experiment_id} ``` Add to `databricks.yml`: ```yaml variables: mlflow_experiment_id: description: "MLflow experiment ID for feedback tracking" ``` ### Creating an MLflow Experiment The feedback experiment MUST be pinned to the same user-and-use-case identity that backs `APP_NAME` so concurrent workshop attendees on a shared workspace never collide on a single experiment, and the MLflow UI never lists a generic `Default` / `Tracing` / `my-app-feedback` entry. **Naming rule (REQUIRED):** `/Users/<user_email>/mlflow/<APP_NAME>-feedback` — e.g. `/Users/jane.doe@example.com/mlflow/jane-d-stayfinder-feedback`. The leaf carries the same `${FIRSTNAME}-${LASTINITIAL}-${use_case_slug}` shape that derives `APP_NAME` (see `apps_lakebase/Instructions.md`). When running on top of `vibecoding-state`, this value is already pinned at `state://Resources.mlflow_feedback_experiment_path` by [`vibecoding-state.migrate_canonical`](../../../skills/vibecoding-state/SKILL.md#operation-migrate_canonical) — read it from state instead of inventing a new path. If you don't have an experiment yet, create one: ```bash # Replace <user_email> and <APP_NAME> with your actual values # (or pull mlflow_feedback_experiment_path directly from .vibecoding-state.md). databricks experiments create \ --name "/Users/<user_email>/mlflow/<APP_NAME>-feedback" \ --profile <PROFILE> ``` > **Client note — Genie Code:** run this through `runDatabricksCli` and **omit `--profile`** (the workspace + OAuth are injected). Capture the `experiment_id` from the JSON output the same way. Note the `experiment_id` from the output and set it in your environment. ### Enabling MLflow Tracing on the Agent Endpoint This is **not** an AppKit concern — it happens at agent deployment time. Without it, the endpoint won't return `trace_id` in its streaming chunks and feedback will report `mlflowStatus: "no_trace_id"`. For Databricks Agent Framework deployments: ```python import databricks.agents databricks.agents.deploy( model_name="catalog.schema.agent_model", model_version=1, enable_trace=True, # emits trace_id in databricks_output ) ``` If you're using a Mosaic AI Agent Evaluation endpoint, tracing is enabled by default. For more on the trace ID shape returned in each streaming chunk, see [references/trace-extraction.md](references/trace-extraction.md). --- ## Step 2: Add Feedback API Routes (AppKit-Native Auth) This is the only file that touches the MLflow REST API directly. The trick is to use AppKit's execution context for OAuth — no `process.env.DATABRICKS_TOKEN` fallback, no manual `x-forwarded-access-token` parsing. ```typescript import { getExecutionContext, AppKit } from "@databricks/appkit"; // Reuse or import the messageMetaStore created in 07-appkit-chat-history (Step 4b). // If you stored it on globalThis as __appkitMessageMetaStore: const messageMetaStore = (globalThis as { __appkitMessageMetaStore?: Map<string, { chatId: string; traceId: string | null }> }) .__appkitMessageMetaStore ?? new Map(); // Dedup: one assessment per (messageId, userId) — subsequent clicks PATCH const assessmentStore = new Map<string, string>(); // "messageId:userId" -> assessmentId // Cached AppKit-authenticated MLflow request helper. // Uses the app's Service Principal token by default. See the "User vs SP Auth" // note at the end of this step if you need user OBO for MLflow calls. async function mlflowRequest( path: string, init: { method: "POST" | "PATCH" | "GET"; body?: unknown }, ): Promise<Response> { const ctx = getExecutionContext(); const config = ctx.client.config; await config.ensureResolved(); const host = (config.host ?? "").replace(/\/$/, ""); if (!host) throw new Error("Databricks host not resolved from execution context"); const headers = new Headers(); await config.authenticate(headers); headers.set("Content-Type", "application/json"); headers.set("Accept", "application/json"); return fetch(`${host}${path}`, { method: init.method, headers, body: init.body ? JSON.stringify(init.body) : undefined, }); } AppKit.server.extend((app) => { // POST /api/feedback — submit or update feedback app.post("/api/feedback", async (req, res) => { const { chatId, messageId, isUpvoted } = req.body; const forwardedEmail = req.headers["x-forwarded-email"]; const userId = typeof forwardedEmail === "string" && forwardedEmail.length > 0 ? forwardedEmail : req.session!.userId; if (!chatId || !messageId || typeof isUpvoted !== "boolean") { return res .status(400) .json({ error: "chatId, messageId, and isUpvoted (boolean) required" }); } // 1. Persist vote in Lakebase try { await AppKit.lakebase.query( `INSERT INTO chat."Vote" ("chatId", "messageId", "isUpvoted") VALUES ($1, $2, $3) ON CONFLICT ("chatId", "messageId") DO UPDATE SET "isUpvoted" = $3`, [chatId, messageId, isUpvoted], ); } catch (err) { console.error("[Feedback] Failed to save vote:", err); return res.status(500).json({ error: "Failed to save feedback" }); } // 2. Resolve traceId (DB first, fall back to in-memory meta store) let traceId: string | null = null; try { const result = await AppKit.lakebase.query( `SELECT "traceId" FROM chat."Message" WHERE id = $1`, [messageId], ); traceId = result.rows[0]?.traceId ?? null; } catch (err) { console.warn("[Feedback] DB lookup failed, checking meta store:", err); } if (!traceId) { const meta = messageMetaStore.get(messageId); if (meta?.traceId) traceId = meta.traceId; } // 3. Log to MLflow Assessments API (if trace ID + experiment available) let mlflowStatus: string = "skipped"; let mlflowError: string | undefined; const experimentId = process.env.MLFLOW_EXPERIMENT_ID; if (traceId && experimentId) { try { const dedupeKey = `${messageId}:${userId}`; const existingAssessmentId = assessmentStore.get(dedupeKey); if (existingAssessmentId) { // Update existing assessment via PATCH const mlflowResponse = await mlflowRequest( `/api/3.0/mlflow/traces/${traceId}/assessments/${existingAssessmentId}`, { method: "PATCH", body: { feedback: { value: isUpvoted } }, }, ); if (mlflowResponse.ok) { mlflowStatus = "updated"; } else { const errBody = await mlflowResponse.text(); console.warn( "[Feedback] MLflow PATCH error:", mlflowResponse.status, errBody, ); mlflowStatus = "mlflow_error"; mlflowError = `${mlflowResponse.status}: ${errBody.slice(0, 200)}`; } } else { // Create new assessment via POST const mlflowResponse = await mlflowRequest( `/api/3.0/mlflow/traces/${traceId}/assessments`, { method: "POST", body: { assessment_name: "user_feedback", source: { source_type: "HUMAN", source_id: userId }, feedback: { value: isUpvoted }, }, }, ); if (mlflowResponse.ok) { const body = await mlflowResponse.json(); const assessmentId = body?.assessment?.assessment_id; if (assessmentId) assessmentStore.set(dedupeKey, assessmentId); mlflowStatus = "logged"; } else { const errBody = await mlflowResponse.text(); console.warn( "[Feedback] MLflow POST error:", mlflowResponse.status, errBody, ); mlflowStatus = "mlflow_error"; mlflowError = `${mlflowResponse.status}: ${errBody.slice(0, 200)}`; } } } catch (err) { console.warn("[Feedback] MLflow API call failed:", err);