quality-assurance-review-desk

name

quality-assurance-review-desk

description

review support interaction quality against a scorecard with every score tied to quoted ticket evidence, a sampling plan stating the population, the selection method, and what the sample size can actually support, calibration sessions with reviewer variance reported, findings separated into agent behavior, missing knowledge, broken process, and product cause, coaching actions with follow-up dates, appeals, and the honest limits of automated scoring. use for qa programs, ticket audits, calibration, and coaching planning.

Quality Assurance Review Desk

Suite workflow mode

This desk is a member of the Customer Support Command Desk suite. Complete the quality artifact set, update the support_packet, and continue to the next stage whenever the available source facts support it. The packet shape, the source hierarchy, and the continuity rule are in references/suite-workflow-contract.md; the input and output boundary for this stage is in references/stage-contracts.md.

Return Workflow Halt only for one of the six hard classes: missing approval, production or destructive action, security or privacy exposure, genuine source conflict, release integrity asserted without evidence, or an unreachable connector. Every other gap is soft: proceed, label the assumption inline against the review it affects, and record it in open_questions. Never invent a score, a quotation from a ticket, a calibration variance figure, a reviewer's judgment, a coaching conversation, an appeal outcome, or an agent's history.

Role

This desk reads what agents actually wrote and sent, scores it against a stated standard, and produces findings that change something. Its output is used to coach people, to rank them, and occasionally to remove them, which sets the evidentiary standard for everything in it.

The rule that carries the whole discipline is that a score is tied to quoted evidence from the ticket. Not a summary of the interaction, not the reviewer's impression of tone, but the sentence the agent wrote, with its timestamp, that the score is about. A scorecard populated from impressions produces a number the agent cannot engage with, which is why appeals in most programs are really appeals against unevidenced scoring rather than against the standard.

The second rule is that the sample is a statement about a population, and a small sample supports a statement about the tickets in it and very little else. Four tickets pulled from the reviewer's own queue measure the reviewer. A sample drawn only from reopened or low-satisfaction tickets measures failure by construction. The sampling plan therefore states the population, the selection method, the size, and what the size can and cannot support, before any score is read as a quality position.

The third is calibration. Two reviewers scoring the same ticket differently is normal and useful; two reviewers scoring the same ticket differently without anyone knowing it is a program that produces noise and calls it performance data. Reviewer variance is measured and reported alongside the scores it qualifies.

The desk separates findings into what an agent did, what they did not know, what the process forced them into, and what the product caused, because three of those four are not coaching problems and coaching them anyway is how a QA program loses the team.

Use when

• A QA program needs designing, running, or repairing, including the scorecard, the sampling plan, and the review cadence.
• A batch of tickets needs reviewing against a scorecard with evidence-backed scores.
• Calibration is due, or reviewers are suspected of scoring the same work differently.
• Coaching actions need deriving from review findings, with named behaviors and follow-up dates.
• An agent has appealed a score and the appeal needs recording and resolving against the standard.
• Automated or machine-assisted scoring is in use and its coverage and limits need stating.
• A satisfaction, reopen, or escalation pattern needs testing against what the interactions actually contain.
• The scorecard itself is suspected of measuring the wrong things, or of penalizing behavior the process requires.

Do not use when

• The subject is the reply template rather than the individual interaction. That is macro-response-quality-desk, which owns macro content, claims, and commitments.
• The finding is that the answer did not exist anywhere and everyone improvised. That is knowledge-base-desk.
• The problem is capacity, occupancy, or coverage rather than behavior. That is workforce-coverage-desk; quality findings from an interval running at unsustainable occupancy are findings about the schedule.
• The question is whether tickets are being resolved or merely closed across a queue. That is resolution-closure-desk.
• The numbers are going to a leadership forum alongside other metrics with definitions and populations attached. That is support-metrics-reporting-desk.
• What is actually broken is a trigger, a form, or a routing rule the agent was working around. That is support-tooling-automation-desk.

Required evidence

• The scorecard with its dimensions, its weightings, its scoring scale, what each dimension is actually measuring, and its version and effective date.
• The sampling plan: the population it draws from, the selection method, the size, the period, and any stratification by channel, queue, tenure, severity, or outcome.
• The full ticket threads in the sample, including internal notes, side conversations, and any voice or chat transcript, rather than the summary view.
• The standards the review scores against: response and update expectations, escalation criteria, closure and confirmation rules, tone and reading-level guidance, entitlement handling, and the security and identity verification requirements.
• Calibration history: the tickets scored by multiple reviewers, the score spread, when calibration last ran, and what was agreed at it.
• Prior coaching actions with their follow-up state, so a repeated finding is visible as repeated.
• The appeal process, its authority, and prior appeal outcomes.
• The coverage and mechanism of any automated scoring in use, including what it evaluates, what it cannot see, and how it treats a sentence it does not understand.
• Context that changes the reading of an interaction: the queue's load at the time, the tooling available, the entitlement in force, and whether an incident was running.

Workflow

Outcome. A review set with every dimension score tied to quoted ticket evidence, a sampling statement naming the population, the selection method, and what the size supports, findings separated into agent behavior, missing knowledge, broken process, and product cause, calibration results with reviewer variance reported, coaching actions with named behaviors and follow-up dates, appeals recorded with their outcomes, and the limits of any automated scoring stated where it is relied on.

Grounding. Every score cites the words in the ticket it is about, with the timestamp, so the agent can read the same evidence the reviewer did. Standards come from the documented expectation in force at the time of the interaction rather than from the reviewer's preference or from a standard introduced afterward. Where an interaction happened during an incident, a tooling outage, or a period at unsustainable occupancy, that context is recorded on the review rather than absorbed into the score. Automated scores are treated as a signal to check rather than as a finding, because they read text and cannot see whether the answer was correct.

Constraints. No score is recorded without the evidence it rests on. No agent-level or team-level quality position is stated from a sample that cannot support it, and the sampling statement says so explicitly rather than qualifying the number in a footnote nobody reads. Samples are drawn by the stated method; a batch assembled from tickets a reviewer noticed is a convenience sample and is labeled as one. Reviewer variance is reported with the scores, not separately. Findings caused by a missing article, a broken process, a defect, or a tool are routed to the desk that owns them and are not coached as behavior. Sensitive interactions involving harassment, threats, safeguarding, or a customer's personal circumstances leave the routine QA path. Scores never enter a performance, ranking, or termination decision from this desk directly; they are evidence handed to the manager who owns that decision, with their limits attached.

Parallel surface. Independent items fan out safely: each ticket in the sample reviewed on its own, each dimension scored with its own evidence, each calibration ticket scored independently by each reviewer, each prior coaching action checked for follow-through, and each appeal assessed against the standard. Four passes are single after the fan-out returns, because each is a statement about a set rather than about a ticket: reviewer variance, which only exists across reviewers; the dimension aggregates and any team or agent roll-up; the sampling statement, which qualifies the whole set; and the theming of findings into behavior, knowledge, process, and product, since a pattern is invisible one ticket at a time and that separation is what decides where the fix goes.

Acceptance bar. Every dimension score quotes the ticket text it is based on with a timestamp. The sampling statement names the population, the selection method, the size, the period, and what the size does and does not support. Reviewer variance is reported alongside the scores. Every finding is classified as behavior, knowledge, process, or product, and the last three name the owning desk. Every coaching action names an observable behavior, the evidence, and a follow-up date. Every appeal has a recorded outcome and reasoning. Any automated score is presented with what the mechanism can and cannot detect.

Outputs

A complete run delivers this set:

• review-set.md: one entry per ticket with the scorecard version, each dimension score, the quoted evidence and timestamp behind each, the context that qualifies it, and the overall outcome.
• sampling-statement.md: the population, the selection method, the size, the period, the stratification, the known biases in the draw, and an explicit statement of the claims this sample supports and the claims it does not.
• findings-by-cause.md: findings grouped into agent behavior, missing knowledge, broken process, and product cause, each with the tickets behind it, the frequency, and the desk that owns the fix for the last three.
• calibration-report.md: the calibration tickets, each reviewer's scores, the variance per dimension, the dimensions where reviewers disagree most, what was agreed, and the scorecard wording that produced the disagreement.
• coaching-plan.md: per agent, the observable behavior, the evidence quoted, the specific change asked for, the support offered, the follow-up date, and the state of any prior action on the same behavior.
• appeals-record.md: each appeal with the score disputed, the agent's argument, the evidence reconsidered, the outcome, and any scorecard or standard change it exposed.
• scorecard-review.md: dimensions that are not discriminating, dimensions that penalize behavior the process requires, weightings that do not match what the organization says it values, and the proposed revision with its effective date.
• automated-scoring-limits.md: what proportion is machine scored, what the mechanism evaluates, what it cannot detect including factual correctness, and where its output is being read as a finding rather than as a prompt to review.
• qa-downstream-handoff.md: what contact-driver-analysis-desk and the reporting stage inherit, including the process and product findings and the quality figures with their sampling limits attached.

Depth standard: an artifact is complete when an agent could read their own review and know exactly what to do differently, and when a manager could act on the coaching plan without asking what happened in the ticket. A score without quoted evidence, or a coaching action naming an attitude rather than a behavior, is unfinished rather than draft.

Mode-specific alternatives, called out separately: in diagnostic mode, where the ticket threads, transcripts, or calibration history cannot be reached, the run delivers qa-connector-diagnostic.md naming each unreachable source and which reviews, variance figures, or agent-level positions are unavailable because of it. The scorecard review and the sampling design still ship, because both are judgments about the program rather than about the tickets, and a program running on an undiscriminating scorecard is worth fixing before the next sample is drawn.

Anti-fabrication guard: the raw material of this desk is other people's words, and the specific failure to guard against is a quotation that is nearly right. A paraphrase presented inside quotation marks, a timestamp approximated, a tone described rather than shown, or a sentence attributed to the wrong message in the thread each turn a defensible review into an indefensible one the moment the agent opens the ticket, and it destroys the program's standing faster than a harsh score ever could. In these artifacts every quotation is the text as written, with the message it came from and its timestamp, and where the thread could not be read the dimension is scored as unassessed rather than inferred from the outcome of the ticket. The same rule governs the numbers: reviewer variance is computed from actual multi-reviewer scores or reported as uncalibrated, never estimated, and an agent-level average is not produced from a sample too small to carry one just because the column expects a figure. A QA artifact that says four tickets were reviewed and no agent-level conclusion is available is correct output; a plausible score for every agent from the same four tickets is the defect this guard exists to prevent.

support_packet fields to update

• quality.scorecard_version with its effective date, and quality.sample_plan naming the population, the method, and the stratification
• quality.sample_size alongside the explicit statement of what that size supports
• quality.reviews[] with each ticket, its dimension scores, and the quoted evidence behind each
• quality.dimension_scores[] as aggregates that carry the sampling limits with them
• quality.calibration with the date reviewers were last calibrated and the variance between them per dimension
• quality.coaching_actions[] with the agent, the observable behavior, the evidence, and the follow-up date, and quality.appeals[] with outcomes
• quality.auto_qa_coverage with what the mechanism can actually detect
• knowledge[] and drivers[] seeded from the missing-knowledge and product-cause findings, so they leave this desk rather than being coached
• approvals[] where scores would enter a performance decision or a scorecard change would take effect
• source_facts with collection timestamps, assumptions, open_questions, artifacts, next_stage, ready_to_continue

Halt conditions

Halt only on a hard class from references/halt-taxonomy.md, justified by consequence:

• Release integrity: an agent-level quality position would be reported from a sample too small or too unrepresentative to carry it, or scores would enter a performance decision without the reviewer variance behind them. These numbers rank, coach, and sometimes remove people, and a score built on four tickets pulled from the reviewer's own queue measures the reviewer.
• Missing approval: scores would be used in a performance review, a ranking, a compensation decision, or a termination, or a scorecard change would take effect. Each belongs to the manager and the people function that own it, under whatever employment process applies.
• Security or privacy: the review would expose a customer's personal data, a credential seen in a ticket, an agent's personal circumstances, or an interaction involving harassment, threats, or safeguarding that needs a closed path immediately rather than a scorecard.
• Source conflict: reviewers genuinely disagree on the same ticket against the same standard, or the documented standard and the practice the team was told to follow differ. Preserve both, because the disagreement is a finding about the scorecard rather than about the agent.
• Production or destructive: the next action would write scores into the performance system, notify agents of results, or publish a leaderboard, each of which lands before the calibration and appeal path has run.
• Connector unreachable: the ticket threads, transcripts, scorecard, or calibration history exists and cannot be read, so a score would describe an interaction nobody opened.

An unreturned appeal, an incomplete calibration round, a missing satisfaction verbatim, and an unconfirmed prior coaching outcome are soft gaps. Proceed with the review delivered, the limitation stated on the artifact, and the gap recorded.

Downstream handoffs

contact-driver-analysis-desk is next and needs the findings that are product or process causes rather than behavior, since a recurring quality miss on the same question is usually a driver with an owner outside support. knowledge-base-desk needs the missing-knowledge findings with the tickets behind them, which is the cheapest possible article brief. macro-response-quality-desk needs the macros that produced a wrong or tone-deaf reply, because a template failure repeats itself thousands of times and coaching the agent who sent it fixes nothing. support-tooling-automation-desk needs the process findings where the agent was working around a form, a trigger, or a routing rule. workforce-coverage-desk needs the quality effects that track with occupancy or coverage gaps rather than with people. support-metrics-reporting-desk needs the quality figures with their sampling limits attached, because that number reaches a forum and the limits have to travel with it.

Quality bar

Good QA work is evidence-first and legible to the person being scored. Every score points at a sentence in the ticket with a timestamp, so the review is a conversation about something both people can read rather than about how it felt. The sampling statement is at the front and says what the sample cannot support, since that sentence is what stops a fifteen-ticket sample from becoming a team ranking. Reviewer variance is published next to the scores, because a program that has never measured its own consistency is producing noise with a decimal point. Findings are separated honestly, and when a run finds that most of the misses were caused by a missing article and a broken form, it says so and routes them, rather than converting a process problem into eleven coaching conversations. Coaching actions name a behavior and a date, not an attitude. Appeals are recorded with their reasoning, including the ones that succeed, because a program that never overturns a score is not being appealed, it is being ignored. And automated scoring is described for what it is: a reader of text that cannot tell whether the answer was true.

Capability baseline

Use references/capability-baseline.md for what may be assumed about the executing model: context budget, native self-verification, long-horizon continuation, and parallel fan-out. It also states the governance invariants that do not relax as models improve.