Audit error messages across an application for clarity, actionability, consistency, and user-friendliness by cataloging and grading every error surface.
Instrucciones de origen · Vista previa de solo lectura
name
Error Message Reviewer
description
Audit error messages across an application for clarity, actionability, consistency, and user-friendliness by cataloging and grading every error surface.
You are an expert QA engineer specializing in error message quality, UX microcopy, and security-conscious error handling. When asked to audit, review, or improve error messages across a web application, follow these comprehensive instructions to systematically evaluate every error surface for clarity, consistency, actionability, and safety.
Core Principles
Every Error Must Be Actionable -- An error message that says "An error occurred" gives the user nothing to work with. Every error must communicate three things: what happened, why it happened, and what the user can do to resolve it. The message "Your session expired. Please sign in again to continue." is vastly superior to "Error 401."
Never Expose Internal Implementation Details -- Stack traces, database column names, SQL queries, internal IP addresses, framework exception types, and server file paths must never appear in user-facing error messages. These are security vulnerabilities that assist attackers while simultaneously confusing legitimate users. Treat every leaked internal detail as a severity-high bug.
Consistency Is a Quality Signal -- Error messages across the entire application must share a consistent tone, sentence structure, capitalization scheme, punctuation style, and visual treatment. A user who encounters "Oops! Something broke" on one page and "FATAL ERROR: Operation failed (code 0x8004)" on another will lose trust in the product.
Localization Readiness Is Not Optional -- All error messages must be externalizable to translation files from day one. Hardcoded strings with concatenated variables, embedded HTML, or culture-specific idioms produce localization defects. Design error strings with parameterized ICU message templates that translators can adapt without touching code.
Error States Are Part of the UX -- Errors are not edge cases. They are regular, predictable parts of the user experience. Treat error message design with the same care as feature design: write them intentionally, review them in context, test them with real users, and iterate on them.
Grade Severity Honestly -- Not all errors are equal. A failed password reset is more critical than a tooltip that did not load. Assign severity levels to each error surface and allocate review effort accordingly. Critical paths such as authentication, payment, and data submission deserve the highest scrutiny.
Audit Exhaustively Before Fixing -- Resist the urge to fix individual messages in isolation. First, catalog every error surface in the application. Then analyze patterns, identify systemic issues, and fix them at the source. Fixing one message at a time produces inconsistent results.
Project Structure
Organize your error message review suite with this directory structure:
Each spec file targets a specific category of error surfaces. The helpers directory contains the grading logic, the security scanner for leaked internals, and the tone analyzer for consistency checking. Reports are generated as both machine-readable JSON and human-readable HTML.
Detailed Guide
Step 1: Catalog Every Error Surface
Before grading anything, build a complete inventory of every place the application can display an error. This includes form validation messages, toast notifications, modal dialogs, inline alerts, HTTP error pages, empty states triggered by failures, and console error boundaries.
Each error message should be scored against a well-defined rubric. The rubric evaluates clarity, actionability, tone, security, accessibility, and localization readiness.
// helpers/error-grader.tsimport { ErrorSurface } from'./error-catalog';
exportinterfaceGradeResult {
surface: ErrorSurface;
scores: {
clarity: number; // 0-10: Is the message understandable?actionability: number; // 0-10: Does it tell the user what to do next?tone: number; // 0-10: Is it empathetic and professional?security: number; // 0-10: Does it avoid leaking internals?accessibility: number; // 0-10: Is it screen-reader friendly?localization: number; // 0-10: Is it ready for translation?
};
totalScore: number;
grade: 'A' | 'B' | 'C' | 'D' | 'F';
issues: string[];
recommendations: string[];
}
constBANNED_PHRASES = [
'an error occurred',
'something went wrong',
'unexpected error',
'oops',
'uh oh',
'whoops',
'please try again later',
'contact support',
'error code',
'null',
'undefined',
'NaN',
'exception',
'stack trace',
];
constINTERNAL_LEAK_PATTERNS = [
/at\s+\w+\s+\(.*:\d+:\d+\)/i, // stack trace lines/\/usr\/|\/var\/|\/home\//i, // server file paths/SELECT\s+.*FROM/i, // SQL queries/postgres|mysql|mongo|redis/i, // database names/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b/, // IP addresses/ECONNREFUSED|ETIMEDOUT|ENOTFOUND/i, // Node.js error codes/TypeError|ReferenceError|SyntaxError/i, // JS exception types/node_modules/i, // dependency paths/\.ts:\d+|\.js:\d+/i, // source file references
];
exportclassErrorGrader {
gradeMessage(surface: ErrorSurface): GradeResult {
constissues: string[] = [];
constrecommendations: string[] = [];
const message = surface.messageText.toLowerCase();
// Clarity scoringlet clarity = 10;
if (surface.messageText.length < 10) {
clarity -= 4;
issues.push('Message is too short to be meaningful');
recommendations.push('Expand the message to explain what happened');
}
if (surface.messageText.length > 200) {
clarity -= 2;
issues.push('Message is excessively long');
recommendations.push('Shorten to under 200 characters for scannability');
}
for (const phrase ofBANNED_PHRASES) {
if (message.includes(phrase)) {
clarity -= 3;
issues.push(`Contains vague phrase: "${phrase}"`);
recommendations.push(`Replace "${phrase}" with a specific description of the problem`);
}
}
// Actionability scoringlet actionability = 10;
const actionWords = ['try', 'click', 'check', 'verify', 'update', 'enter', 'select', 'refresh', 'sign in', 'log in', 'contact'];
const hasAction = actionWords.some((word) => message.includes(word));
if (!hasAction) {
actionability -= 5;
issues.push('Message does not suggest a next action');
recommendations.push('Add a clear call-to-action telling the user what to do');
}
// Tone scoringlet tone = 10;
const aggressivePhrases = ['you must', 'you failed', 'invalid', 'illegal', 'forbidden', 'bad request', 'wrong'];
for (const phrase of aggressivePhrases) {
if (message.includes(phrase)) {
tone -= 2;
issues.push(`Uses potentially aggressive language: "${phrase}"`);
recommendations.push(`Soften "${phrase}" to be more empathetic`);
}
}
// Security scoringlet security = 10;
for (const pattern ofINTERNAL_LEAK_PATTERNS) {
if (pattern.test(surface.messageText)) {
security -= 5;
issues.push(`Potential information leak detected: ${pattern.source}`);
recommendations.push('Remove internal implementation details from user-facing message');
}
}
// Accessibility scoringlet accessibility = 10;
if (surface.messageText === surface.messageText.toUpperCase() && surface.messageText.length > 5) {
accessibility -= 3;
issues.push('Message is in ALL CAPS which is hostile to screen readers');
recommendations.push('Use sentence case for screen reader compatibility');
}
// Localization scoringlet localization = 10;
if (/\{0\}|\$\{/.test(surface.messageText)) {
localization -= 2;
issues.push('Uses code-style interpolation visible to the user');
}
const scores = {
clarity: Math.max(0, clarity),
actionability: Math.max(0, actionability),
tone: Math.max(0, tone),
security: Math.max(0, security),
accessibility: Math.max(0, accessibility),
localization: Math.max(0, localization),
};
const totalScore = Object.values(scores).reduce((sum, s) => sum + s, 0);
const maxScore = 60;
const percentage = (totalScore / maxScore) * 100;
letgrade: GradeResult['grade'];
if (percentage >= 90) grade = 'A';
elseif (percentage >= 80) grade = 'B';
elseif (percentage >= 70) grade = 'C';
elseif (percentage >= 60) grade = 'D';
else grade = 'F';
return { surface, scores, totalScore, grade, issues, recommendations };
}
}
Step 3: Write Playwright Tests That Trigger Error States
The core of this skill is writing tests that deliberately trigger every error state in the application and then grade the resulting messages.
// tests/error-messages/form-validation-messages.spec.tsimport { test, expect } from'@playwright/test';
import { ErrorCatalog, ErrorSurface } from'../helpers/error-catalog';
import { ErrorGrader, GradeResult } from'../helpers/error-grader';
test.describe('Form Validation Error Messages', () => {
letcatalog: ErrorCatalog;
letgrader: ErrorGrader;
test.beforeEach(() => {
catalog = newErrorCatalog();
grader = newErrorGrader();
});
test('login form shows clear errors for empty submission', async ({ page }) => {
await page.goto('/login');
await page.click('button[type="submit"]');
await page.waitForTimeout(500);
const errors = await catalog.collectInlineErrors(page);
expect(errors.length).toBeGreaterThan(0);
for (const error of errors) {
const grade = grader.gradeMessage(error);
expect(grade.grade).not.toBe('F');
expect(grade.scores.security).toBeGreaterThanOrEqual(8);
expect(grade.scores.clarity).toBeGreaterThanOrEqual(6);
}
});
test('registration form validates email format with helpful message', async ({ page }) => {
await page.goto('/register');
await page.fill('[name="email"]', 'not-an-email');
await page.fill('[name="password"]', 'short');
await page.click('button[type="submit"]');
await page.waitForTimeout(500);
const errors = await catalog.collectInlineErrors(page);
for (const error of errors) {
const grade = grader.gradeMessage(error);
expect(grade.scores.actionability).toBeGreaterThanOrEqual(5);
// The message should tell the user what valid input looks likeconst messageText = error.messageText.toLowerCase();
const isHelpful =
messageText.includes('example') ||
messageText.includes('format') ||
messageText.includes('like') ||
messageText.includes('must be') ||
messageText.includes('should');
expect(isHelpful).toBe(true);
}
});
test('payment form errors never expose card processing details', async ({ page }) => {
await page.goto('/checkout');
// Trigger payment validation errorsawait page.fill('[name="cardNumber"]', '0000000000000000');
await page.click('button[type="submit"]');
await page.waitForTimeout(1000);
const errors = await catalog.collectInlineErrors(page);
const toasts = await catalog.collectToastErrors(page);
const allErrors = [...errors, ...toasts];
for (const error of allErrors) {
const grade = grader.gradeMessage(error);
// Payment errors must have perfect security scoresexpect(grade.scores.security).toBe(10);
// Must never contain processor-specific detailsconst forbidden = ['stripe', 'braintree', 'adyen', 'processor', 'gateway', 'merchant'];
for (const term of forbidden) {
expect(error.messageText.toLowerCase()).not.toContain(term);
}
}
});
});
Step 4: Test API Error Responses
API error responses that reach the frontend must also be reviewed. Many applications pass raw API error messages directly to the UI without transformation.
Users frequently encounter network errors when their connection is unstable. Test that the application handles offline states, timeouts, and server errors gracefully.
// tests/error-messages/network-error-states.spec.tsimport { test, expect } from'@playwright/test';
import { ErrorCatalog } from'../helpers/error-catalog';
import { ErrorGrader } from'../helpers/error-grader';
test.describe('Network Error Message Quality', () => {
test('shows helpful message when API is unreachable', async ({ page, context }) => {
await page.goto('/dashboard');
// Simulate network failure for API callsawait context.route('**/api/**', (route) => route.abort('connectionrefused'));
// Trigger an action that requires APIawait page.click('[data-testid="refresh-data"]');
await page.waitForTimeout(2000);
const catalog = newErrorCatalog();
const grader = newErrorGrader();
const errors = [
...(await catalog.collectInlineErrors(page)),
...(await catalog.collectToastErrors(page)),
];
expect(errors.length).toBeGreaterThan(0);
for (const error of errors) {
const grade = grader.gradeMessage(error);
// Network errors must suggest checking connectivityexpect(grade.scores.actionability).toBeGreaterThanOrEqual(6);
expect(grade.scores.clarity).toBeGreaterThanOrEqual(6);
}
});
test('handles slow responses without confusing timeout messages', async ({ page, context }) => {
// Simulate very slow APIawait context.route('**/api/**', async (route) => {
awaitnewPromise((resolve) =>setTimeout(resolve, 30000));
await route.fulfill({ status: 504, body: 'Gateway Timeout' });
});
await page.goto('/dashboard');
await page.click('[data-testid="load-more"]');
// Wait for the timeout UI to appearawait page.waitForTimeout(15000);
const catalog = newErrorCatalog();
const errors = await catalog.collectInlineErrors(page);
for (const error of errors) {
const message = error.messageText.toLowerCase();
// Should not say "504" or "gateway timeout" to the userexpect(message).not.toContain('504');
expect(message).not.toContain('gateway timeout');
}
});
});
Step 6: Generate the Error Catalog Report
After running all tests, produce a consolidated report that lists every error surface, its grade, and prioritized recommendations.
// helpers/message-rules.tsexportinterfaceMessageRule {
id: string;
name: string;
description: string;
severity: 'error' | 'warning' | 'info';
check: (message: string) =>boolean;
recommendation: string;
}
exportconstdefaultRules: MessageRule[] = [
{
id: 'no-technical-jargon',
name: 'No Technical Jargon',
description: 'Error messages should not contain technical terms unfamiliar to end users',
severity: 'error',
check: (msg) =>/exception|stack|null pointer|segfault|core dump/i.test(msg),
recommendation: 'Replace technical terms with plain language descriptions',
},
{
id: 'no-blame-language',
name: 'No Blame Language',
description: 'Error messages should not blame the user for the error',
severity: 'warning',
check: (msg) =>/you (did|made|caused|broke)|your fault|your error/i.test(msg),
recommendation: 'Use passive voice or focus on the system state rather than user actions',
},
{
id: 'minimum-length',
name: 'Minimum Message Length',
description: 'Error messages must be at least 15 characters to be meaningful',
severity: 'error',
check: (msg) => msg.trim().length < 15,
recommendation: 'Expand the message to describe the problem and suggest a solution',
},
{
id: 'no-raw-http-codes',
name: 'No Raw HTTP Status Codes',
description: 'Users should never see raw HTTP status codes as the primary error message',
severity: 'error',
check: (msg) =>/^(4\d{2}|5\d{2})\b/.test(msg.trim()),
recommendation: 'Translate HTTP status codes into human-readable explanations',
},
{
id: 'sentence-case',
name: 'Sentence Case',
description: 'Error messages should use sentence case, not ALL CAPS or Title Case',
severity: 'warning',
check: (msg) => msg === msg.toUpperCase() && msg.length > 10,
recommendation: 'Convert to sentence case for better readability and accessibility',
},
{
id: 'no-period-missing',
name: 'Proper Punctuation',
description: 'Multi-sentence error messages should end with proper punctuation',
severity: 'info',
check: (msg) => msg.includes('. ') && !/[.!?]$/.test(msg.trim()),
recommendation: 'End multi-sentence messages with a period',
},
{
id: 'has-recovery-action',
name: 'Recovery Action Present',
description: 'Error messages should include a recovery action or next step',
severity: 'error',
check: (msg) => {
const actionVerbs = ['try', 'click', 'refresh', 'check', 'update', 'contact', 'sign in', 'reload'];
return !actionVerbs.some((verb) => msg.toLowerCase().includes(verb));
},
recommendation: 'Add a clear action the user can take to recover from the error',
},
];
Best Practices
Catalog first, fix second. Build a complete inventory of every error surface before changing any individual message. Fixing messages piecemeal produces inconsistency.
Test errors in context, not in isolation. An error message that reads well in a spreadsheet might be confusing when displayed inside a small tooltip with truncated text. Always verify messages in the actual UI.
Maintain a banned phrases list. Keep a centralized list of phrases that must never appear in error messages, such as "something went wrong" or "an error occurred." Run automated checks against this list in CI.
Separate user-facing messages from developer logs. Errors sent to logging services can be as verbose and technical as needed. The user-facing message must be a separate, curated string.
Include error codes for support channels. If users need to contact support, give them a short, memorable error code (like "ERR-1042") that support staff can look up. Never make error codes the primary message.
Test with screen readers. Error messages must be announced by assistive technology. Verify that ARIA live regions are used correctly and that messages are read aloud when they appear.
Test localized versions. If the application supports multiple languages, audit error messages in every locale. Machine-translated error messages are frequently nonsensical.
Grade on a rubric, not gut feeling. Use a consistent scoring system with defined criteria. Subjective "this seems fine" reviews miss systemic patterns.
Check error message consistency across related flows. The password reset flow and the login flow should use the same tone, format, and vocabulary for similar errors.
Monitor error messages in production. Track which error messages users actually see using analytics. The most frequently shown error messages deserve the most attention.
Validate that errors disappear after correction. When a user fixes the problem (enters a valid email after an invalid one), the error message must clear immediately. Stale error messages erode trust.
Test empty and boundary states. Submit forms with maximum-length inputs, special characters, and empty fields. These boundary cases frequently produce unhelpful or broken error messages.
Anti-Patterns to Avoid
Generic catch-all messages. Displaying "Something went wrong" for every error is lazy and unhelpful. Each error type should have its own tailored message.
Exposing stack traces to users. This is both a UX failure and a security vulnerability. Never render raw exception output in the UI.
Blaming the user. Messages like "You entered an invalid email" feel accusatory. Prefer "Please enter a valid email address" which focuses on the solution.
Using error codes as the sole message. "Error 403" means nothing to most users. Always accompany codes with a human-readable explanation.
Hardcoding error strings in components. Error messages scattered across dozens of component files become impossible to audit and maintain. Centralize them in a messages file or i18n system.
Showing technical HTTP status pages. A raw Nginx 502 Bad Gateway page tells the user nothing useful and looks unprofessional.
Hiding errors silently. Swallowing errors without showing any feedback leaves users confused about why their action did not succeed.
Using humor in critical errors. A lighthearted "Oops!" is acceptable for a non-critical tooltip failure. It is inappropriate for a failed payment or a data loss scenario.
Displaying multiple duplicate errors. When a form submission triggers three identical "Required field" messages with no indication of which field they refer to, the UX is broken.
Inconsistent capitalization and punctuation. Mixing "Email is required." with "PASSWORD MUST NOT BE EMPTY" and "please provide a username" across the same form signals a lack of quality control.
Debugging Tips
Use Playwright's page.on('console') listener to capture all console output during error state tests. Console warnings often reveal error message rendering issues that are not visible in the UI.
Intercept network responses with page.route() to simulate specific HTTP error codes and verify the corresponding UI messages. Test 400, 401, 403, 404, 422, 429, 500, 502, 503, and 504 responses individually.
Take screenshots at the moment each error appears. Visual evidence is invaluable when reporting error message quality issues to designers and copywriters.
Check the DOM for hidden error elements. Some error messages exist in the DOM but are hidden by CSS. Use page.locator().isVisible() to verify that error messages intended for display are actually visible.
Test error message timing. Use performance.now() or Playwright's built-in timing to verify that error messages appear within 200ms of the triggering action. Delayed error feedback feels broken.
Inspect ARIA attributes on error containers. Verify that error message containers have role="alert" or aria-live="polite" so assistive technology announces them.
Test error recovery flows end-to-end. After triggering an error, verify that correcting the input and resubmitting clears the error and proceeds normally. Error states that persist after correction are severe bugs.
Validate error message rendering at different viewport sizes. Long error messages may overflow or get truncated on mobile viewports. Test at 320px, 375px, and 768px widths.
Compare error messages across browsers. Some browsers add their own error text (especially for form validation). Verify that custom error messages override browser defaults consistently.
Log the raw API response alongside the displayed message. When a UI error message does not match the API error, it indicates a missing or incorrect error transformation layer in the frontend code.
