Production smoke testing is the last line of defense between a deployment and a broken user experience. Unlike comprehensive end-to-end suites that may run for 30 minutes or more, a smoke suite must complete in under 2 minutes and verify that the application's critical paths are functional. This skill covers the philosophy behind production smoke testing, the architecture of a reliable smoke suite, concrete implementation patterns with Playwright and raw HTTP clients, integration with deployment pipelines, and strategies for handling third-party dependencies, authentication flows, and flaky network conditions in real production environments.
Core Principles
1. Speed Over Coverage
A smoke suite is not a regression suite. Its sole purpose is to answer one question: "Did this deployment break anything critical?" If the suite takes more than 2 minutes, it is too slow. Every test must justify its inclusion by covering a revenue-critical or user-critical path. Ruthlessly prune tests that do not protect high-value user journeys.
2. Production-Safe Execution
Smoke tests run against real production infrastructure. They must never create permanent data, modify user accounts, trigger billing events, or send real notifications. Every test must be read-only or use dedicated smoke-test accounts with sandboxed permissions. A smoke test that accidentally charges a customer or sends spam emails is worse than having no smoke tests at all.
3. Deterministic Assertions
Production environments experience variable latency, CDN caching, and third-party service delays. Smoke tests must use generous timeouts, retry logic, and assertions that tolerate minor variations. Check that a response contains expected fields rather than expecting exact string matches. Verify that status codes are in acceptable ranges rather than asserting a single value.
4. Immediate Rollback Signal
The smoke suite must integrate with your deployment pipeline to trigger automatic rollback on failure. A passing smoke suite promotes the deployment; a failing suite halts the rollout and alerts the on-call team. Without this integration, smoke tests are just monitoring with extra steps.
5. Independent and Isolated
Each smoke test must be fully independent. No test should depend on another test's side effects. Tests should run in parallel where possible to minimize total suite duration. Shared state between tests is a reliability liability that will eventually cause false positives or negatives.
The src/tests/ directory contains one file per critical path. The src/helpers/ directory holds shared utilities for configuration, HTTP requests with retry logic, custom assertions, and alerting integrations. The src/fixtures/ directory manages smoke test account credentials and test data.
Configuration
Playwright Configuration for Smoke Testing
Smoke-specific Playwright configuration prioritizes speed, reliability, and minimal resource usage:
Key decisions in this configuration: fullyParallel: true runs all tests concurrently for speed. retries: 2 tolerates transient network issues. video: 'off' avoids the overhead of recording video for every test. The X-Smoke-Test header allows backend services to identify and filter smoke test traffic from real user analytics.
Smoke Configuration Module
Centralize all smoke test configuration in a typed module:
Every production application should expose a health endpoint. The smoke suite should verify it first, as a failing health check makes all subsequent tests pointless:
Run smoke tests on a cron schedule to catch production issues before users report them. This catches problems caused by infrastructure changes, certificate expirations, or third-party outages that happen independently of your deployments:
Keep the suite under 2 minutes total execution time. If it takes longer, you have too many tests or tests that are too slow. A smoke suite should contain 10-25 focused tests, not hundreds.
Use dedicated smoke test accounts. Never use real user credentials. Create service accounts with limited permissions specifically for smoke testing. Rotate credentials regularly and store them in your secrets manager.
Mark smoke test traffic explicitly. Add custom headers like X-Smoke-Test: true or use dedicated user agents. This allows your monitoring tools to filter out smoke traffic from real user metrics and prevents smoke tests from polluting analytics data.
Test the deployment, not the test environment. Smoke tests should target the actual deployment URL, not a pre-production environment. If your deployment platform provides a unique preview URL per deployment, use that.
Implement graceful degradation for third-party checks. If a third-party service is down, your smoke suite should report a warning, not fail the entire deployment. Distinguish between first-party failures that block deployment and third-party failures that warrant a warning.
Capture screenshots and traces only on failure. Recording screenshots and traces for every test wastes resources and slows execution. Configure Playwright to capture these artifacts only when a test fails.
Use API-level checks where possible. Browser-based tests are slower and more brittle than API checks. If you can verify a critical path with an HTTP request instead of a full browser interaction, prefer the API approach.
Run smoke tests on a schedule, not just after deployments. Production issues can emerge from infrastructure changes, certificate expirations, database migrations, or third-party outages that happen independently of your deployments.
Integrate with your incident management system. Failed smoke tests should trigger PagerDuty alerts, Slack notifications, or whatever your team uses for incident response. Do not rely on engineers manually checking CI dashboards.
Version your smoke test configuration. Store all smoke test code, configuration, and pipeline definitions in version control. Treat smoke tests as production infrastructure code with the same review and deployment rigor.
Never test write operations against production. If your smoke suite must verify a form submission or API write, use a sandboxed endpoint or a mock mode. Accidentally creating real orders or sending real emails from smoke tests is a common and costly mistake.
Use health check endpoints as the first gate. Before running browser-based smoke tests, verify that the application's health endpoints return healthy. If the health check fails, skip the more expensive browser tests and report immediately.
Anti-Patterns to Avoid
Running the full regression suite as a smoke suite. This defeats the purpose. A 30-minute regression suite is not a smoke test. Extract only the critical paths into a dedicated, lightweight smoke suite with a strict time budget.
Testing implementation details instead of user outcomes. Smoke tests should verify that users can accomplish critical tasks, not that specific DOM elements have specific CSS classes. Assertions should be about functionality, not about internal implementation choices.
Ignoring flaky smoke tests. A flaky smoke test is worse than no smoke test because it erodes trust in the entire suite. When a smoke test becomes flaky, either fix it immediately or remove it until the root cause is resolved. There is no middle ground.
Hard-coding URLs and selectors. Use environment variables for base URLs and data-testid attributes for element selection. Hard-coded URLs break when deployment targets change; hard-coded CSS selectors break with every UI refactor.
Running smoke tests only in CI. Developers should be able to run the smoke suite locally against any environment. This enables pre-deployment validation and faster debugging when smoke tests fail in CI.
Sharing state between smoke tests. Tests that depend on execution order, such as "create user" must run before "user can login," are fragile and fail unpredictably when parallelized. Each test must be completely self-contained.
Not having a rollback mechanism. Smoke tests without automated rollback are just monitoring with extra steps. The primary value of post-deployment smoke tests is their ability to trigger automatic rollback before users are affected.
Over-testing error paths in smoke suites. Smoke tests verify that happy paths work. Error handling, edge cases, and validation rules belong in the regression suite. Every test in the smoke suite should exercise the golden path of a critical feature.
Debugging Tips
Smoke tests pass locally but fail in CI. Check for differences in network configuration, DNS resolution, and SSL certificates. CI runners may not have access to your production network. Verify that environment variables are correctly propagated to the test runner. Also check if your production environment has IP-based rate limiting that blocks CI runner IP addresses.
Inconsistent timeouts across runs. Production latency varies based on traffic patterns, geographic distance, and server load. Increase your default timeout to accommodate peak traffic periods. Use Playwright's expect timeout separately from the navigation timeout, and set both generously for smoke tests running against production.
Authentication tests fail intermittently. Rate limiting, CAPTCHA challenges, and bot detection systems may block smoke test logins. Work with your security team to whitelist your smoke test IP addresses or user agents. Consider using API-based authentication with tokens instead of form-based login.
Tests fail after a CDN cache purge. CDN edge caches may take several minutes to populate after a deployment. Add a short delay or retry loop for the first request to accommodate cache warming. Alternatively, bypass the CDN for smoke tests by hitting the origin server directly.
Screenshots show blank pages. This often indicates that JavaScript failed to load or execute. Check the browser console for errors by adding page.on('console', msg => console.log(msg.text())) and page.on('pageerror', err => console.error(err)) to your test setup.
Third-party status checks are unreliable. External status APIs can be slow or rate-limited. Set aggressive timeouts for third-party checks (5 seconds maximum) and do not allow them to block deployment decisions. Log warnings for slow third-party responses so you can track degradation trends over time.
Docker-based smoke runner cannot resolve internal DNS. When running smoke tests in Docker, ensure the container uses the correct DNS resolver. Use --network=host during local testing or configure the container's DNS settings appropriately for your CI environment.
