zero-script-qa

# Zero Script QA Expert Knowledge ## Overview Zero Script QA is a methodology that verifies features through **structured logs** and **real-time monitoring** without writing test scripts. ``` Traditional: Write test code → Execute → Check results → Maintain Zero Script: Build log infrastructure → Manual UX test → AI log analysis → Auto issue detection ``` ## Core Principles ### 1. Log Everything - All API calls (including 200 OK) - All errors - All important business events - Entire flow trackable via Request ID ### 2. Structured JSON Logs - Parseable JSON format - Consistent fields (timestamp, level, request_id, message, data) - Different log levels per environment ### 3. Real-time Monitoring - Docker log streaming - Claude Code analyzes in real-time - Immediate issue detection and documentation --- ## Logging Architecture ### JSON Log Format Standard ```json { "timestamp": "2026-01-08T10:30:00.000Z", "level": "INFO", "service": "api", "request_id": "req_abc123", "message": "API Request completed", "data": { "method": "POST", "path": "/api/users", "status": 200, "duration_ms": 45 } } ``` ### Required Log Fields | Field | Type | Description | |-------|------|-------------| | timestamp | ISO 8601 | Time of occurrence | | level | string | DEBUG, INFO, WARNING, ERROR | | service | string | Service name (api, web, worker, etc.) | | request_id | string | Request tracking ID | | message | string | Log message | | data | object | Additional data (optional) | ### Log Level Policy | Environment | Minimum Level | Purpose | |-------------|---------------|---------| | Local | DEBUG | Development and QA | | Staging | DEBUG | QA and integration testing | | Production | INFO | Operations monitoring | --- ## Request ID Propagation ### Concept ``` Client → API Gateway → Backend → Database ↓ ↓ ↓ ↓ req_abc req_abc req_abc req_abc Trackable with same Request ID across all layers ``` ### Implementation Patterns #### 1. Request ID Generation (Entry Point) ```typescript // middleware.ts import { v4 as uuidv4 } from 'uuid'; export function generateRequestId(): string { return `req_${uuidv4().slice(0, 8)}`; } // Propagate via header headers['X-Request-ID'] = requestId; ``` #### 2. Request ID Extraction and Propagation ```typescript // API client const requestId = headers['X-Request-ID'] || generateRequestId(); // Include in all logs logger.info('Processing request', { request_id: requestId }); // Include in header when calling downstream services await fetch(url, { headers: { 'X-Request-ID': requestId } }); ``` --- ## Backend Logging (FastAPI) ### Logging Middleware ```python # middleware/logging.py import logging import time import uuid import json from fastapi import Request class JsonFormatter(logging.Formatter): def format(self, record): log_record = { "timestamp": self.formatTime(record), "level": record.levelname, "service": "api", "request_id": getattr(record, 'request_id', 'N/A'), "message": record.getMessage(), } if hasattr(record, 'data'): log_record["data"] = record.data return json.dumps(log_record) class LoggingMiddleware: async def __call__(self, request: Request, call_next): request_id = request.headers.get('X-Request-ID', f'req_{uuid.uuid4().hex[:8]}') request.state.request_id = request_id start_time = time.time() # Request logging logger.info( f"Request started", extra={ 'request_id': request_id, 'data': { 'method': request.method, 'path': request.url.path, 'query': str(request.query_params) } } ) response = await call_next(request) duration = (time.time() - start_time) * 1000 # Response logging (including 200 OK!) logger.info( f"Request completed", extra={ 'request_id': request_id, 'data': { 'status': response.status_code, 'duration_ms': round(duration, 2) } } ) response.headers['X-Request-ID'] = request_id return response ``` ### Business Logic Logging ```python # services/user_service.py def create_user(data: dict, request_id: str): logger.info("Creating user", extra={ 'request_id': request_id, 'data': {'email': data['email']} }) # Business logic user = User(**data) db.add(user) db.commit() logger.info("User created", extra={ 'request_id': request_id, 'data': {'user_id': user.id} }) return user ``` --- ## Frontend Logging (Next.js) ### Logger Module ```typescript // lib/logger.ts type LogLevel = 'DEBUG' | 'INFO' | 'WARNING' | 'ERROR'; interface LogData { request_id?: string; [key: string]: any; } const LOG_LEVELS: Record<LogLevel, number> = { DEBUG: 0, INFO: 1, WARNING: 2, ERROR: 3, }; const MIN_LEVEL = process.env.NODE_ENV === 'production' ? 'INFO' : 'DEBUG'; function log(level: LogLevel, message: string, data?: LogData) { if (LOG_LEVELS[level] < LOG_LEVELS[MIN_LEVEL]) return; const logEntry = { timestamp: new Date().toISOString(), level, service: 'web', request_id: data?.request_id || 'N/A', message, data: data ? { ...data, request_id: undefined } : undefined, }; console.log(JSON.stringify(logEntry)); } export const logger = { debug: (msg: string, data?: LogData) => log('DEBUG', msg, data), info: (msg: string, data?: LogData) => log('INFO', msg, data), warning: (msg: string, data?: LogData) => log('WARNING', msg, data), error: (msg: string, data?: LogData) => log('ERROR', msg, data), }; ``` ### API Client Integration ```typescript // lib/api-client.ts import { logger } from './logger'; import { v4 as uuidv4 } from 'uuid'; export async function apiClient<T>( endpoint: string, options: RequestInit = {} ): Promise<T> { const requestId = `req_${uuidv4().slice(0, 8)}`; const startTime = Date.now(); logger.info('API Request started', { request_id: requestId, method: options.method || 'GET', endpoint, }); try { const response = await fetch(`/api${endpoint}`, { ...options, headers: { 'Content-Type': 'application/json', 'X-Request-ID': requestId, ...options.headers, }, }); const duration = Date.now() - startTime; const data = await response.json(); // Log 200 OK too! logger.info('API Request completed', { request_id: requestId, status: response.status, duration_ms: duration, }); if (!response.ok) { logger.error('API Request failed', { request_id: requestId, status: response.status, error: data.error, }); throw new ApiError(data.error); } return data; } catch (error) { logger.error('API Request error', { request_id: requestId, error: error instanceof Error ? error.message : 'Unknown error', }); throw error; } } ``` --- ## Nginx JSON Logging ### nginx.conf Configuration ```nginx http { log_format json_combined escape=json '{' '"timestamp":"$time_iso8601",' '"level":"INFO",' '"service":"nginx",' '"request_id":"$http_x_request_id",' '"message":"HTTP Request",' '"data":{' '"remote_addr":"$remote_addr",' '"method":"$request_method",' '"uri":"$request_uri",' '"status":$status,' '"body_bytes_sent":$body_bytes_sent,' '"request_time":$request_time,' '"upstream_response_time":"$upstream_response_time",' '"http_referer":"$http_referer",' '"http_user_agent":"$http_user_agent"' '}' '}'; access_log /var/log/nginx/access.log json_combined; } ``` --- ## Docker-Based QA Workflow ### docker-compose.yml Configuration ```yaml version: '3.8' services: api: build: ./backend environment: - LOG_LEVEL=DEBUG - LOG_FORMAT=json logging: driver: json-file options: max-size: "10m" max-file: "3" web: build: ./frontend environment: - NODE_ENV=development depends_on: - api nginx: image: nginx:alpine volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf ports: - "80:80" depends_on: - api - web ``` ### Real-time Log Monitoring ```bash # Stream all service logs docker compose logs -f # Specific service only docker compose logs -f api # Filter errors only docker compose logs -f | grep '"level":"ERROR"' # Track specific Request ID docker compose logs -f | grep 'req_abc123' ``` --- ## QA Automation Workflow ### 1. Start Environment ```bash # Start development environment docker compose up -d # Start log monitoring (Claude Code monitors) docker compose logs -f ``` ### 2. Manual UX Testing ``` User tests actual features in browser: 1. Sign up attempt 2. Login attempt 3. Use core features 4. Test edge cases ``` ### 3. Claude Code Log Analysis ``` Claude Code in real-time: 1. Monitor log stream 2. Detect error patterns 3. Detect abnormal response times 4. Track entire flow via Request ID 5. Auto-document issues ``` ### 4. Issue Documentation ```markdown # QA Issue Report ## Issues Found ### ISSUE-001: Insufficient error handling on login failure - **Request ID**: req_abc123 - **Severity**: Medium - **Reproduction path**: Login → Wrong password - **Log**: ```json {"level":"ERROR","message":"Login failed","data":{"error":"Invalid credentials"}} ``` - **Problem**: Error message not user-friendly