serverless-debugging

# Serverless Debugging Systematic debugging methodology for serverless/edge functions where console.log may not appear in logs and traditional debugging tools are unavailable. ## When to Use This Skill - Feature deployed but not triggering (code exists but never executes) - Console.log statements don't appear in logs - No visible errors but behavior doesn't match expectations - Database state doesn't update as expected - Type coercion or default value bugs - Async/timing issues in distributed systems ## Core Methodology: 4-Phase Approach ### Phase 1: Evidence Gathering Collect evidence from ALL layers before forming hypotheses. Avoid jumping to conclusions. **Database Layer:** - Query actual data state (not assumptions) - Check default values vs. expected values - Verify row counts, timestamps, update patterns - Look for NULL vs. 0 vs. empty string differences **Code Layer:** - Verify deployed code matches local code - Check function version numbers - Review conditional logic and type checks - Identify all code paths that could prevent execution **Runtime Layer:** - Check deployment status (ACTIVE, version number) - Review available logs (even if incomplete) - Note what SHOULD appear in logs but doesn't - Check for silent failures (no error, no success) **Environment Layer:** - Verify environment variables are set - Check permissions and RLS policies - Confirm API keys and authentication - Review timeout settings **Evidence Checklist:** See `references/evidence-checklist.md` for comprehensive list ### Phase 2: Pattern Analysis Compare working vs. broken behavior to identify differences. **Key Questions:** - What works? What doesn't? - When did it last work? What changed? - Are there similar features that DO work? How do they differ? - What's the simplest test case that reproduces the issue? **Common Patterns:** - Logging doesn't work in this environment (find alternatives) - Code path never reached (condition always false) - Type mismatch (0 vs null, string vs number) - Async race condition (timing-dependent) - Environment-specific behavior (works locally, fails in production) ### Phase 3: Hypothesis Formation Form specific, testable hypotheses based on evidence. **Good Hypothesis Characteristics:** - Explains ALL observed symptoms - Testable with available tools - Specific about root cause (not vague) - Based on evidence, not assumptions **Example Hypotheses:** - "Database default is 0, but code checks for null" - "Function never reaches line X because condition Y is always false" - "Environment variable Z is not set in production" - "RLS policy blocks access for this user role" **Testing Without Logs:** - Use database writes as "print statements" - Add error responses with diagnostic info - Create minimal reproduction cases - Test conditions in isolation ### Phase 4: Implementation & Verification Fix the root cause and verify across all layers. **Fix Implementation:** - Make minimal, targeted changes - Document the bug and fix - Update tests to prevent regression - Deploy new version **Verification:** - Database state updates as expected - Feature triggers in expected scenarios - No side effects on other features - End-to-end test passes ## Common Serverless Bugs See `references/common-serverless-bugs.md` for detailed catalog. Quick reference: **Type Coercion Issues:** - `0 === null` → false (check for both) - `undefined === null` → false - Empty string vs null vs undefined - Number strings vs numbers ("0" vs 0) **Async/Timing Issues:** - Fire-and-forget operations that fail silently - Race conditions in parallel operations - Timeout before completion - Promise not awaited **Environment Differences:** - Console.log works locally but not in production - Environment variables not set - Different runtime versions - Permission differences **Database Issues:** - RLS policies blocking access - Default values vs. expected values - Transaction isolation issues - Connection pool exhaustion ## Alternative Debugging Strategies When console.log doesn't work, use these alternatives: **Database-Driven Debugging:** ```sql -- Create debug log table CREATE TABLE debug_logs ( id SERIAL PRIMARY KEY, timestamp TIMESTAMPTZ DEFAULT NOW(), function_name TEXT, checkpoint TEXT, data JSONB ); -- Write debug info from code INSERT INTO debug_logs (function_name, checkpoint, data) VALUES ('my-function', 'before-extraction', jsonb_build_object('count', message_count)); ``` **Error Response Debugging:** ```typescript // Return diagnostic info in error response return new Response(JSON.stringify({ debug: { messageCount: messageCount, lastExtraction: lastExtraction, shouldExtract: shouldExtractMemories(messageCount, lastExtraction) } }), { status: 200 }); ``` **Code Analysis:** - Read deployed code directly (if accessible) - Trace execution paths on paper - Check type of every variable in conditions - Verify all branches can be reached See `references/logging-alternatives.md` for more strategies. ## Practical Tools **Deployment Verification:** ```bash # Use scripts/check_deployment.py python scripts/check_deployment.py <function-name> <local-path> ``` **Database State Analysis:** ```bash # Use scripts/database_debugger.py python scripts/database_debugger.py <table-name> <condition> ``` **Type Analysis:** ```bash # Use scripts/type_analyzer.py python scripts/type_analyzer.py <file-path> <function-name> ``` ## Real-World Example: Wade Memory Extraction Bug **Symptom:** Proactive memory extraction never triggered despite code being deployed. **Phase 1 Evidence:** - ✅ Code deployed (v48) - ✅ Database schema correct - ✅ Messages stored (8-10 messages) - ❌ Console.log never appeared - ❌ `last_extraction_at` stayed at 0 **Phase 2 Pattern:** - Console.log doesn't work in Supabase Edge Functions logs API - Needed alternative debugging approach **Phase 3 Hypothesis:** - Database default for `last_extraction_at` is 0, not null - Code checks `if (lastExtraction === null)` → always false when 0 - Takes wrong branch: `messageCount - 0 >= 10` instead of `messageCount >= 4` **Phase 4 Fix:** ```typescript // Before: Only checked null if (lastExtractionAt === null) { return messageCount >= 4; } // After: Check both null AND 0 if (lastExtractionAt === null || lastExtractionAt === 0) { return messageCount >= 4; } ``` **Result:** Feature now works correctly, triggers after 4 messages. ## Best Practices 1. **Gather evidence before hypothesizing** - Resist the urge to guess 2. **Use database as debugging tool** - When logs fail, database writes work 3. **Check type coercion carefully** - 0, null, undefined are all different 4. **Verify deployed code** - Don't assume local matches production 5. **Test in isolation** - Minimal reproduction cases reveal root cause 6. **Document the process** - Future debugging benefits from past patterns ## Quick Start 1. Read `references/evidence-checklist.md` and gather evidence from all layers 2. Identify patterns using Phase 2 questions 3. Form testable hypothesis 4. Use `scripts/` tools to verify hypothesis 5. Implement fix and verify across all layers 6. Document bug and fix for future reference