serverless-debugging

name	serverless-debugging
description	Debug serverless and edge functions where traditional logging is limited. Use when features are deployed but not triggering, logs are missing, or runtime behavior differs from expectations in Supabase Edge Functions, AWS Lambda, Cloudflare Workers, or similar platforms.

Serverless Debugging

Systematic debugging methodology for serverless/edge functions where console.log may not appear in logs and traditional debugging tools are unavailable.

When to Use This Skill

• Feature deployed but not triggering (code exists but never executes)
• Console.log statements don't appear in logs
• No visible errors but behavior doesn't match expectations
• Database state doesn't update as expected
• Type coercion or default value bugs
• Async/timing issues in distributed systems

Core Methodology: 4-Phase Approach

Phase 1: Evidence Gathering

Collect evidence from ALL layers before forming hypotheses. Avoid jumping to conclusions.

Database Layer:

• Query actual data state (not assumptions)
• Check default values vs. expected values
• Verify row counts, timestamps, update patterns
• Look for NULL vs. 0 vs. empty string differences

Code Layer:

• Verify deployed code matches local code
• Check function version numbers
• Review conditional logic and type checks
• Identify all code paths that could prevent execution

Runtime Layer:

• Check deployment status (ACTIVE, version number)
• Review available logs (even if incomplete)
• Note what SHOULD appear in logs but doesn't
• Check for silent failures (no error, no success)

Environment Layer:

• Verify environment variables are set
• Check permissions and RLS policies
• Confirm API keys and authentication
• Review timeout settings

Evidence Checklist: See references/evidence-checklist.md for comprehensive list

Phase 2: Pattern Analysis

Compare working vs. broken behavior to identify differences.

Key Questions:

• What works? What doesn't?
• When did it last work? What changed?
• Are there similar features that DO work? How do they differ?
• What's the simplest test case that reproduces the issue?

Common Patterns:

• Logging doesn't work in this environment (find alternatives)
• Code path never reached (condition always false)
• Type mismatch (0 vs null, string vs number)
• Async race condition (timing-dependent)
• Environment-specific behavior (works locally, fails in production)

Phase 3: Hypothesis Formation

Form specific, testable hypotheses based on evidence.

Good Hypothesis Characteristics:

• Explains ALL observed symptoms
• Testable with available tools
• Specific about root cause (not vague)
• Based on evidence, not assumptions

Example Hypotheses:

• "Database default is 0, but code checks for null"
• "Function never reaches line X because condition Y is always false"
• "Environment variable Z is not set in production"
• "RLS policy blocks access for this user role"

Testing Without Logs:

• Use database writes as "print statements"
• Add error responses with diagnostic info
• Create minimal reproduction cases
• Test conditions in isolation

Phase 4: Implementation & Verification

Fix the root cause and verify across all layers.

Fix Implementation:

• Make minimal, targeted changes
• Document the bug and fix
• Update tests to prevent regression
• Deploy new version

Verification:

• Database state updates as expected
• Feature triggers in expected scenarios
• No side effects on other features
• End-to-end test passes

Common Serverless Bugs

See references/common-serverless-bugs.md for detailed catalog. Quick reference:

Type Coercion Issues:

• 0 === null → false (check for both)
• undefined === null → false
• Empty string vs null vs undefined
• Number strings vs numbers ("0" vs 0)

Async/Timing Issues:

• Fire-and-forget operations that fail silently
• Race conditions in parallel operations
• Timeout before completion
• Promise not awaited

Environment Differences:

• Console.log works locally but not in production
• Environment variables not set
• Different runtime versions
• Permission differences

Database Issues:

• RLS policies blocking access
• Default values vs. expected values
• Transaction isolation issues
• Connection pool exhaustion

Alternative Debugging Strategies

When console.log doesn't work, use these alternatives:

Database-Driven Debugging:

-- Create debug log table
CREATE TABLE debug_logs (
  id SERIAL PRIMARY KEY,
  timestamp TIMESTAMPTZ DEFAULT NOW(),
  function_name TEXT,
  checkpoint TEXT,
  data JSONB
);

-- Write debug info from code
INSERT INTO debug_logs (function_name, checkpoint, data)
VALUES ('my-function', 'before-extraction', jsonb_build_object('count', message_count));

Error Response Debugging:

// Return diagnostic info in error response
return new Response(JSON.stringify({
  debug: {
    messageCount: messageCount,
    lastExtraction: lastExtraction,
    shouldExtract: shouldExtractMemories(messageCount, lastExtraction)
  }
}), { status: 200 });

Code Analysis:

• Read deployed code directly (if accessible)
• Trace execution paths on paper
• Check type of every variable in conditions
• Verify all branches can be reached

See references/logging-alternatives.md for more strategies.

Practical Tools

Deployment Verification:

# Use scripts/check_deployment.py
python scripts/check_deployment.py <function-name> <local-path>

Database State Analysis:

# Use scripts/database_debugger.py
python scripts/database_debugger.py <table-name> <condition>

Type Analysis:

# Use scripts/type_analyzer.py
python scripts/type_analyzer.py <file-path> <function-name>

Real-World Example: Wade Memory Extraction Bug

Symptom: Proactive memory extraction never triggered despite code being deployed.

Phase 1 Evidence:

• ✅ Code deployed (v48)
• ✅ Database schema correct
• ✅ Messages stored (8-10 messages)
• ❌ Console.log never appeared
• ❌ last_extraction_at stayed at 0

Phase 2 Pattern:

• Console.log doesn't work in Supabase Edge Functions logs API
• Needed alternative debugging approach

Phase 3 Hypothesis:

• Database default for last_extraction_at is 0, not null
• Code checks if (lastExtraction === null) → always false when 0
• Takes wrong branch: messageCount - 0 >= 10 instead of messageCount >= 4

Phase 4 Fix:

// Before: Only checked null
if (lastExtractionAt === null) {
  return messageCount >= 4;
}

// After: Check both null AND 0
if (lastExtractionAt === null || lastExtractionAt === 0) {
  return messageCount >= 4;
}

Result: Feature now works correctly, triggers after 4 messages.

Best Practices

1. Gather evidence before hypothesizing - Resist the urge to guess
2. Use database as debugging tool - When logs fail, database writes work
3. Check type coercion carefully - 0, null, undefined are all different
4. Verify deployed code - Don't assume local matches production
5. Test in isolation - Minimal reproduction cases reveal root cause
6. Document the process - Future debugging benefits from past patterns

Quick Start

1. Read references/evidence-checklist.md and gather evidence from all layers
2. Identify patterns using Phase 2 questions
3. Form testable hypothesis
4. Use scripts/ tools to verify hypothesis
5. Implement fix and verify across all layers
6. Document bug and fix for future reference

When 3+ Fixes Fail: Question the Architecture

Red flag pattern: Multiple fix attempts, each revealing new problems in different layers.

Symptoms

• ✅ Fix #1: Added environment variables → Still failing
• ✅ Fix #2: Force redeployed → Still failing
• ✅ Fix #3: Added to all environments → Still failing
• ✅ Fix #4: Created lazy initialization → Deployment failed
• ✅ Fix #5: Tried different deployment method → New error

Pattern: Not converging on a solution. Each fix reveals a NEW problem.

What This Means

Per systematic-debugging Phase 4.5:

If 3+ fixes failed: STOP and question the architecture

This is NOT a bug. This is an ARCHITECTURAL PROBLEM.

Root Cause Analysis

The real issues:

1. Generic error messages - Can't see actual problem
2. No direct code access - Can't iterate quickly
3. Complex deployment pipeline - Adds failure points
4. No local testing - Can't reproduce issue
5. Limited observability - Can't see runtime state

What to Do

STOP attempting more fixes. Instead:

1. Get better error visibility

   • Add comprehensive error logging (see references/enhanced-error-logging.md)
   • Access full logs (not truncated)
   • Add diagnostic endpoints

2. Simplify the workflow

   • Test locally first
   • Use proper version control (GitHub)
   • Set up CI/CD pipeline
   • Remove manual deployment steps

3. Question the fundamentals

   • Is this deployment method sound?
   • Are we fighting the platform?
   • Should we refactor the approach?

Real-World Example: Stripe Integration

Attempts:

1. Added Stripe keys to Vercel → Still failing
2. Force redeployed → Still failing
3. Added keys to all environments → Still failing
4. Created lazy initialization fix → Deployment failed
5. Tried Google Drive transfer → Downloaded 0 bytes

Root cause: Generic error message hid the actual Stripe error

Solution: Add comprehensive error logging to see real error, then fix that

Lesson: After 3 failed fixes, we should have stopped and added better logging FIRST

Key Takeaways

• 3+ failed fixes = architectural problem
• Stop fixing, start observing
• Add logging before adding fixes
• Question the deployment workflow
• Test locally before deploying

See also: systematic-debugging skill Phase 4.5

Lazy Initialization for External Services

Problem: Module-level initialization fails in serverless environments.

// ❌ DON'T: Initialize at module load
const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

Why it fails:

• Environment variables may not be ready at cold start
• No validation that keys exist
• Initialization errors hidden
• Cached instances retain old values

Solution: Lazy initialization pattern

// ✅ DO: Initialize on first use
let stripeInstance = null;

function getStripe() {
  if (!stripeInstance) {
    const key = process.env.STRIPE_SECRET_KEY;
    if (!key) throw new Error('STRIPE_SECRET_KEY not set');
    stripeInstance = require('stripe')(key);
  }
  return stripeInstance;
}

Benefits:

• ✅ Environment variables validated
• ✅ Clear error if missing
• ✅ Initialization after env vars loaded
• ✅ Proper error logging

See references/lazy-initialization-patterns.md for complete examples.

Enhanced Error Logging

Problem: Generic error messages hide root cause.

// ❌ DON'T: Hide error details
catch (error) {
  res.status(500).json({ error: 'An error occurred' });
}

Solution: Log full error details

// ✅ DO: Log comprehensive error info
catch (error) {
  console.error('Operation failed:', {
    message: error.message,
    type: error.type,
    code: error.code,
    stack: error.stack,
    context: { priceId, userId }
  });
  res.status(500).json({ 
    error: error.message,
    type: error.type,
    code: error.code
  });
}

See references/enhanced-error-logging.md for patterns.

Diagnostic Health Endpoints

Add health check endpoints to verify configuration:

app.get('/api/health', (req, res) => {
  const config = {
    stripe: !!process.env.STRIPE_SECRET_KEY,
    supabase: !!process.env.SUPABASE_URL
  };
  
  const keyPrefixes = {
    stripe: config.stripe ? process.env.STRIPE_SECRET_KEY.substring(0, 7) : 'none'
  };
  
  res.json({ 
    status: 'ok',
    configured: config,
    keyPrefixes: keyPrefixes
  });
});

Usage:

curl https://your-api.vercel.app/api/health

This immediately shows:

• Which services are configured
• Key prefixes (safe to log)
• Whether environment variables are set