evernote-incident-runbook

name	evernote-incident-runbook
description	Incident response runbook for Evernote integration issues. Use when troubleshooting production incidents, handling outages, or responding to Evernote service issues. Trigger with phrases like "evernote incident", "evernote outage", "evernote emergency", "troubleshoot evernote production".
allowed-tools	Read, Write, Edit, Bash(curl:*), Grep
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>

Evernote Incident Runbook

Overview

Step-by-step procedures for responding to Evernote integration incidents including outages, rate limits, authentication failures, and data issues.

Prerequisites

• Access to monitoring dashboards
• Production logs access
• Evernote API credentials
• Communication channels for escalation

Incident Classification

Severity	Impact	Response Time	Example
P1 - Critical	All users affected	15 min	Complete API outage
P2 - High	Major feature broken	30 min	OAuth failures
P3 - Medium	Partial degradation	2 hours	High error rate
P4 - Low	Minor issues	1 day	Slow response times

Incident Response Procedures

INC-01: Complete API Outage

Symptoms:

• All Evernote API calls failing
• 5xx errors from Evernote
• Connection timeouts

Investigation:

# Step 1: Check Evernote service status
curl -I https://www.evernote.com/
curl -I https://sandbox.evernote.com/

# Step 2: Check status page
# https://status.evernote.com/

// Step 3: Run diagnostic
async function diagnoseOutage() {
  const results = {
    timestamp: new Date().toISOString(),
    checks: []
  };

  // DNS resolution
  try {
    const dns = require('dns').promises;
    const addresses = await dns.resolve4('www.evernote.com');
    results.checks.push({ name: 'DNS', status: 'ok', addresses });
  } catch (error) {
    results.checks.push({ name: 'DNS', status: 'failed', error: error.message });
  }

  // TCP connectivity
  try {
    const net = require('net');
    await new Promise((resolve, reject) => {
      const socket = net.connect(443, 'www.evernote.com');
      socket.setTimeout(5000);
      socket.(,  { socket.(); (); });
      socket.(, reject);
      socket.(,  { socket.(); ( ()); });
    });
    results..({ : , :  });
  }  (error) {
    results..({ : , : , : error. });
  }

  
   {
     https = ();
      ( {
       req = https.(, { :  },  {
        results..({ : , : , : res. });
        ();
      });
      req.(, reject);
    });
  }  (error) {
    results..({ : , : , : error. });
  }

   results;
}

Mitigation:

1. Activate circuit breaker to prevent cascading failures
2. Enable graceful degradation (show cached data)
3. Display user-friendly error message
4. Monitor Evernote status page

// Activate graceful degradation
const degradationMode = {
  enabled: true,
  reason: 'Evernote service unavailable',
  startTime: Date.now(),

  shouldServeFromCache: true,
  shouldBlockWrites: true,
  userMessage: 'Note syncing is temporarily unavailable. Your changes will sync when service is restored.'
};

// Apply to endpoints
app.use((req, res, next) => {
  if (degradationMode.enabled) {
    res.locals.degradationMode = degradationMode;
  }
  next();
});

Resolution:

1. Monitor Evernote status for resolution
2. Gradually re-enable API calls
3. Trigger sync for affected users
4. Document incident timeline

---

INC-02: Rate Limit Crisis

Symptoms:

• Frequent RATE_LIMIT_REACHED errors
• rateLimitDuration > 300 seconds
• Multiple users affected

Investigation:

// Check rate limit metrics
async function analyzeRateLimits() {
  const metrics = await prometheusQuery(`
    sum(increase(evernote_rate_limits_total[1h])) by (api_key)
  `);

  const apiCallRate = await prometheusQuery(`
    sum(rate(evernote_api_calls_total[5m])) by (operation)
  `);

  return {
    rateLimitsLastHour: metrics,
    currentCallRate: apiCallRate,
    suspectedCauses: identifyCauses(apiCallRate)
  };
}

function identifyCauses(callRate) {
  const causes = [];

  // Check for polling abuse
  if (callRate['NoteStore.getSyncState'] > 1) {
    causes.push('Excessive sync state polling');
  }

  // Check for inefficient requests
  if (callRate['NoteStore.getResource'] > 10) {
    causes.push('Individual resource fetching (should batch)');
  }

  return causes;
}

Mitigation:

// Emergency rate limiting
class EmergencyRateLimiter {
  constructor() {
    this.globalPause = false;
    this.pauseUntil = 0;
  }

  async activateEmergencyPause(durationSeconds) {
    this.globalPause = true;
    this.pauseUntil = Date.now() + (durationSeconds * 1000);

    console.warn(`Emergency rate limit pause activated for ${durationSeconds}s`);

    // Notify operations
    await alertOps('Emergency rate limit pause', {
      duration: durationSeconds,
      reason: 'Excessive rate limits detected'
    });

    // Auto-deactivate
    setTimeout(() => {
      this.globalPause = false;
      console.info('Emergency rate limit pause deactivated');
    }, durationSeconds * 1000);
  }

  async checkBeforeRequest() {
    if (this.) {
       waitTime = . - .();
       (waitTime > ) {
          ();
      }
    }
  }
}

Resolution:

1. Identify and fix inefficient API usage
2. Increase cache TTLs
3. Implement request coalescing
4. Request rate limit boost from Evernote

---

INC-03: Authentication Failures

Symptoms:

• Users receiving auth errors
• OAuth flow failing
• Token rejections

Investigation:

// Auth diagnostic
async function diagnoseAuthIssue(userId) {
  const user = await db.users.findById(userId);
  const token = await db.tokens.findByUserId(userId);

  const diagnosis = {
    userId,
    hasToken: !!token,
    tokenExpired: token ? (Date.now() > token.expiresAt) : null,
    tokenExpiresIn: token ? Math.floor((token.expiresAt - Date.now()) / 1000 / 60 / 60) + ' hours' : null
  };

  // Test token validity
  if (token && !diagnosis.tokenExpired) {
    try {
      const client = new Evernote.Client({ token: token.accessToken, sandbox: false });
      const userStore = client.getUserStore();
      await userStore.getUser();
      diagnosis.tokenValid = true;
    } catch (error) {
      diagnosis. = ;
      diagnosis. = {
        : error.,
        : error.
      };
    }
  }

   diagnosis;
}

Common Causes & Fixes:

Error Code	Cause	Fix
4 (INVALID_AUTH)	Token revoked	Re-authenticate user
5 (AUTH_EXPIRED)	Token expired	Re-authenticate user
3 (PERMISSION_DENIED)	Insufficient permissions	Check API key permissions

Resolution:

// Batch re-auth notification
async function notifyUsersToReauth(userIds) {
  for (const userId of userIds) {
    await sendNotification(userId, {
      type: 'REAUTH_REQUIRED',
      message: 'Please reconnect your Evernote account to continue syncing.',
      action: { type: 'REDIRECT', url: '/auth/evernote' }
    });

    await db.tokens.markInvalid(userId);
  }
}

---

INC-04: Data Sync Issues

Symptoms:

• Notes not appearing
• Sync state stuck
• Missing changes

Investigation:

// Sync state diagnostic
async function diagnoseSyncIssue(userId) {
  const syncState = await db.syncState.findByUserId(userId);
  const client = await getClientForUser(userId);

  const remoteSyncState = await client.noteStore.getSyncState();

  return {
    localUSN: syncState.lastUpdateCount,
    remoteUSN: remoteSyncState.updateCount,
    behind: remoteSyncState.updateCount - syncState.lastUpdateCount,
    lastSyncAt: syncState.lastSyncAt,
    needsSync: remoteSyncState.updateCount > syncState.lastUpdateCount,
    fullSyncRequired: syncState.fullSyncRequired
  };
}

// Force resync
async function forceResync(userId) {
  await db.syncState.update(userId, {
    lastUpdateCount: 0,
    fullSyncRequired: true,
    lastSyncAt: null
  });

  // Queue sync job
  await syncQueue.add('full-sync', { userId, :  });

   { : , :  };
}

---

Incident Communication Templates

Status Page Update

## [Investigating] Evernote Integration Issue

**Time:** [TIMESTAMP]
**Status:** Investigating

We are currently investigating issues with Evernote note synchronization.
Some users may experience delays in note updates.

We will provide updates as we learn more.

User Notification

## Temporary Sync Delay

Hi [USER],

We're experiencing a temporary delay in syncing notes with Evernote.
Your local changes are saved and will sync automatically when the
issue is resolved.

No action is needed on your part.

Expected resolution: Within 2 hours

Thank you for your patience.

Resolution Update

## [Resolved] Evernote Integration Issue

**Time:** [TIMESTAMP]
**Status:** Resolved

The Evernote synchronization issue has been resolved.
All notes should now be syncing normally.

**Root Cause:** [BRIEF DESCRIPTION]
**Duration:** [X hours Y minutes]
**Impact:** [NUMBER] users affected

We apologize for any inconvenience.

Post-Incident Checklist

## Post-Incident Review

### Timeline
- [ ] Document incident timeline
- [ ] Record all actions taken
- [ ] Note what worked/didn't work

### Root Cause
- [ ] Identify root cause
- [ ] Determine contributing factors
- [ ] Assess detection time

### Prevention
- [ ] Define preventive measures
- [ ] Update monitoring/alerts
- [ ] Improve runbooks

### Follow-up
- [ ] Schedule post-mortem meeting
- [ ] Assign action items
- [ ] Update documentation

Output

• Incident classification guide
• Step-by-step response procedures
• Diagnostic scripts
• Mitigation strategies
• Communication templates
• Post-incident checklist

Resources

• Evernote Status Page
• Evernote Developer Support
• Error Handling

Next Steps

For data handling best practices, see evernote-data-handling.