| name | media-lens-gcp-debug |
| type | diagnostic |
| description | Debug and diagnose media-lens production issues on Google Cloud Platform |
| domain | Media Lens Infrastructure |
| triggers | ["debug gcp","gcp debug","check gcp status","media lens isn't running"] |
media-lens-gcp-debug Skill
Summary
Comprehensive debugging and diagnostics skill for media-lens production issues on Google Cloud Platform. Helps investigate VM status, container logs, job execution history, Cloud Storage, and scheduling configuration.
Triggers
Activate this skill when:
- User says: "debug gcp", "gcp debug", "investigate production", "what's wrong with gcp", "check gcp status"
- User reports: "media lens isn't running", "no jobs yesterday", "results missing for date X"
- User asks: "why didn't the job run", "check if vm is running", "see container logs"
- User needs: "diagnose deployment issue", "troubleshoot cloud run", "why is production broken"
Capabilities
1. VM Instance Diagnostics
- Check instance status (running/stopped)
- Get external IP address
- Retrieve instance metadata and configuration
- List all compute instances
- Check startup script execution logs
- SSH into VM for manual investigation
2. Container & Application State
- View Docker container logs (with time ranges, tail limits)
- Check container status and lifecycle
- List running/stopped containers
- Access container shell for debugging
- Monitor application health (HTTP endpoints)
3. Job & Output Tracking
- List recent job directories on VM persistent disk
- Check job timestamps and execution order
- Verify job output files (daily_news.txt, JSON artifacts, HTML)
- Examine cursor files (format_cursor, deploy_cursor) to track pipeline stage
- Compare job counts and coverage by date
4. Cloud Storage & GCS
- List bucket contents and structure
- Check storage usage and quotas
- Verify files in gs://media-lens-storage
- Compare local vs cloud storage state
- Validate bucket permissions
5. Cloud Scheduler & Cron
- List all Cloud Scheduler jobs
- Check job execution history
- View last execution time and status
- Manually trigger jobs for testing
- Examine cron job configuration on VM
6. Persistent Disk & Mounts
- Check disk attachment status
- Verify mount points and permissions
- Monitor disk space usage
- Detect mount issues or corruption
7. Network & Connectivity
- Test health check endpoints (port 8080)
- Trigger manual runs via HTTP API
- Check web server responsiveness
- Validate external IP accessibility
Implementation Details
Prerequisites
- Google Cloud SDK (
gcloud) installed and authenticated
- Project ID:
medialens
- VM zone:
us-central1-a
- Container name:
media-lens
- Persistent disk:
media-lens-data mounted at /app/working
Critical: Scheduled Lifecycle
⚠️ The VM and container are NOT continuously running. They follow this schedule:
- Normal state: VM is STOPPED (saves costs)
- 6:00 AM UTC (1 AM PT): Cloud Scheduler starts the VM
- ~7:00 AM UTC (2 AM PT): Container runs pipeline (harvest → extract → interpret → format → deploy)
- ~9:00 AM UTC (4 AM PT): Cloud Scheduler stops the VM (unless pipeline still running)
- Rest of day: VM remains STOPPED
When debugging:
- If VM is stopped, offer to start it (understand this keeps costs running)
- After diagnostics, ALWAYS shut down the VM (unless user explicitly wants to leave it running)
- Note: If VM is already running, container may or may not be active (could be between job runs, stuck, or restarting)
Command Structure
All commands follow this pattern:
gcloud compute instances describe media-lens-vm --zone=us-central1-a --format="get(status)"
gcloud compute instances start media-lens-vm --zone=us-central1-a
gcloud compute ssh media-lens-vm --zone=us-central1-a --command="docker logs media-lens --tail=50"
gcloud compute instances stop media-lens-vm --zone=us-central1-a
gsutil ls gs://media-lens-storage/
Key File Locations
On VM (/app/working/out/):
- Jobs:
jobs/YYYY/MM/DD/HHmmss/ (structured by date/time)
- Outputs:
intermediate/, staging/
- Cursors:
format_cursor.txt, deploy_cursor.txt
- Logs: via
docker logs media-lens
In GCS (gs://media-lens-storage/):
- Mirrors
/app/working/out/jobs/ structure
- Historical backups and archives
Workflow
-
Check VM Lifecycle State
- Is VM running or stopped?
- If stopped (normal): Offer to start it for diagnostics (with cost warning)
- If running: Check how long and why (scheduled run, stuck, user-initiated)
- Note: After diagnostics, remind user to stop VM to avoid unnecessary costs
-
Quick Status Check (if VM is running or started)
- Container health (docker ps, HTTP health check)
- Last job timestamp on persistent disk
- Format/deploy cursor values (indicates pipeline progress)
-
Deep Dive (if issues found)
- SSH to VM and inspect working directory
- Check docker logs for errors and timestamps
- Examine failed step (harvest, extract, interpret, format, deploy)
- Verify network connectivity and permissions
- Check Cloud Scheduler and cron job configuration
-
Root Cause Analysis
- Compare timestamps (job run vs cursor update vs deployment)
- Check for stalled processes or zombie containers
- Validate SFTP credentials and network access
- Verify Cloud Scheduler jobs and cron settings
-
Remediation
- Restart container if stuck
- Reset cursors to reprocess dates
- Manually trigger job via API (only if VM is running)
- Review startup script for configuration issues
- Check Cloud Scheduler job schedule/history
-
Cleanup (Critical)
- After diagnostics complete:
gcloud compute instances stop media-lens-vm --zone=us-central1-a
- Confirm VM is STOPPED before closing
- Document any findings for follow-up investigation
Integration Points
With Project Documentation
- CLAUDE.md: Deployment architecture and setup
- readme-deployment.md: Detailed GCP deployment guide
- startup-script.sh: Automated VM initialization
With Application
- Web API: POST
/run, GET /status, GET /health
- CLI:
python -m src.media_lens.runner commands
- Environment:
.env file or Secret Manager (GCP)
With GCP
- Compute Engine: VM instances, disks, networking
- Cloud Storage: gs://media-lens-storage bucket
- Cloud Scheduler: Daily job triggers
- Cloud Logging: Centralized log aggregation (optional)
Error Patterns & Solutions
| Pattern | Likely Cause | Debug Steps |
|---|
| No jobs since date X | Cloud Scheduler disabled, cron misconfigured, startup script failed | Start VM (if stopped), check scheduler jobs + cron, review startup logs |
| VM won't start | GCP quota issue, disk corruption, startup script error | Check GCP quotas, try manual restart, review google-startup-scripts.service logs |
| Container crashed | Out of memory, missing dependencies, unhandled exception | Start VM, SSH in, run docker logs media-lens, check disk space (df -h) |
| Jobs run but not deployed | Deploy step fails, SFTP connectivity, permissions | Check deploy_cursor age vs last job, search logs for "upload|deploy" errors |
| Partial output (few sites) | Scraping failures, Cloudflare blocks, API quota | Check harvest logs for specific site errors, verify rate limits |
| Cannot SSH to VM when running | Network security groups, VM not actually running | Verify VM status, check firewall rules, wait for startup to complete |
| VM is stopped (expected state) | Not an error - normal behavior | Only start if needed for diagnostics; remember to stop it after |
Output Format
This skill provides:
- Quick Summary: Current VM status, last job time, any obvious issues
- Detailed Findings: Specific errors, timestamps, file states
- Recommended Actions: Exact commands to run or next investigation steps
- Links to Docs: References to CLAUDE.md, readme-deployment.md sections
Limitations & Guardrails
This skill does:
- Start VM only with explicit user request (offers, doesn't auto-start)
- Always warn before starting VM (cost implications)
- Always ask before stopping VM
- Provide diagnostic commands for user to run manually
- Inspect logs and files (read-only)
This skill does not:
- Auto-start/stop VM without explicit user confirmation
- Modify code or configuration files
- Delete, archive, or modify data without explicit confirmation
- Make architectural changes or permanent fixes
- Troubleshoot code-level bugs (only deployment/operational issues)
- Keep VMs running after diagnostics (user responsibility to stop)
Cost Warning: Starting the VM enables billing for Compute Engine. Always stop when diagnostics are complete.
For application logic issues, refer to code review and testing workflows.
Example Invocations
"Debug the media-lens GCP deployment - tell me why there are no results for Monday"
"Check if the VM is running and what state the container is in"
"I want to know the full timeline: when did each step last run, what's in storage, any errors?"
"media-lens production is broken, help me figure out what went wrong"
"Manually trigger a test run and show me the output"
Version: 1.0
Last Updated: 2026-03-25
Skill Type: Diagnostic / Operational Support
Domain: Media Lens GCP Infrastructure
