en un clic
model-debugging
// Debug and diagnose model errors in Pollinations services. Analyze logs, find error patterns, identify affected users. For taking action on user tiers, see tier-management skill.
// Debug and diagnose model errors in Pollinations services. Analyze logs, find error patterns, identify affected users. For taking action on user tiers, see tier-management skill.
| name | model-debugging |
| description | Debug and diagnose model errors in Pollinations services. Analyze logs, find error patterns, identify affected users. For taking action on user tiers, see tier-management skill. |
Use this skill when:
Related skill: Use tier-management to upgrade users or check balances after identifying issues here.
Why does the Model Monitor show high error rates when models work fine manually?
The Model Monitor at https://monitor.pollinations.ai shows all real-world traffic, including:
openai-audio without modalities param)When you test manually with a valid secret key (sk_), you bypass auth/quota issues, so models appear to work fine.
Key insight: High 401/402/403/400 rates are expected from real-world usage. Focus investigation on 500/504 errors.
User Request → enter.pollinations.ai (Cloudflare Worker)
↓
Logs to Cloudflare Workers Observability
↓
Events stored in D1 database
↓
Batched to Tinybird (async, 100-500 events)
↓
Model Monitor queries Tinybird (model_health.pipe)
Structured Logging: enter.pollinations.ai uses LogTape with:
requestId: Unique per request (passed to downstream via x-request-id header)status, body: Full error response from downstream servicesmethod, routePath, userAgent, ipAddressView current model health at: https://monitor.pollinations.ai
# Via enter.pollinations.ai worker (requires wrangler)
cd enter.pollinations.ai
npx wrangler d1 execute pollinations-db --remote --command "SELECT model_requested, response_status, error_message, COUNT(*) as count FROM event WHERE response_status >= 400 AND created_at > datetime('now', '-1 hour') GROUP BY model_requested, response_status, error_message ORDER BY count DESC LIMIT 20"
cd enter.pollinations.ai
wrangler tail --format json | tee logs.jsonl
# Or with formatting:
wrangler tail --format json | npx tsx scripts/format-logs.ts
Image and text generation now run inside the gen Cloudflare Worker (the legacy EC2 image-pollinations and text-pollinations services are decommissioned). Use wrangler tail from gen.pollinations.ai/:
cd gen.pollinations.ai
wrangler tail --format json | tee gen-logs.jsonl
Anonymous traffic to image.pollinations.ai still terminates on the OVH host:
# Real-time logs
ssh -i ~/.ssh/id_rsa_ovh ubuntu@57.130.31.42 "sudo journalctl -u image-pollinations -f"
# Last 3 minutes
ssh -i ~/.ssh/id_rsa_ovh ubuntu@57.130.31.42 "sudo journalctl -u image-pollinations --since '3 minutes ago' --no-pager" > legacy-image-logs.txt
Error: getaddrinfo ENOTFOUND gptimagemain1-resource.cognitiveservices.azure.com
Cause: Azure Content Safety resource deleted or misconfigured
Impact: Fail-open (content proceeds without safety check)
Fix: Create new Azure Content Safety resource and update .env:
AZURE_CONTENT_SAFETY_ENDPOINT=https://<new-resource>.cognitiveservices.azure.com/
AZURE_CONTENT_SAFETY_API_KEY=<new-key>
Error: Content rejected due to sexual/hate/violence content detection
Cause: Azure's content moderation blocking prompts/images
Impact: 400 error returned to user
Fix: User error - prompt violates content policy
Error: Provided image is not valid
Cause: User passing unsupported image URL (e.g., Google Drive links)
Impact: 400 error returned to user
Fix: User error - need direct image URL
Error: No active translate servers available
Cause: Translation service unavailable
Impact: Prompts not translated (non-fatal)
Fix: Check translation service status
Error: Invalid value for audio.voice
Cause: User requesting unsupported voice name
Impact: 400 error returned to user
Fix: User error - use supported voices: alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, etc.
Error: No video data in response
Cause: Vertex AI returned empty video response
Impact: 500 error
Fix: Check Vertex AI quota/status, may be transient
Image and text env vars now live in the gen Worker secrets (gen.pollinations.ai/secrets/{dev,staging,prod}.vars.json, SOPS-encrypted). Decrypt to inspect:
sops -d gen.pollinations.ai/secrets/prod.vars.json | jq 'keys[] | select(test("AZURE|GOOGLE|CLOUDFLARE|OPENAI"))'
Key variables:
AZURE_CONTENT_SAFETY_ENDPOINT - Azure Content Safety API endpointAZURE_CONTENT_SAFETY_API_KEY - Azure Content Safety API keyGOOGLE_PROJECT_ID - Google Cloud project for Vertex AIAZURE_MYCELI_PROD_SWEDEN_API_KEY - Shared Azure API key (Kontext, GPT Image, GPT Image 1.5)Secrets are stored encrypted with SOPS:
gen.pollinations.ai/secrets/{dev,staging,prod}.vars.jsonenter.pollinations.ai/secrets/{dev,staging,prod}.vars.jsonTo update:
# Decrypt, edit, re-encrypt
sops gen.pollinations.ai/secrets/prod.vars.json
# Deploy to the gen Worker (secrets ship with the deploy)
cd gen.pollinations.ai && npm run deploy
# Count errors by type (against captured wrangler-tail JSON)
jq -r '.logs[]?.message[]? // .message? // empty' gen-logs.jsonl | grep -oE "(Azure Flux Kontext|Vertex AI|No active translate|getaddrinfo ENOTFOUND)" | sort | uniq -c | sort -rn
# Find content filter rejections
jq -r '.logs[]?.message[]? // .message? // empty' gen-logs.jsonl | grep -i "Content rejected" | sort | uniq -c
| Model | Backend | Common Issues |
|---|---|---|
flux | Azure/Replicate | Rate limits, content filter |
kontext | Azure Flux Kontext | Content filter (strict) |
nanobanana | Vertex AI Gemini | Invalid image URLs, content filter |
seedream-pro | ByteDance ARK | NSFW filter, API key issues |
veo | Vertex AI | Quota, empty responses |
openai-audio | Azure OpenAI | Invalid voice names |
deepseek | DeepSeek API | Rate limits, API key |
The enter.pollinations.ai worker has structured logging enabled. You can query logs programmatically via the Cloudflare Workers Observability API.
# From wrangler.toml
grep account_id enter.pollinations.ai/wrangler.toml
# Or from existing .env
grep CLOUDFLARE_ACCOUNT_ID image.pollinations.ai/.env
Via Cloudflare Dashboard:
Workers Observability ReadThe token is stored in SOPS-encrypted secrets:
enter.pollinations.ai/secrets/env.jsonCLOUDFLARE_OBSERVABILITY_TOKENTo add/update:
# Step 1: Decrypt to temp file
cd /path/to/pollinations
sops -d enter.pollinations.ai/secrets/env.json > /tmp/env.json
# Step 2: Add the token (use jq)
jq '. + {"CLOUDFLARE_OBSERVABILITY_TOKEN": "your_token"}' /tmp/env.json > /tmp/env_updated.json
# Step 3: Re-encrypt (must rename to match .sops.yaml pattern)
cp /tmp/env_updated.json /tmp/env.json
sops -e /tmp/env.json > enter.pollinations.ai/secrets/env.json
# Step 4: Cleanup
rm /tmp/env.json /tmp/env_updated.json
# Verify
sops -d enter.pollinations.ai/secrets/env.json | jq 'keys'
Note: The .sops.yaml config requires filenames matching env.json$ pattern.
POST https://api.cloudflare.com/client/v4/accounts/{account_id}/workers/observability/telemetry/query
# Extract credentials from encrypted secrets
ACCOUNT_ID=$(sops -d enter.pollinations.ai/secrets/env.json | jq -r '.CLOUDFLARE_ACCOUNT_ID')
API_TOKEN=$(sops -d enter.pollinations.ai/secrets/env.json | jq -r '.CLOUDFLARE_OBSERVABILITY_TOKEN')
This endpoint works and shows what fields are available:
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/keys" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"timeframe": {"from": '$(( $(date +%s) - 86400 ))'000, "to": '$(date +%s)'000}, "datasets": ["workers"]}' | jq '.result[:10]'
Note: The /query endpoint requires a saved queryId. For ad-hoc queries, use the Cloudflare Dashboard Query Builder or wrangler tail.
# This format requires a saved query ID
# Query errors with status >= 400
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"timeframe": {
"from": '$(( $(date +%s) - 900 ))'000,
"to": '$(date +%s)'000
},
"parameters": {
"datasets": ["workers"],
"filters": [
{"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"},
{"key": "$metadata.statusCode", "operation": "gte", "type": "number", "value": 400}
],
"calculations": [{"operator": "count"}],
"groupBys": [
{"type": "string", "value": "$metadata.statusCode"},
{"type": "string", "value": "$metadata.error"}
],
"limit": 50
}
}' | jq '.result.events.events[:20]'
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"timeframe": {
"from": '$(( $(date +%s) - 3600 ))'000,
"to": '$(date +%s)'000
},
"parameters": {
"datasets": ["workers"],
"filters": [
{"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"},
{"key": "$metadata.statusCode", "operation": "gte", "type": "number", "value": 400}
],
"calculations": [{"operator": "count"}],
"groupBys": [
{"type": "string", "value": "model"},
{"type": "string", "value": "$metadata.statusCode"}
],
"limit": 100
}
}' | jq '.result.calculations[0].aggregates'
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"timeframe": {
"from": '$(( $(date +%s) - 900 ))'000,
"to": '$(date +%s)'000
},
"parameters": {
"datasets": ["workers"],
"filters": [
{"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"},
{"key": "$metadata.statusCode", "operation": "gte", "type": "number", "value": 500}
],
"limit": 20
}
}' | jq '.result.events.events[] | {
timestamp: .timestamp,
statusCode: ."$metadata".statusCode,
error: ."$metadata".error,
message: ."$metadata".message,
requestId: ."$workers".requestId,
url: ."$metadata".url
}'
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/keys" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"timeframe": {
"from": '$(( $(date +%s) - 3600 ))'000,
"to": '$(date +%s)'000
},
"datasets": ["workers"],
"filters": [
{"key": "$workers.scriptName", "operation": "eq", "type": "string", "value": "enter-pollinations-ai"}
]
}' | jq '.result.keys'
The worker uses LogTape for structured logging with these key fields:
Downstream errors are logged with:
log.warn("Chat completions error {status}: {body}", {
status: response.status,
body: responseText,
});
For aggregated model health stats, query Tinybird directly:
# Get model health stats (last 5 minutes)
curl "https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_health.json?token=$TINYBIRD_TOKEN" | jq '.data'
# Get detailed error breakdown
curl "https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_errors.json?token=$TINYBIRD_TOKEN" | jq '.data'
The Tinybird token is a read-only public token found in:
apps/model-monitor/src/hooks/useModelMonitor.jsWorkspace: The hardcoded
TINYBIRD_TOKENbelow and the public token inuseModelMonitor.jsboth target the prod workspace (pollinations_enter). For debugging staging-only issues, swap in a read token from the staging workspace (pollinations_enter_staging) — same URL, same pipe name, different data.
Check Model Monitor - https://monitor.pollinations.ai
Query Cloudflare Logs - Use the API queries above
Correlate with Request ID - If you have a specific request ID:
# Filter by request ID
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workers/observability/telemetry/query" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"timeframe": {"from": '$(( $(date +%s) - 86400 ))'000, "to": '$(date +%s)'000},
"parameters": {
"datasets": ["workers"],
"filters": [
{"key": "$workers.requestId", "operation": "eq", "type": "string", "value": "REQUEST_ID_HERE"}
],
"limit": 100
}
}' | jq '.result.events.events'
Check Gateway Logs - Tail the gen Worker (image + text both run here):
cd gen.pollinations.ai && wrangler tail --format json | tee gen-logs.jsonl
Test Model Directly - Verify if model is actually broken:
TOKEN=$(grep ENTER_API_TOKEN_REMOTE enter.pollinations.ai/.testingtokens | cut -d= -f2)
# Test text model
curl -s 'https://gen.pollinations.ai/v1/chat/completions' \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"model": "MODEL_NAME", "messages": [{"role": "user", "content": "Test"}]}' \
-w "\nHTTP: %{http_code}\n"
# Test image model
curl -s 'https://gen.pollinations.ai/image/test?model=MODEL_NAME&width=256&height=256' \
-H "Authorization: Bearer $TOKEN" \
-w "\nHTTP: %{http_code}\n" -o /dev/null
What works:
/telemetry/keys - List available log fields ✅/telemetry/values - Get unique values for a field ✅enter.pollinations.ai/secrets/env.json ✅Limitations:
/telemetry/query requires a saved queryId from the dashboardwrangler tail for real-time logsTinybird provides pre-aggregated model health stats and raw event data.
apps/model-monitor/src/hooks/useModelMonitor.jsenter.pollinations.ai/observability/.tinyb (in token field)# Public read-only token from apps/model-monitor
TINYBIRD_TOKEN="p.eyJ1IjogImFjYTYzZjc5LThjNTYtNDhlNC05NWJjLWEyYmFjMTY0NmJkMyIsICJpZCI6ICJmZTRjODM1Ni1iOTYwLTQ0ZTYtODE1Mi1kY2UwYjc0YzExNjQiLCAiaG9zdCI6ICJnY3AtZXVyb3BlLXdlc3QyIn0.Wc49vYoVYI_xd4JSsH_Fe8mJk7Oc9hx0IIldwc1a44g"
# Get model health (last 5 min)
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/pipes/model_health.json?token=$TINYBIRD_TOKEN" | jq '.data'
For querying the raw generation_event datasource, use the admin token from .tinyb:
# Get admin token from .tinyb file
TINYBIRD_ADMIN_TOKEN=$(jq -r '.token' enter.pollinations.ai/observability/.tinyb)
# Find users with frequent 403 errors (last 24 hours)
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN" \
--data-urlencode "q=SELECT user_id, user_github_username, user_tier, count() as error_403_count
FROM generation_event
WHERE response_status = 403
AND start_time > now() - interval 24 hour
AND user_id != ''
AND user_id != 'undefined'
GROUP BY user_id, user_github_username, user_tier
ORDER BY error_403_count DESC
LIMIT 20"
# Find users with 500 errors (actual backend issues)
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN" \
--data-urlencode "q=SELECT user_github_username, model_requested, error_message, count() as error_count
FROM generation_event
WHERE response_status >= 500
AND start_time > now() - interval 24 hour
GROUP BY user_github_username, model_requested, error_message
ORDER BY error_count DESC
LIMIT 20"
# Check specific user's recent errors
curl -s "https://api.europe-west2.gcp.tinybird.co/v0/sql?token=$TINYBIRD_ADMIN_TOKEN" \
--data-urlencode "q=SELECT start_time, response_status, model_requested, error_message
FROM generation_event
WHERE user_github_username = 'USERNAME_HERE'
AND start_time > now() - interval 24 hour
ORDER BY start_time DESC
LIMIT 50"
The generation_event datasource is defined in enter.pollinations.ai/observability/datasources/generation_event.datasource and includes:
user_id, user_github_username, user_tierresponse_status, error_message, error_response_codemodel_requested, model_usedtotal_price, total_coststart_time, end_time, response_timeHelper scripts for common debugging tasks. Run from repo root.
# Find users with >10 403 errors in last 24 hours
.claude/skills/model-debugging/scripts/find-403-users.sh 24 10
# Filter by tier (e.g., only spore users)
.claude/skills/model-debugging/scripts/find-403-users.sh 24 10 spore
# Find 500+ errors grouped by user/model/message
.claude/skills/model-debugging/scripts/find-500-errors.sh 24
# See a user's recent errors
.claude/skills/model-debugging/scripts/check-user-errors.sh superbrainai 24
| Model | Type | Endpoint | Status |
|---|---|---|---|
openai | text | POST /v1/chat/completions | ✅ |
openai-fast | text | POST /v1/chat/completions | ✅ |
openai-large | text | POST /v1/chat/completions | ✅ |
openai-audio | text | GET /text/{prompt}?model=openai-audio&voice=alloy | ✅ (MP3) |
claude | text | POST /v1/chat/completions | ✅ |
gemini-fast | text | POST /v1/chat/completions | ✅ |
flux | image | GET /image/{prompt} | ✅ |
nanobanana-pro | image | GET /image/{prompt} | ✅ |
seedream-pro | image | GET /image/{prompt} | ✅ |
seedance-pro | video | GET /image/{prompt} | ✅ (MP4) |
Add, update, or remove text/image/video/audio/embeddings models. Covers the full lifecycle: files to touch, what to verify, and how to test empirically before merging.
Detect and analyze abusive accounts on Pollinations. IP clustering, multi-signal scoring, ban recommendations. Use when investigating abuse, bot farms, or suspicious usage patterns.
Query billing, usage, credits, and resource deployments across all our cloud and SaaS providers (Azure, AWS, Cloudflare, GCP, Tinybird, Vercel, Stripe, Polar, etc.) via their native CLIs and APIs. Use for any question about provider costs, spend by service/day/month, credit eligibility, invoice totals, which resources are running, or how to deploy/inspect resources. Each provider has a dedicated playbook under `providers/`.
Analyze Pollinations revenue, pack purchases, and tier spending patterns. Query Polar for payment history and Tinybird for usage data.
Deploy Tinybird pipes and datasources for enter.pollinations.ai observability. Validates and pushes changes to Tinybird Cloud.
Generate images, text, audio, video, and transcribe speech via the Pollinations API using the polli CLI. Use when asked to generate media, call pollinations.ai, check pollen balance, list models, manage API keys, or run polli commands.