scrask-bot

name	scrask-bot
version	3.0.0
description	When the user sends a screenshot via Telegram, parse it using Gemini (fast, default) with automatic Claude fallback when confidence is low. Saves results to Google Calendar (events) or Google Tasks (reminders and tasks). Saves silently if confidence is high; asks for confirmation if uncertain.
author	your-name
metadata	{"openclaw":{"emoji":"🦞","primaryEnv":"GEMINI_API_KEY","requires":{"env":["GOOGLE_CREDENTIALS","GEMINI_API_KEY"],"bins":["python3"]},"config":{"vision_provider":{"type":"string","description":"Vision model provider. 'auto' = Gemini first, Claude fallback if any item confidence < fallback_threshold. 'gemini' = Gemini only. 'claude' = Claude only.\n","default":"auto"},"fallback_threshold":{"type":"number","description":"Confidence floor for auto mode. If any item is below this, Claude reruns the parse.","default":0.6},"timezone":{"type":"string","description":"User's IANA timezone. Used when none is detected in the screenshot.","default":"UTC"},"confidence_threshold":{"type":"number","description":"0.0–1.0. Items below this score ask for confirmation. Items above save silently.","default":0.75},"reminder_minutes_before":{"type":"integer","description":"Popup reminder lead time in minutes for Google Calendar events.","default":30}}}}

Scrask Bot

Overview

This skill activates when the user sends a screenshot via Telegram. It uses vision AI to extract actionable information from the image, then:

• High confidence (≥ 0.75): Saves immediately and replies with a brief confirmation.
• Low confidence (< 0.75): Shows a structured preview in Telegram and asks for confirmation before saving.

Provider behaviour (auto mode, default):

Step	What happens
1	Gemini 2.0 Flash parses the screenshot (fast, cheap)
2	If any item confidence < 0.60, Claude Opus reruns the parse
3	Whichever provider scores higher average confidence wins
4	Output includes provider, fallback_triggered, and confidence delta

Set vision_provider to "claude" or "gemini" to lock a specific provider.

Output destinations (AI-decided by content type):

Detected type	Destination
Event (has date+time, venue, or invite link)	Google Calendar
Reminder (deadline, due date, personal action)	Google Tasks (with due date)
Task (no date, pure action item)	Google Tasks (no due date)

---

Trigger Conditions

Activate when:

1. The user sends a message in Telegram that contains an image attachment
2. The image appears to be a screenshot — not a photo of a person, place, or physical object
3. No other skill has already claimed the image

Do not activate for:

• Photos of people, places, food, scenery
• Screenshots of code, errors, or UI bugs (leave for other skills)
• Images the user explicitly asks to edit, describe, or analyze for another purpose

---

Step-by-Step Instructions

Step 1: Acknowledge Immediately

Reply in Telegram right away so the user knows the skill is working:

"📸 Got it — analyzing your screenshot..."

Do not make the user wait silently.

---

Step 2: Run the Parser

python3 ~/.openclaw/skills/scrask-bot/scripts/scrask_bot.py \
  --image-path "<path-to-temp-image>" \
  --provider "$CONFIG_VISION_PROVIDER" \
  --timezone "$CONFIG_TIMEZONE" \
  --google-credentials "$GOOGLE_CREDENTIALS"

The script auto-resolves the API key from ANTHROPIC_API_KEY or GEMINI_API_KEY
depending on the provider — no need to pass it explicitly.

The script returns a JSON object with:

• success — whether parsing worked
• no_actionable_content — true if nothing found
• results[] — one entry per detected item, each with confidence, type, destination, needs_confirmation, action_taken
• telegram_reply — the pre-formatted message to send back to the user

---

Step 3: Handle the Output

If no_actionable_content is true: Reply: "🤷 I couldn't find any event, reminder, or task info in that screenshot. Could you describe what you'd like to add?"

If success is true: Send the telegram_reply value directly back to the user in Telegram. The script has already:

• Saved high-confidence items silently
• Formatted confirmation prompts for low-confidence items

Do not rephrase or reformat the telegram_reply — send it as-is.

---

Step 4: Handle Confirmation Responses

If the script returned items with needs_confirmation: true, wait for the user's reply.

"yes" or "save" or "add": Re-run the script for that specific item with confirmed=true, or use the calendar_create / tasks_create tools directly with the extracted fields.

"edit": Ask what to change, update the relevant field, then save.

"skip" or "no": Reply: "Got it, skipped ✓"

---

Step 5: Confirm Saves

For items saved silently (high confidence), the telegram_reply from the script already contains the confirmation message. Examples of what the user will see:

• 📅 Added to Calendar: **Team Standup** — 2026-03-01 at 09:00
• 🔔 Added to Tasks: **Pay electricity bill** (due 2026-02-28)
• ✅ Added to Tasks: **Review PR for Arjun**

---

Edge Cases

Scenario	Behavior
Screenshot is in Hindi, Tamil, or another language	Extract and translate silently; save title in English
Recurring event ("every Monday")	Set RRULE on the calendar event; mention it in the reply
Date has already passed	Flag in the reply: "⚠️ This date has already passed (Feb 10). Save anyway?"
Multiple items in one screenshot	Process each independently; confirm per item if needed
Screenshot of someone's calendar	Detect already_in_calendar_hint; reply: "Looks like this event is already in your calendar 🗓️"
Google API auth failure	Reply with the specific error and suggest re-checking GOOGLE_CREDENTIALS
Zoom/Meet link found	Add to Calendar as both location and description

---

Configuration

{
  "skills": {
    "entries": {
      "scrask-bot": {
        "enabled": true,
        "env": {
          "GEMINI_API_KEY": "AIza-your-gemini-key",
          "ANTHROPIC_API_KEY": "sk-ant-your-key-here",
          "GOOGLE_CREDENTIALS": "/home/user/.openclaw/google-creds.json"
        },
        "config": {
          "vision_provider": "auto",
          "fallback_threshold": 0.60,
          "timezone": "Asia/Kolkata",
          "confidence_threshold": 0.75,
          "reminder_minutes_before": 30
        }
      }
    
  

---

Permissions Required

• image:read — to access the screenshot from Telegram
• network:outbound — to call Anthropic API and Google APIs
• telegram:reply — to send confirmation messages back to the user