wolfmanstout/talon-gaze-ocr

Latest recorded source activity: September 18, 2026 at 05:34

SkillsMP catalog refreshed: October 29, 2026 at 18:30

skills collected: 1

GitHub stars: 79
GitHub forks: 35

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/wolfmanstout/talon-gaze-ocr.

Repository README

Repository README source: README.md

Read the source README

Expand full README

# talon-gaze-ocr [Talon](https://talonvoice.com/) scripts to enable advanced cursor control using eye tracking and text recognition (OCR). This is alpha functionality which uses experimental/unsupported APIs, so it could break at any time. See [my blog post](https://handsfreecoding.org/2022/11/27/gaze-ocr-talon-support-and-10-new-features/) for an overview. ## Installation Prerequisite: Requires [community](https://github.com/talonhub/community) to be installed as a sibling in the same directory. `git clone` this repo into your Talon user scripts folder. If you are on Talon Beta, switch to the "beta" branch (add flag `-b beta`) to take advantage of beta-only features. If you use [Talon HUD](https://github.com/chaosparrot/talon_hud), consider setting `user.talon_hud_allows_capture = false` to make the HUD invisible to OCR (only supported on Talon Beta). Restart Talon to ensure that the package is fully reloaded. Unlike most Talon scripts, this package requires Talon to be restarted when it is updated or settings are changed. Required permissions: - On Mac, Talon requires the ability to read the screen. As per the [macOS User Guide](https://support.apple.com/guide/mac-help/control-access-to-screen-recording-on-mac-mchld6aa7d23/mac): 1. Choose Apple menu > System Settings, then click Privacy & Security in the sidebar. (You may need to scroll down.) 2. Click Screen Recording. 3. Turn screen recording on for Talon. Required Python packages: - All Python package dependencies are present in the `.subtrees` directory so that no pip installations are needed. If packages are have been installed manually through pip, these will be preferred (e.g. so that faster binary packages can be used.) Note: This installation process is for Windows and Mac. Linux is not officially supported because Talon does not provide an OCR API. You may be able to get this working using Tesseract, but this configuration is not tested by the author. ### Migrating from `beta-pi` If you use the `beta-pi` branch, switch to `beta`: `beta-pi` has been merged and retired. Update to the latest Talon Beta, then run these commands in your `talon-gaze-ocr` directory and restart Talon: ```sh git fetch origin git switch beta git pull --ff-only ``` For single-branch clones, run `git remote set-branches origin beta` before fetching. If you need older Talon Beta support temporarily, check out the [`beta-before-pi` tag](https://github.com/wolfmanstout/talon-gaze-ocr/tree/beta-before-pi). ## Features: - Click, select, or position caret adjacent to any text visible onscreen. - Tracks eye position as you speak to filter matches. - Offers disambiguation if multiple matches are present. - Applies fuzzy matching to one or more spoken words. - Works with or without an eye tracker (just expect slower processing and more disambiguation). - Matches homophones of recognized words (based on CSV in [community](https://github.com/talonhub/community)). Also matches digits and punctuation in either their spoken form (e.g. "two" and "period") or their symbolic form (e.g. "2" and "."). - Briefly displays debugging overlay if no matches are present. - Beta-only: match on any `<user.prose>`, including numbers. - Beta-only: match on any onscreen words, even outside of your vocabulary. - Beta-only: [intuitive editing commands](https://handsfreecoding.org/2024/03/15/making-writing-and-editing-with-your-voice-feel-natural/) including "revise" and "insert". ## Scrolling Visualization Scroll commands (`eye scroll up`, `eye scroll down`, `eye scroll up half`, `eye scroll down half`) provide two key improvements: - **Consistent scroll distance**: Scrolls a fixed fraction of the viewport regardless of window height. - **Visual indicator**: Shows how content shifted so you don't lose your place. See [my blog post on voice-native scrolling](https://handsfreecoding.org/2026/02/15/voice-native-scrolling-in-talon-gaze-ocr/) for details on this feature. To enable unprefixed scroll commands (e.g. `scroll up`, `scroll down half`), add this tag to your Talon settings: ```talon tag(): user.gaze_ocr_unprefixed_scroll ``` ### Enable faster scrolling on Windows and Linux On Windows and Linux (not Mac), browsers use smooth scrolling by default, which slows down scroll detection. To speed this up, disable smooth scrolling and add a tag to your Talon settings: **Chromium (Chrome, Edge, Brave, Vivaldi, Opera):** 1. Navigate to `chrome://flags/#smooth-scrolling` 2. Set "Smooth Scrolling" to **Disabled** 3. Restart your browser **Firefox:** 1. Navigate to `about:preferences` 2. Search for "smooth scrolling" 3. Uncheck "Use smooth scrolling" Then add to your Talon settings (e.g., settings.talon in [community](https://github.com/talonhub/community)): ```talon tag(): user.browser_smooth_scrolling_disabled ``` ## Known limitations: - Only operates on the main screen, as defined by Talon. - Updates (via git pull) and some settings changes require Talon restart. - Numbers must be referred to by their individual digits (except in Beta where you can say `numb <number>`). - Modifications to punctuation and digit names in [community](https://github.com/talonhub/community) not leveraged. - Depends on OS-provided text recognition (OCR), which is not perfectly accurate. - Cursor positioning often imprecise around text with underline, strikethrough, or squiggly. - Text caret can interfere with text recognition. - Command subtitles may cause disambiguation when selecting a range of text. - Dragon recognition timestamps are slightly off, leading to lower accuracy especially during text selection. Works best with Conformer. - Text selection only works in apps that support shift-click to select (vs. dragging), which excludes some PDF viewers. - See the [issue tracker](https://github.com/wolfmanstout/talon-gaze-ocr/issues) for bugs that have been discovered. ## Dependencies The .subtrees directory contains dependency packages needed by talon-gaze-ocr: - gaze-ocr was cloned from https://github.com/wolfmanstout/gaze-ocr - screen-ocr was cloned from https://github.com/wolfmanstout/screen-ocr - rapidfuzz was cloned from https://github.com/maxbachmann/RapidFuzz Changes in this repository are automatically pushed out to `gaze-ocr` and `screen-ocr` via a GitHub Action that uses `git subtree`. ## Running without `community` As noted in the installation instructions, [community](https://github.com/talonhub/community) is highly recommended, but most functionality will still be available in degraded form without it (and you will see some warning logs). Missing functionality: - The main `user.timestamped_prose` capture is missing custom vocabulary and punctuation support. - No homophones means no automatic smart handling of homophones (e.g. if "hear" is recognized it won't match "here" onscreen). - "scroll" commands use `user.mouse_scroll_*()`, so they won't work. - "replace" and "say" commands use `user.prose` to insert text, so they won't work. - `actions.user.dictation_peek` not available means text deletion/replacement isn't as smart (i.e. extra space is left over when a word is deleted).

Loading the repository README…

Skills in this repository

Network & Computer Systems Administrators

1 occupation category · 100% classified

skill

occupation

description

updated

commit-main

Network & Computer Systems Administrators

Commit stable-compatible changes to main, merge main into beta, and rebase any feature branch onto beta.

2026-09-18

Showing 1 of 1 collected skill.