anki-gui-automation

name	anki-gui-automation
description	Develop and verify HyperTTS GUI screens against a real, running Anki instance: launch Anki headlessly on a throwaway profile, inject notes with AnkiConnect, read the live Qt widget tree as text, drive dialogs, screenshot them, and tear everything down
user_invocable	true

Agentic GUI development for HyperTTS

Use this whenever you add or change a HyperTTS dialog. pytest-qt tests prove the logic; this harness proves the dialog is actually reachable, laid out correctly, and correct against a real Anki collection (real notes, real update_note, real undo).

Everything runs on a virtual display against an isolated Anki base folder and profile, so the developer's own Anki profile and collection are never touched. Both service ports are deliberately non-default (AnkiConnect on 8766, not 8765) so the harness can never talk to a real Anki session that happens to be open.

You must run scripts/gui_automation/teardown.sh when you are done. Leaving Xvfb, x11vnc, websockify and a headless Anki running is the main failure mode of this workflow.

The loop

cd scripts/gui_automation

./start_anki.sh                 # Xvfb + openbox + x11vnc + noVNC + Anki with HyperTTS
./ankiconnect.py seed --reset   # deck + note type + 5 notes with assorted sound tags
./ankiconnect.py browse         # open the browser on those notes
./gui_probe.py browser-select-all

./gui_probe.py actions --text Audio                   # what menu entries exist
./gui_probe.py trigger --text 'Remove Audio (Collection)...'   # open the dialog
./gui_probe.py windows                                # confirm it opened, and is modal
./gui_probe.py tree --window hypertts_remove_audio_dialog      # read the whole dialog as text
./gui_probe.py table --object-name hypertts_remove_audio_preview_table
./gui_probe.py screenshot --path /tmp/hypertts-gui-automation/artifacts/dialog.png \
    --window hypertts_remove_audio_dialog             # then Read the png

./teardown.sh                   # ALWAYS

After editing any HyperTTS python file, run ./start_anki.sh --restart. The add-on is symlinked into the profile, but Anki only imports it at startup. --fresh additionally throws away the collection.

Read text first, screenshot second. tree costs a few hundred tokens and tells you state (enabled, checked, current index, combo items); a screenshot costs thousands but is the only way to catch layout problems. Both matter — the two bugs found while building the Remove Audio dialog (a QLabel rendering <b> as literal text, and preview columns truncated by an even stretch) were invisible in the widget tree and obvious in the screenshot.

Watch it live in a browser at http://localhost:6099/vnc.html while it runs.

Widget naming convention (do this first, it unblocks everything)

Give every widget you create a stable objectName, prefixed hypertts_<screen>_:

self.field_combobox = aqt.qt.QComboBox()
self.field_combobox.setObjectName('hypertts_remove_audio_field')

Name the dialog too (self.setObjectName('hypertts_remove_audio_dialog')) so --window can target it. Without object names you have to address widgets by class + index path (RemoveAudioDialog/QGroupBox[0]/QComboBox[0]), which breaks the moment the layout changes. hypertts_addon/component_remove_audio.py is the reference implementation.

Older components (component_batch.py, component_voiceselection.py, …) have no object names yet. For those, address widgets by --class + --text, or by the path printed by tree.

scripts/gui_automation reference

script	what it does
start_display.sh	Xvfb :99, openbox, x11vnc 5999, noVNC 6099. Idempotent.
start_anki.sh [--restart|--fresh]	prepares the profile, launches Anki, waits for both ports
setup_profile.py	symlinks HyperTTS + installs AnkiConnect + anki_gui_probe, creates the profile
stop_anki.sh	stops only Anki, leaves the display up
teardown.sh	stops everything, cleans the X lock
status.sh	what is running, which ports answer, which windows are open
screenshot.sh [name]	full-screen grab into the artifacts dir (includes window decorations)
ankiconnect.py	inject/read collection data
gui_probe.py	inspect and drive the live GUI

Runtime state lives under /tmp/hypertts-gui-automation: anki_base/ (throwaway collection and add-ons), logs/anki.log, logs/hypertts.log (HyperTTS debug logging is on), artifacts/ (screenshots), pids/.

gui_probe.py

Inspect:

• windows — every top-level window: class, title, modal, active, geometry
• tree [--window W] [--named-only] [--all] [--max-depth N] — indented widget tree
• info --object-name X — one widget in detail
• table --object-name X — dump a QTableView/QTreeView model as rows (QVariant unwrapped)
• actions [--text substring] — every QAction, i.e. every menu entry
• undo-status — the label of the next undoable operation
• note-fields --note-id N — a note's fields straight from the collection

Drive:

• click --object-name X [--no-wait] — also --class QPushButton --text Cancel
• set-text --object-name X --value '...'
• combo --object-name X --text '...' (or --index N)
• check --object-name X [--off]
• select-row --object-name X --row N
• trigger --text 'Menu entry...' — opens dialogs; does not wait, by design
• close --title '...' [--class ...], undo, browser-select-all, browser-search --query ...
• screenshot --path P [--window W] — QWidget.grab() of one window
• raw '{"action": "eval", "params": {"expression": "mw.pm.name"}}' — escape hatch, main-thread eval with aqt, qt and mw in scope

Modal dialogs block the Qt main thread. Any action that opens one must not wait for the main thread, or the request times out. trigger defaults to not waiting; pass --no-wait to click when the click opens a dialog (including a QMessageBox such as HyperTTS's "Save changes to current preset ?" confirmation) or closes the window it lives on. When a probe call returns {"status": "pending"} that is not an error: a modal is open and holding the main thread — call windows to see it and deal with it.

Anki's own dialogs are reachable the same way, which is how you dismiss things like the add-on startup error box:

./gui_probe.py tree --window QMessageBox
./gui_probe.py click --class QPushButton --text '&No' --window QMessageBox --no-wait

ankiconnect.py

• seed [--reset] — creates the deck HyperTTS Automation, the note type HyperTTS Automation Note (fields Chinese / English / Sound / Sound English), stores dummy media files, and adds 5 notes: a lone HyperTTS sound tag, text plus a HyperTTS sound tag, a foreign sound tag (external-recording.mp3, i.e. audio HyperTTS did not generate), HyperTTS audio in two fields, and a note with no audio. Extend seed_notes() for new cases.
• notes [--query Q] — dump note fields, the cheapest way to assert what an operation did
• browse [--query Q] — open the browser on a query
• invoke <action> [--params JSON] — any AnkiConnect action

Two AnkiConnect gotchas, already handled in the helper but worth knowing:

• AnkiConnect is unmaintained and still assigns the deck through the legacy note-type dict, which modern Anki ignores, so addNotes lands cards in Default. seed moves them with changeDeck.
• Anki search syntax wants the quotes around the whole term: "deck:HyperTTS Automation", not deck:"HyperTTS Automation".

Verifying a feature end to end

The pattern that actually proves a collection-modifying dialog works — this is how the Remove Audio dialog was validated:

./ankiconnect.py notes            # state before
./gui_probe.py undo-status        # "" - nothing to undo yet
./gui_probe.py click --object-name hypertts_remove_audio_remove_button
./ankiconnect.py notes            # state after: only the intended fields changed
./gui_probe.py undo-status        # "HyperTTS: Remove Audio from Notes" <- undo support works
./gui_probe.py undo
./ankiconnect.py notes            # back to the original state

Undo support in HyperTTS comes from running the mutation inside anki_utils.run_in_background_collection_op(parent, update_fn, success_fn, undo_entry_name=...), which wraps it in a custom undo entry. update_fn receives the collection and must call collection.update_note(note); never call aqt.mw.col directly from a dialog.

pytest-qt tests are still required

The harness complements the test suite, it does not replace it. Every new dialog needs a tests/test_component_<name>.py following the existing pattern:

• build a mock instance with testing_utils.TestConfigGenerator().build_hypertts_instance_test_servicemanager('default')
• component-level tests: gui_testing_utils.build_empty_dialog(), then component.draw(dialog.getLayout())
• full-workflow tests: register a dialog_input_fn_map[constants.DIALOG_ID_<X>] callback and call the create_component_* factory
• one test_<name>_manual guarded by HYPERTTS_<X>_DIALOG_DEBUG=yes which calls dialog.exec(), and a matching entry in scripts/openbox_menu_hypertts so the dialog can be eyeballed by hand

Run pytest -n auto before finishing. The tests/test_tts_services/ tests hit real TTS APIs and a couple can fail for unrelated network/speech-recognition reasons.

Troubleshooting

• "Add-on Startup Failed" — HyperTTS raised during import. The add-on folder must be named anki-hyper-tts (it is what constants.CONFIG_ADDON_NAME looks up; any other name makes getConfig() return None). Read the message with ./gui_probe.py raw '{"action":"eval","params":{"expression":"[t.toPlainText() for w in qt.QApplication.topLevelWidgets() for t in w.findChildren(qt.QTextBrowser)]"}}'
• probe port not answering — tail -40 /tmp/hypertts-gui-automation/logs/anki.log
• port already in use — a previous run was not torn down: ./teardown.sh
• table dump shows QVariant objects — HyperTTS models return QVariant; the probe unwraps them, so this means the probe is stale: ./start_anki.sh --restart
• import/screenshot is black — nothing is mapped on the display yet, or Anki is still starting; check ./status.sh

Requirements

System packages (Fedora, installed with sudo dnf install; see docs/AI_GUI_AUTOMATION_SETUP.md): xorg-x11-server-Xvfb openbox x11vnc novnc python3-websockify xdotool wmctrl ImageMagick

The helper scripts only use the python standard library, so there are no additions to requirements.txt. xdotool/wmctrl are available for real X11 input events if a widget ever resists QWidget.click().

The harness deliberately does not use the AT-SPI accessibility tree — the probe is cheaper and more precise — and start_anki.sh exports QT_ACCESSIBILITY=0 / NO_AT_BRIDGE=1 so Qt does not publish the widget tree over D-Bus. Do not install dbus-x11 / at-spi2-core for this workflow: nothing here needs them.