| name | xvfb |
| description | Linux GUI app automation on a virtual X display. Launch non-browser Linux GUI apps in Docker, see screenshots, and interact by pixel coordinates. |
Xvfb GUI Automation
When to use
Use xvfb_gui for non-browser Linux GUI programs inside Docker: small desktop utilities, installers, dialogs, editors, or test apps that need an X display.
For web browsing, use browser-use. Do not route normal browser work through Xvfb.
Prerequisites
- Docker runtime only. Xvfb is Linux-only, and the tool is not exposed on macOS.
- The Docker image bundles
Xvfb, xdotool, xterm, x11-apps, and ImageMagick import.
Mental model
One Xvfb display is shared by launched apps. The tool returns screenshots when the model supports vision. Coordinates are pixels from the top-left of the root window.
Core Workflow
1. xvfb_gui action=launch app="xterm"
-> auto-creates the Xvfb session, launches the app, returns a screenshot
2. Look at the screenshot and choose pixel coordinates.
3. xvfb_gui action=click x=420 y=310
-> clicks and returns a new screenshot
4. xvfb_gui action=type text="echo hello"
5. xvfb_gui action=key key="Return"
6. xvfb_gui action=close
Always verify with a screenshot after meaningful actions.
Actions
# App lifecycle
xvfb_gui action=launch app="<command with args>" # e.g. "xterm" or "xclock"
xvfb_gui action=kill_app pid=<pid>
# Observation and interaction
xvfb_gui action=screenshot
xvfb_gui action=click x=420 y=310 [button=1|2|3] [double=true] [settle_ms=1000]
xvfb_gui action=drag x1=100 y1=200 x2=500 y2=400 [settle_ms=1000]
xvfb_gui action=type text="hello world"
xvfb_gui action=key key="Return"
xvfb_gui action=scroll x=640 y=400 [amount=3] [direction=up|down]
# Cleanup
xvfb_gui action=close
All interaction actions accept settle_ms in milliseconds. It defaults to 700 and is capped at 10000.
Tips
- Screenshot first; do not interact blindly.
- Use
key, type, click, drag, and scroll; there are no app-specific shortcuts.
- Use
settle_ms for slow dialogs or apps that animate.
- Call
close when done so the Xvfb process and launched apps are cleaned up.