github-copilot-app-automation

# GitHub Copilot app Automation Use this skill when the task is about **driving, testing, mapping, or experimenting with the GitHub Copilot app**. Prefer app/session APIs first, then app-native canvases/automations, and use GUI Accessibility only when the visible desktop UI itself must be exercised. ## Where to Run This This skill's power depends heavily on where the agent runs: | Context | Session/app APIs | Capture reliability | Best for | |---|---|---|---| | **Inside the GitHub Copilot app agent** (recommended) | Available for the host instance. They do not select a separately launched persona. | High when the agent launches, drives, and captures the persona by process ID | End-to-end screenshot capture with a dedicated persona | | **GitHub Copilot CLI** (terminal) | Not available | Low/manual — a human must navigate the app, and its window must be on the same macOS Space as the terminal | Read-only mapping (`map-app.sh`), the dynamic `screenshots.sh` list/embed flow, and one-off captures of an already-visible window | For course screenshots, **drive this from inside the app**, but do not use the host instance as the screenshot target. Launch the dedicated persona below, then use process-scoped Accessibility and capture. The shell scripts here run in both contexts because the app agent is built on Copilot CLI. Regardless of context: pixels still come from `screencapture` (there is no pure render/export API yet, so the window must be visible), Screen Recording permission is required, captures must come from a **sanitized training account** on the training fork, and advanced features (Agent Merge, cloud automations) stay policy/billing-gated. For each screenshot, launch a new app process from the persistent, signed-in `demo` persona instead of creating a new persona or reusing an open process: ```bash SK=.github/skills/github-copilot-app-automation/sample_codes/macos-accessibility PERSONA="demo" # The local clone of the training fork, not the upstream course repository. REPO="$HOME/copilot-app-for-beginners" COPILOT_PID="$(bash "$SK/prepare-persona.sh" "$PERSONA" "$REPO" 60)" ``` This keeps the existing login and persona data, starts a separate app process, sets only that window to a standard 1920x1080-point frame, resets the display zoom and zooms in four times (the zoom is measured and confirmed), enables and verifies **Streamer Mode**, verifies that the account is signed in, verifies the exact local repository, dismisses transient banners, and returns its process ID. If the repository is missing, the pre-step opens the exact folder through the native folder picker and verifies it before capture. A setup failure keeps the process for inspection and does not return a successful process ID. Do not select a target by app name when another Copilot instance is open. Do not use full screen: it can draw the menu bar and title bar over the app content. To keep working on other displays during capture, set `COPILOT_CAPTURE_DISPLAY=builtin` for both `prepare-persona.sh` and `capture-window.sh`. The window then opens on the built-in laptop display and zooms in three times instead of four, because the laptop frame is smaller. After setup, focus goes back to the app that you used before. The pointer is not moved, and the capture does not activate the persona. Do not type during the first launch seconds, and keep the laptop display on its current Space. Before preparing the visible state, extract the Markdown around the image reference: ```bash python3 "$SK/screenshot-context.py" <chapter>/README.md <image-name> ``` Use the section heading, instructions before the image, and instructions after the image to define the exact screen, open menu, selected item, and callouts. Do not capture a generic approximation. Confirm that current Accessibility labels agree with the text. If the app label changed, update the directly related course text before capture. Add numbered red callouts when one screenshot shows two or more controls that the reader must select in order. The nearby instructions must identify each number, for example, "select **+** (callout 1), then select **Add GitHub repository** (callout 2)." Do not add callouts to a single-action screenshot, an output example, or an evidence screenshot. To point at one item without a sequence, such as a menu item to select, use a red highlight box (`--box`) instead. To point at one small status or control, such as a status pill, use a red arrow (`--arrow TAIL_X:TAIL_Y:HEAD_X:HEAD_Y`). Recreate the callouts, boxes, or arrows of an existing image when you retake it, at the new positions of the same controls. Keep the process ID and use it for all UI operations and capture. Do not use `navigate_to` or other app/session APIs to prepare this window because those APIs can route to the instance that hosts the agent. Drive the persona through Accessibility selectors scoped to its process ID. An isolated `HOME` does not isolate credentials stored in the macOS Keychain. The persona can still show the current GitHub account name and avatar. Process-specific capture automatically derives the visible profile name from the accessibility tree and replaces that text with **Copilot Dev**. If an account handle or repository owner matches the normalized profile name, it replaces that text with **copilotdev**, including inside paths and branch names. The macOS account name and machine name, which terminal prompts and paths can show, become **copilotdev** and **copilot-dev-mac**. It preserves the avatar and unrelated people and organizations. On a Settings screen, it also removes the displayed app version. Never hard-code a person's name or an app version. Never treat the persona name alone as proof that the capture is sanitized. Zoom in on every replaced span: a partial replacement can leave letters of the private name next to the new text. ## Non-Interactive / Hidden-Session Rule If the user asks for a **fresh**, **separate**, or **hidden** Copilot App instance/session: - Stay on app/session APIs: `list_projects`, `create_session`, `send_session_message`, `get_session`, and filesystem verification. - Do **not** call `navigate_to`, foreground `open -a`, menu-bar actions, folder pickers, or Accessibility clicks unless the user explicitly asks to see the App UI. - Do **not** make the user select a folder or project. If a project cannot be registered through available non-interactive surfaces, report that limitation instead of opening a visible picker. - For local folders, first make the target folder a distinct repo root when needed (`git init` in the target folder) so the App can distinguish it from a parent repository. - If a hidden registration attempt is necessary, prefer `open -g -j -a "GitHub Copilot" <folder>` and immediately verify with `list_projects`; if it does not register the project, stop and report the gap. - A screenshot of a hidden/minimized/background-only session is not possible with macOS `screencapture`; screenshot capture requires a visible window or a future App-provided render/export API. Do not switch the user's current App window just to satisfy a hidden screenshot request. ## Operating Model Work in layers, from safest to most brittle: 1. **Session orchestration tools**: list projects/sessions, create isolated sessions, message sessions, inspect sessions, navigate to sessions. 2. **App-native surfaces**: canvases, automations, project settings, MCP, skills, issue/PR sessions. 3. **Copilot CLI remote control**: useful for CLI sessions via GitHub.com/GitHub Mobile, not for native app GUI control. 4. **macOS Accessibility/UI scripting**: use AppleScript/System Events for named controls such as `Message`, `Select model`, and `Open changes`. 5. **Screenshot capture**: use Accessibility window bounds plus `screencapture` to save PNGs, then convert to WebP for web delivery. 6. **Low-level GUI clicking**: avoid unless the user explicitly approves and no named accessibility control exists. ## Safety Rules - **Create a separate session for experiments** unless the user explicitly wants the current session modified. - For hidden/separate-session tasks, **never navigate the user's visible App window** unless explicitly requested. - **Never submit a prompt, create a PR, merge, delete, approve permissions, or enable Agent Merge** without explicit user intent. - For GUI probes, start read-only: enumerate windows, controls, roles, names, and actions. - For composer tests, type a clearly marked draft and clear it without pressing Return/Enter. - Avoid pixel-coordinate clicks. Target controls by accessibility role/name/action. - Do not persist secrets or private UI data in repo files. Store scratch probes in `/tmp` or the session artifact directory. - Screenshots may contain private repo names, prompts, diffs, terminal output, or tokens. Capture only the requested app state and save course-ready images under an intentional assets folder. - Expect transient app/window states. Re-discover the window and `AXWebArea` before each UI operation. - If Accessibility is denied, stop and ask the user to grant access to the automation caller. Do not try to bypass macOS privacy controls. - If `screencapture` fails or returns a privacy prompt, the automation caller also needs macOS Screen Recording permission. ## Known Local App Facts Observed on this machine: - App path: `/Applications/GitHub Copilot.app` - Bundle id: `com.github.githubapp` - Bundle version: read dynamically with `map-app.sh`; do not rely on a stored version - Executable: `/Applications/GitHub Copilot.app/Contents/MacOS/github` - Binary type: native macOS Mach-O arm64 - Frameworks observed include AppKit and WebKit. - The desktop app is not obviously Electron/Chromium; do not assume Playwright/CDP control is available. ## Known Accessibility Map After macOS Accessibility permission was enabled, `System Events` could address the app as process `GitHub Copilot`. Useful exposed controls (app 1.1.26; see [app-ui-map.md](references/app-ui-map.md) for the full map): | Control | Role | Use | |---|---|---| | `GitHub Copilot` | `AXWindow` | Main app window | | `GitHub Copilot` | `AXWebArea` | Main webview-backed app content | | `Sidebar` | `AXGroup` | Sidebar region | | `Toggle sidebar, Command + B` | `AXCheckBox` | Collapse/expand sidebar | | `New`, `Pull requests`, `Issues`, `Automations`, `Customize` | `AXButton` | Sidebar views (Quick links) | | `More` | `AXPopUpButton` | Sidebar menu with **Edit sidebar...** | | `New project or session` | `AXPopUpButton` | Projects **+** menu | | `New session in <project>` | `AXButton` | Project-specific new session entry (visible on hover or focus) | | `Message` | `AXTextArea` | Prompt composer | | `Project: <project>` | `AXComboBox` | Composer project picker | | `Workspace: <choice>, branch: <branch>` | `AXPopUpButton` | **Where to work** menu | | `Toggle panel, Command + Option + B` | `AXCheckBox` | Review panel | | `Add tab, Command + T` | `AXPopUpButton` | Review panel tab menu | See [macos-accessibility.md](references/macos-accessibility.md) for scripts and caveats. ## Preferred Workflows ### Map the App Safely 1. Confirm the app is running and discover bundle/process state. 2. Enumerate projects and sessions with app/session tools. 3. Run a read-only Accessibility window/menu probe. 4. Enumerate `AXWebArea` children. 5. Save findings to session artifacts, not the repo, unless the user asked for durable docs. Use: - [probe-readonly.applescript](sample_codes/macos-accessibility/probe-readonly.applescript) - [automation-map.md](references/automation-map.md) ### Test Composer Control Without Submitting 1. Create a separate idle session. 2. Navigate to that session. 3. Focus `AXTextArea name=Message`. 4. Set a draft value directly. 5. Read the value back. 6. Clear it with Command+A and Delete. 7. Verify the returned `cleared-value` is empty. Use [type-and-clear-draft.applescript](sample_codes/macos-accessibility/type-and-clear-draft.applescript). ### Capture Screenshots for Course Assets Only use this workflow when the user has agreed the App can be visible or when the target state is already visible in a dedicated App window. Capture from a **sanitized training account** on the training fork — never the user's real private projects. 1. Run `screenshot-context.py` for the referenced image and turn the nearby course text into a concrete capture checklist. If you retake an existing image, view the old image first and list its callouts, boxes, and crop. 2. Launch a new process from the persistent `demo` persona with `prepare-persona.sh`. The pre-step sets a standard 1920x1080-point window, resets zoom, zooms in four times, and enables and verifies **Streamer Mode**. It fails closed if Streamer Mode or the zoom cannot be verified. Pass the exact local training repository and keep the returned process ID. Turn off Siri before you capture: its waveform orb can appear next to a focused text field, and the capture script fails when it sees the orb. 3. Map that process with `COPILOT_PID="$COPILOT_PID" map-app.sh` so steps match the current app build. 4. Navigate that process to the exact state with Accessibility controls and let spinners settle. Use `prepare-copilot-state.swift` for states that a named action cannot create: `focus` shows a hover-only control (such as the **+** on the **Chats** row), `highlight-menu-item` selects a menu item with the keyboard (web menus ignore synthetic pointer moves), `set-value` fills a field without pressing Return, and `park` moves the pointer and focus away so no hover effects or focus rings remain. 5. Find the final coordinates of each control with `locate-copilot-element.swift`, then capture the window owned by that process ID and convert it to WebP. Pass callouts in instruction order, highlight boxes, and an optional crop, all in final 1920x1080 coordinates: ```bash bash sample_codes/macos-accessibility/capture-window.sh \ <chapter>/assets <base-name> 40 "$COPILOT_PID" bash sample_codes/macos-accessibility/capture-window.sh \ 00-setup/assets app-add-project 40 "$COPILOT_PID" \ --callout 1:372:338 --callout 2:585:437 bash sample_codes/macos-accessibility/capture-window.sh \ 06-canvases/assets app-open-repo-issues-canvas 40 "$COPILOT_PID" \ --box 1287:266:1551:309 --crop 352:0:1920:700 ``` 6. For callout images, confirm that every badge is next to its control, does not cover a label or icon, and matches the instruction order. The standard style is a 26px-radius `#ff594b` circle with a white number. Keep each badge at least its radius plus 2 px from the image edge. 7. Keep PNG as the source artifact; use WebP in web/course pages. 8. Confirm that full-window files are exactly 1920x1080. A crop keeps the size of its crop rectangle. 9. Confirm that the image has a 2px `#cccccc` border inside its edges. 10. Confirm that the output keeps the avatar, shows **Copilot Dev** instead of a person's profile name, and uses **copilotdev** for matching personal repository owners, paths, and branch names. 11. For Settings screenshots, confirm that no app version is visible. 12. Review the screenshot for other private data before committing or publishing. 13. Restore any persona state that you changed for the capture (hidden projects, installed plugins or MCP servers, test automations or sessions), then close the exact process with `cleanup-persona.sh` only after the capture is verified. Keep the signed-in persona for the next screenshot. To work through the course's pending shots, discover them dynamically and process one at a time (never hardcode the list): `screenshots.sh list` -> `screenshots.sh next` -> capture -> `screenshots.sh embed`. See [missing-screenshots.md](references/missing-screenshots.md). Use: - [screenshot-capture.md](references/screenshot-capture.md) — full method, permissions, and the macOS Spaces constraint - [missing-screenshots.md](references/missing-screenshots.md) — the generic, one-at-a-time capture workflow (shots are discovered dynamically, never hardcoded) - [screenshots.sh](sample_codes/macos-accessibility/screenshots.sh) — dynamically `list` / `next` / `embed` the chapters' `<!-- app-screenshot: ... -->` placeholders - [screenshot-context.py](sample_codes/macos-accessibility/screenshot-context.py) — extracts the course section that defines a screenshot's required state - [prepare-persona.sh](sample_codes/macos-accessibility/prepare-persona.sh) — launches a new process from a signed-in persona, verifies the exact local repository, and returns the process ID - [ensure-streamer-mode.swift](sample_codes/macos-accessibility/ensure-streamer-mode.swift) — enables and verifies Streamer Mode before screenshot preparation - [cleanup-persona.sh](sample_codes/macos-accessibility/cleanup-persona.sh) — stops one exact screenshot process and preserves the signed-in persona - [launch-persona.sh](sample_codes/macos-accessibility/launch-persona.sh) — low-level launcher used by `prepare-persona.sh` - [control-copilot-window.swift](sample_codes/macos-accessibility/control-copilot-window.swift) — sets the 1920x1080-point capture frame and the measured 175% capture zoom (150% on the built-in display) for one exact app process - [control-copilot-ui.swift](sample_codes/macos-accessibility/control-copilot-ui.swift) — presses one named Accessibility control in an exact app process - [prepare-copilot-state.swift](sample_codes/macos-accessibility/prepare-copilot-state.swift) — prepares transient states: focus a hover-only control, highlight a menu item, set a field value, dismiss banners, and park the pointer - [locate-copilot-element.swift](sample_codes/macos-accessibility/locate-copilot-element.swift) — prints the final 1920x1080 coordinates of named controls for callouts, boxes, and crops - [find-private-identities.swift](sample_codes/macos-accessibility/find-private-identities.swift) — derives profile names and matching personal repository owners without hard-coded identities - [sanitize-screenshot.sh](sample_codes/macos-accessibility/sanitize-screenshot.sh) — replaces identity text while preserving avatars; caches identities per process because dialogs can hide the sidebar - [app-ui-map.md](references/app-ui-map.md) — sanitized factual UI map (menus, sidebar, views, composer controls), regenerable via map-app.sh - [capture-window.sh](sample_codes/macos-accessibility/capture-window.sh) — recommended: process-filtered CoreGraphics window capture + WebP, with `--callout`, `--box`, `--arrow`, and `--crop` - [finalize-screenshot.py](sample_codes/macos-accessibility/finalize-screenshot.py) — enforces 1920x1080 output, adds the required 2px `#cccccc` inside border, and crops - [add-step-callouts.py](sample_codes/macos-accessibility/add-step-callouts.py) — adds ordered red step badges, red highlight boxes, and red arrows to a finalized screenshot - [find-copilot-window.swift](sample_codes/macos-accessibility/find-copilot-window.swift) — lists on-screen Copilot windows with their window and process IDs - [map-app.sh](sample_codes/macos-accessibility/map-app.sh) — version-stamped UI map for grounding steps and diffing app updates - [capture-copilot-window.sh](sample_codes/macos-accessibility/capture-copilot-window.sh) — older Accessibility-rectangle fallback ### Drive App Work Without GUI Scripting Prefer session orchestration: - `list_projects` to find a project id. - `create_session` to create a new isolated session. - `send_session_message` to give that session work. - `get_session` to inspect path and metadata. - `navigate_to` only when the user explicitly wants to see the session. See [session-orchestration.md](references/session-orchestration.md). ### Use Canvases for Structured Human-Agent Work For test boards, dashboards, course trackers, or visible state, use canvas surfaces instead of GUI clicking. Canvases are bidirectional and can expose agent-callable actions. See [canvas-and-automation.md](references/canvas-and-automation.md). ## Decision Table | Goal | Best mechanism | |---|---| | Learn how sessions behave | Create separate sessions and compare outputs | | Type/clear a prompt in the desktop UI | macOS Accessibility `Message` text area | | Click a named app button | Accessibility by role/name/action | | Capture a course screenshot | Accessibility window bounds + `screencapture` PNG + WebP conversion |