os-use

# OS Use - Cross-Platform OS Automation A comprehensive cross-platform toolkit for OS automation, screenshot capture, visual recognition, mouse/keyboard control, and window management. Supports **macOS 12+** and **Windows 10+**. ## Platform Support Matrix | Feature | macOS Implementation | Windows Implementation | |---------|---------------------|----------------------| | **Screenshot** | `pyautogui` + `PIL` | `pyautogui` + `PIL` | | **Visual Recognition** | `opencv-python` + `pyautogui` | `opencv-python` + `pyautogui` | | **Mouse/Keyboard** | `pyautogui` | `pyautogui` | | **Window Management** | `AppleScript` (native) | `pywinauto` / `pygetwindow` | | **Application Control** | `AppleScript` / `subprocess` | `subprocess` / `pywinauto` | | **Browser Automation** | Chrome DevTools MCP | Chrome DevTools MCP | ## Capabilities ### 1. Screenshot Capture 📸 **Universal (macOS & Windows):** - Full screen capture - Region capture (specified coordinates) - Window capture (specific application window) - Clipboard screenshot access **Implementation:** `pyautogui.screenshot()` + `PIL.Image` ### 2. Visual Recognition 👁️ **Universal (macOS & Windows):** - Image matching/locating on screen - Template matching with confidence threshold - Multi-scale matching (handle different resolutions) - Color detection and region extraction **Optional OCR:** - Text recognition from screenshots (requires `pytesseract` + Tesseract OCR engine) **Implementation:** `opencv-python` + `pyautogui.locateOnScreen()` ### 3. Mouse & Keyboard Control 🖱️⌨️ **Universal (macOS & Windows):** - Mouse movement (absolute and relative coordinates) - Mouse clicking (left, right, middle, double-click) - Mouse dragging and dropping - Scroll wheel operations - Keyboard text input - Keyboard shortcuts and hotkeys - Special key combinations **Implementation:** `pyautogui` ### 4. Window Management 🪟 **macOS Implementation:** - List all application windows - Get window position, size, title - Activate/minimize/close windows - Move and resize windows - Launch/quit applications **Implementation:** `AppleScript` via `subprocess` **Windows Implementation:** - Same capabilities as macOS - Additional: Get window handle (HWND), process information - Better integration with Windows window manager **Implementation:** `pywinauto` or `pygetwindow` ### 5. Browser Automation 🌐 **Universal (macOS & Windows):** - Webpage screenshots - Element screenshots - Page navigation - Form filling and clicking - Network monitoring - Performance analysis **Implementation:** Chrome DevTools MCP (separate tool) ### 6. System Integration 🔧 **Clipboard Operations:** - Read/write clipboard content - Support images and text **Implementation:** `pyperclip` + `pyautogui` ## Technical Implementation Details ### Python Environment Setup ```bash # Create virtual environment python3 -m venv ~/.nanobot/workspace/macos-automation/.venv # Activate source ~/.nanobot/workspace/macos-automation/.venv/bin/activate # Install dependencies pip install pyautogui opencv-python-headless numpy Pillow pyperclip # macOS specific # (AppleScript is built-in, no installation needed) # Windows specific pip install pywinauto pygetwindow ``` ### Key Libraries Reference | Library | Version | Purpose | |---------|---------|---------| | `pyautogui` | 0.9.54+ | Screenshot, mouse/keyboard control | | `opencv-python-headless` | 4.11.0.84+ | Image recognition, computer vision | | `numpy` | 2.4.2+ | Numerical operations for OpenCV | | `Pillow` | 12.1.1+ | Image processing | | `pyperclip` | Latest | Clipboard operations | | `pywinauto` | Latest | Windows window management | | `pygetwindow` | Latest | Cross-platform window control | ### Platform-Specific Notes #### macOS Specifics **Permissions Required:** - **Accessibility**: System Settings > Privacy & Security > Accessibility - **Screen Recording**: System Settings > Privacy & Security > Screen Recording **AppleScript Quirks:** - Some modern apps (e.g., Chrome) may have limited AppleScript support - Window titles may be truncated or localized - Some operations require app to be frontmost **Coordinate System:** - Origin (0, 0) at top-left - Retina displays: pyautogui automatically handles scaling #### Windows Specifics **Administrator Privileges:** - Some operations (e.g., interacting with elevated windows) may require admin rights **High DPI Displays:** - Windows scaling may affect coordinate accuracy - Use `pyautogui.size()` to get actual screen dimensions **Window Handle (HWND):** - Windows provides low-level window handles for precise control - `pywinauto` provides both high-level and low-level access ### Error Handling Patterns ```python import pyautogui import time # Pattern 1: Retry with backoff def retry_with_backoff(func, max_retries=3, base_delay=1): for i in range(max_retries): try: return func() except Exception as e: if i == max_retries - 1: raise delay = base_delay * (2 ** i) print(f"Retry {i+1}/{max_retries} after {delay}s: {e}") time.sleep(delay) # Pattern 2: Safe operations with fallback def safe_screenshot(output_path): try: screenshot = pyautogui.screenshot() screenshot.save(output_path) return output_path except Exception as e: print(f"Screenshot failed: {e}") return None # Pattern 3: Coordinate boundary checking def safe_click(x, y, max_x=None, max_y=None): """安全点击，确保坐标在屏幕范围内""" if max_x is None or max_y is None: max_x, max_y = pyautogui.size() x = max(0, min(x, max_x - 1)) y = max(0, min(y, max_y - 1)) pyautogui.click(x, y) ``` ## Usage Examples by Scenario ### Scenario 1: Automated Testing ```python """ 自动化 UI 测试示例 测试一个假设的登录页面 """ import pyautogui import time def test_login_flow(): # 1. 截取初始状态 initial_screenshot = pyautogui.screenshot() initial_screenshot.save("test_01_initial.png") # 2. 查找并点击登录按钮 button_location = pyautogui.locateOnScreen( "login_button.png", confidence=0.9 ) if button_location: center = pyautogui.center(button_location) pyautogui.click(center.x, center.y) time.sleep(1) # 3. 输入用户名 pyautogui.typewrite("testuser@example.com", interval=0.01) pyautogui.press('tab') # 4. 输入密码 pyautogui.typewrite("TestPassword123", interval=0.01) # 5. 点击提交 pyautogui.press('return') time.sleep(2) # 6. 验证结果 result_screenshot = pyautogui.screenshot() result_screenshot.save("test_02_result.png") # 检查是否出现成功提示 success_indicator = pyautogui.locateOnScreen( "success_message.png", confidence=0.8 ) if success_indicator: print("✅ 测试通过：登录成功") return True else: print("❌ 测试失败：未找到成功提示") return False # 运行测试 if __name__ == "__main__": test_login_flow() ``` ### Scenario 2: Data Entry Automation ```python """ 数据录入自动化示例 将 Excel 数据自动填入网页表单 """ import pyautogui import pandas as pd import time def automate_data_entry(excel_file, form_template): """ 从 Excel 读取数据并自动填入表单 Args: excel_file: Excel 文件路径 form_template: 表单字段与 Excel 列的映射 """ # 1. 读取 Excel 数据 df = pd.read_excel(excel_file) print(f"读取到 {len(df)} 条记录") # 2. 遍历每条记录 for index, row in df.iterrows(): print(f"\n正在处理第 {index + 1} 条记录...") # 3. 填写每个字段 for field_name, column_name in form_template.items(): value = row.get(column_name, '') # 查找表单字段（需要提前准备字段截图） field_location = pyautogui.locateOnScreen( f"form_field_{field_name}.png", confidence=0.8 ) if field_location: # 点击字段 center = pyautogui.center(field_location) pyautogui.click(center.x, center.y) time.sleep(0.2) # 输入值 pyautogui.hotkey('ctrl', 'a') # 全选 pyautogui.typewrite(str(value), interval=0.01) time.sleep(0.2) else: print(f" ⚠️ 未找到字段: {field_name}") # 4. 提交表单 submit_btn = pyautogui.locateOnScreen( "submit_button.png", confidence=0.8 ) if submit_btn: center = pyautogui.center(submit_btn) pyautogui.click(center.x, center.y) print(" ✅ 已提交") time.sleep(2) # 等待提交完成 else: print(" ⚠️ 未找到提交按钮") # 5. 准备下一条记录 # 可能需要点击"添加新记录"或返回列表 time.sleep(1) print("\n🎉 所有记录处理完成！") # 使用示例 if __name__ == "__main__": # 表单模板：字段名 -> Excel 列名 form_template = { "name": "姓名", "email": "邮箱", "phone": "电话", "address": "地址" } automate_data_entry("data.xlsx", form_template) ``` ### Scenario 3: Screen Monitoring & Alerting ```python """ 屏幕监控与告警示例 监控特定区域变化，发现变化时发送通知 """ import pyautogui import cv2 import numpy as np import time from datetime import datetime def monitor_screen_region(region, template_image=None, check_interval=5, callback=None): """ 监控屏幕特定区域的变化 Args: region: (left, top, width, height) 监控区域 template_image: 要查找的模板图像路径（可选） check_interval: 检查间隔（秒） callback: 发现变化时的回调函数 Returns: 监控会话对象（可调用 stop() 停止） """ class MonitorSession: def __init__(self): self.running = True self.baseline = None def stop(self): self.running = False session = MonitorSession() print(f"🔍 开始监控区域: {region}") print(f"⏱️ 检查间隔: {check_interval}秒") print("按 Ctrl+C 停止监控\n") try: while session.running: # 捕获当前区域 current = pyautogui.screenshot(region=region) current_array = np.array(current) if template_image: # 模式1: 查找模板图像 template_location = pyautogui.locateOnScreen( template_image, confidence=0.8 ) if template_location: print(f"✅ [{datetime.now()}] 找到模板图像: {template_location}") if callback: callback('template_found', { 'location': template_location, 'screenshot': current }) else: # 模式2: 检测变化 if session.baseline is None: session.baseline = current_array print(f"📸 [{datetime.now()}] 已建立基准图像") else: # 计算差异 diff = cv2.absdiff(session.baseline, current_array) diff_gray = cv2.cvtColor(diff, cv2.COLOR_RGB2GRAY) diff_score = np.mean(diff_gray) if diff_score > 10: # 阈值可调 print(f"⚠️ [{datetime.now()}] 检测到变化! 差异分数: {diff_score:.2f}") if callback: callback('change_detected', { 'diff_score': diff_score, 'screenshot': current, 'baseline': session.baseline }) # 更新基准 session.baseline = current_array time.sleep(check_interval) except KeyboardInterrupt: print("\n🛑 监控已停止") return session # 使用示例 def alert_callback(event_type, data): """告警回调函数示例""" if event_type == 'template_found': print(f"🎯 模板出现在: {data['location']}") # 可以在这里发送通知、发送邮件、执行操作等 elif event_type == 'change_detected': print(f"📊 变化强度: {data['diff_score']}") # 保存差异图像 timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") data['screenshot'].save(f"change_{timestamp}.png") if __name__ == "__main__": # 示例1: 监控屏幕变化 print("=== 监控屏幕变化 ===") monitor = monitor_screen_region( region=(0, 0, 1920, 1080), # 全屏