OS Use - Cross-Platform OS Automation

A comprehensive cross-platform toolkit for OS automation, screenshot capture, visual recognition, mouse/keyboard control, and window management. Supports macOS 12+ and Windows 10+.

Platform Support Matrix

Feature	macOS Implementation	Windows Implementation
Screenshot	pyautogui + PIL	pyautogui + PIL
Visual Recognition	opencv-python + pyautogui	opencv-python + pyautogui
Mouse/Keyboard	pyautogui	pyautogui
Window Management	AppleScript (native)	pywinauto / pygetwindow
Application Control	AppleScript / subprocess	subprocess / pywinauto
Browser Automation	Chrome DevTools MCP	Chrome DevTools MCP

Capabilities

1. Screenshot Capture 📸

Universal (macOS & Windows):

• Full screen capture
• Region capture (specified coordinates)
• Window capture (specific application window)
• Clipboard screenshot access

Implementation: pyautogui.screenshot() + PIL.Image

2. Visual Recognition 👁️

Universal (macOS & Windows):

• Image matching/locating on screen
• Template matching with confidence threshold
• Multi-scale matching (handle different resolutions)
• Color detection and region extraction

Optional OCR:

• Text recognition from screenshots (requires pytesseract + Tesseract OCR engine)

Implementation: opencv-python + pyautogui.locateOnScreen()

3. Mouse & Keyboard Control 🖱️⌨️

Universal (macOS & Windows):

• Mouse movement (absolute and relative coordinates)
• Mouse clicking (left, right, middle, double-click)
• Mouse dragging and dropping
• Scroll wheel operations
• Keyboard text input
• Keyboard shortcuts and hotkeys
• Special key combinations

Implementation: pyautogui

4. Window Management 🪟

macOS Implementation:

• List all application windows
• Get window position, size, title
• Activate/minimize/close windows
• Move and resize windows
• Launch/quit applications

Implementation: AppleScript via subprocess

Windows Implementation:

• Same capabilities as macOS
• Additional: Get window handle (HWND), process information
• Better integration with Windows window manager

Implementation: pywinauto or pygetwindow

5. Browser Automation 🌐

Universal (macOS & Windows):

• Webpage screenshots
• Element screenshots
• Page navigation
• Form filling and clicking
• Network monitoring
• Performance analysis

Implementation: Chrome DevTools MCP (separate tool)

6. System Integration 🔧

Clipboard Operations:

• Read/write clipboard content
• Support images and text

Implementation: pyperclip + pyautogui

Technical Implementation Details

Python Environment Setup

# Create virtual environment
python3 -m venv ~/.nanobot/workspace/macos-automation/.venv

# Activate
source ~/.nanobot/workspace/macos-automation/.venv/bin/activate

# Install dependencies
pip install pyautogui opencv-python-headless numpy Pillow pyperclip

# macOS specific
# (AppleScript is built-in, no installation needed)

# Windows specific
pip install pywinauto pygetwindow

Key Libraries Reference

Library	Version	Purpose
pyautogui	0.9.54+	Screenshot, mouse/keyboard control
opencv-python-headless	4.11.0.84+	Image recognition, computer vision
numpy	2.4.2+	Numerical operations for OpenCV
Pillow	12.1.1+	Image processing
pyperclip	Latest	Clipboard operations
pywinauto	Latest	Windows window management
pygetwindow	Latest	Cross-platform window control

Platform-Specific Notes

macOS Specifics

Permissions Required:

• Accessibility: System Settings > Privacy & Security > Accessibility
• Screen Recording: System Settings > Privacy & Security > Screen Recording

AppleScript Quirks:

• Some modern apps (e.g., Chrome) may have limited AppleScript support
• Window titles may be truncated or localized
• Some operations require app to be frontmost

Coordinate System:

• Origin (0, 0) at top-left
• Retina displays: pyautogui automatically handles scaling

Windows Specifics

Administrator Privileges:

• Some operations (e.g., interacting with elevated windows) may require admin rights

High DPI Displays:

• Windows scaling may affect coordinate accuracy
• Use pyautogui.size() to get actual screen dimensions

Window Handle (HWND):

• Windows provides low-level window handles for precise control
• pywinauto provides both high-level and low-level access

Error Handling Patterns

import pyautogui
import time

# Pattern 1: Retry with backoff
def retry_with_backoff(func, max_retries=3, base_delay=1):
    for i in range(max_retries):
        try:
            return func()
        except Exception as e:
            if i == max_retries - 1:
                raise
            delay = base_delay * (2 ** i)
            print(f"Retry {i+1}/{max_retries} after {delay}s: {e}")
            time.sleep(delay)

# Pattern 2: Safe operations with fallback
def safe_screenshot(output_path):
    try:
        screenshot = pyautogui.screenshot()
        screenshot.save(output_path)
        return output_path
    except Exception as e:
        print(f"Screenshot failed: {e}")
        return None

# Pattern 3: Coordinate boundary checking
def safe_click(x, y, max_x=None, max_y=None):
    """安全点击，确保坐标在屏幕范围内"""
    if max_x is None or max_y is None:
        max_x, max_y = pyautogui.size()
    
    x = max(0, min(x, max_x - 1))
    y = max(0, min(y, max_y - 1))
    
    pyautogui.click(x, y)

Usage Examples by Scenario

Scenario 1: Automated Testing

"""
自动化 UI 测试示例
测试一个假设的登录页面
"""
import pyautogui
import time

def test_login_flow():
    # 1. 截取初始状态
    initial_screenshot = pyautogui.screenshot()
    initial_screenshot.save("test_01_initial.png")
    
    # 2. 查找并点击登录按钮
    button_location = pyautogui.locateOnScreen(
        "login_button.png",
        confidence=0.9
    )
    if button_location:
        center = pyautogui.center(button_location)
        pyautogui.click(center.x, center.y)
        time.sleep(1)
    
    # 3. 输入用户名
    pyautogui.typewrite("[email protected]", interval=0.01)
    pyautogui.press('tab')
    
    # 4. 输入密码
    pyautogui.typewrite("TestPassword123", interval=0.01)
    
    # 5. 点击提交
    pyautogui.press('return')
    time.sleep(2)
    
    # 6. 验证结果
    result_screenshot = pyautogui.screenshot()
    result_screenshot.save("test_02_result.png")
    
    # 检查是否出现成功提示
    success_indicator = pyautogui.locateOnScreen(
        "success_message.png",
        confidence=0.8
    )
    
    if success_indicator:
        print("✅ 测试通过：登录成功")
        return True
    else:
        print("❌ 测试失败：未找到成功提示")
        return False

# 运行测试
if __name__ == "__main__":
    test_login_flow()

Scenario 2: Data Entry Automation

"""
数据录入自动化示例
将 Excel 数据自动填入网页表单
"""
import pyautogui
import pandas as pd
import time

def automate_data_entry(excel_file, form_template):
    """
    从 Excel 读取数据并自动填入表单