OS Use - Cross-Platform OS Automation

A comprehensive cross-platform toolkit for OS automation, screenshot capture, visual recognition, mouse/keyboard control, and window management. Supports macOS 12+ and Windows 10+.

Platform Support Matrix

Feature	macOS Implementation	Windows Implementation
Screenshot	pyautogui + PIL	pyautogui + PIL
Visual Recognition	opencv-python + pyautogui	opencv-python + pyautogui
Mouse/Keyboard	pyautogui	pyautogui
Window Management	AppleScript (native)	pywinauto / pygetwindow
Application Control	AppleScript / subprocess	subprocess / pywinauto
Browser Automation	Chrome DevTools MCP	Chrome DevTools MCP

Capabilities

1. Screenshot Capture 📸

Universal (macOS & Windows):

• Full screen capture
• Region capture (specified coordinates)
• Window capture (specific application window)
• Clipboard screenshot access

Implementation: pyautogui.screenshot() + PIL.Image

2. Visual Recognition 👁️

Universal (macOS & Windows):

• Image matching/locating on screen
• Template matching with confidence threshold
• Multi-scale matching (handle different resolutions)
• Color detection and region extraction

Optional OCR:

• Text recognition from screenshots (requires pytesseract + Tesseract OCR engine)

Implementation: opencv-python + pyautogui.locateOnScreen()

3. Mouse & Keyboard Control 🖱️⌨️

Universal (macOS & Windows):

• Mouse movement (absolute and relative coordinates)
• Mouse clicking (left, right, middle, double-click)
• Mouse dragging and dropping
• Scroll wheel operations
• Keyboard text input
• Keyboard shortcuts and hotkeys
• Special key combinations

Implementation: pyautogui

4. Window Management 🪟

macOS Implementation:

• List all application windows
• Get window position, size, title
• Activate/minimize/close windows
• Move and resize windows
• Launch/quit applications

Implementation: AppleScript via subprocess

Windows Implementation:

• Same capabilities as macOS
• Additional: Get window handle (HWND), process information
• Better integration with Windows window manager

Implementation: pywinauto or pygetwindow

5. Browser Automation 🌐

Universal (macOS & Windows):

• Webpage screenshots
• Element screenshots
• Page navigation
• Form filling and clicking
• Network monitoring
• Performance analysis

Implementation: Chrome DevTools MCP (separate tool)

6. System Integration 🔧

Clipboard Operations:

• Read/write clipboard content
• Support images and text

Implementation: pyperclip + pyautogui

Technical Implementation Details

Python Environment Setup

# Create virtual environment
python3 -m venv ~/.nanobot/workspace/macos-automation/.venv

# Activate
source ~/.nanobot/workspace/macos-automation/.venv/bin/activate

# Install dependencies
pip install pyautogui opencv-python-headless numpy Pillow pyperclip

# macOS specific
# (AppleScript is built-in, no installation needed)

# Windows specific
pip install pywinauto pygetwindow

Key Libraries Reference

Library	Version	Purpose
pyautogui	0.9.54+	Screenshot, mouse/keyboard control
opencv-python-headless	4.11.0.84+	Image recognition, computer vision
numpy	2.4.2+	Numerical operations for OpenCV
Pillow	12.1.1+	Image processing
pyperclip	Latest	Clipboard operations
pywinauto	Latest	Windows window management
pygetwindow	Latest	Cross-platform window control

Platform-Specific Notes

macOS Specifics

Permissions Required:

• Accessibility: System Settings > Privacy & Security > Accessibility
• Screen Recording: System Settings > Privacy & Security > Screen Recording

AppleScript Quirks:

• Some modern apps (e.g., Chrome) may have limited AppleScript support
• Window titles may be truncated or localized
• Some operations require app to be frontmost

Coordinate System:

• Origin (0, 0) at top-left
• Retina displays: pyautogui automatically handles scaling

Windows Specifics

Administrator Privileges:

• Some operations (e.g., interacting with elevated windows) may require admin rights

High DPI Displays:

• Windows scaling may affect coordinate accuracy
• Use pyautogui.size() to get actual screen dimensions

Window Handle (HWND):

• Windows provides low-level window handles for precise control
• pywinauto provides both high-level and low-level access

Error Handling Patterns

import pyautogui
import time

# Pattern 1: Retry with backoff
def retry_with_backoff(func, max_retries=3, base_delay=1):
    for i in range(max_retries):
        try:
            return func()
        except Exception as e:
            if i == max_retries - 1:
                raise
            delay = base_delay * (2 ** i)
            print(f"Retry {i+1}/{max_retries} after {delay}s: {e}")
            time.sleep(delay)

# Pattern 2: Safe operations with fallback
def safe_screenshot(output_path):
    try:
        screenshot = pyautogui.screenshot()
        screenshot.save(output_path)
        return output_path
    except Exception as e:
        print(f"Screenshot failed: {e}")
        return None

# Pattern 3: Coordinate boundary checking
def safe_click(x, y, max_x=None, max_y=None):
    """安全点击，确保坐标在屏幕范围内"""
    if max_x is None or max_y is None:
        max_x, max_y = pyautogui.size()
    
    x = max(0, min(x, max_x - 1))
    y = max(0, min(y, max_y - 1))
    
    pyautogui.click(x, y)

Usage Examples by Scenario

Scenario 1: Automated Testing

"""
自动化 UI 测试示例
测试一个假设的登录页面
"""
import pyautogui
import time

def test_login_flow():
    # 1. 截取初始状态
    initial_screenshot = pyautogui.screenshot()
    initial_screenshot.save("test_01_initial.png")
    
    # 2. 查找并点击登录按钮
    button_location = pyautogui.locateOnScreen(
        "login_button.png",
        confidence=0.9
    )
    if button_location:
        center = pyautogui.center(button_location)
        pyautogui.click(center.x, center.y)
        time.sleep(1)
    
    # 3. 输入用户名
    pyautogui.typewrite("[email protected]", interval=0.01)
    pyautogui.press('tab')
    
    # 4. 输入密码
    pyautogui.typewrite("TestPassword123", interval=0.01)
    
    # 5. 点击提交
    pyautogui.press('return')
    time.sleep(2)
    
    # 6. 验证结果
    result_screenshot = pyautogui.screenshot()
    result_screenshot.save("test_02_result.png")
    
    # 检查是否出现成功提示
    success_indicator = pyautogui.locateOnScreen(
        "success_message.png",
        confidence=0.8
    )
    
    if success_indicator:
        print("✅ 测试通过：登录成功")
        return True
    else:
        print("❌ 测试失败：未找到成功提示")
        return False

# 运行测试
if __name__ == "__main__":
    test_login_flow()

Scenario 2: Data Entry Automation

"""
数据录入自动化示例
将 Excel 数据自动填入网页表单
"""
import pyautogui
import pandas as pd
import time

def automate_data_entry(excel_file, form_template):
    """
    从 Excel 读取数据并自动填入表单
how to use os-use
CursorClaude CodeClineCodexWindsurfGooseGitHub CopilotVS CodeGemini CLIKiloOpencodeKimi CLIRoo ClineAiderContinueZedBolt.newLovablev0Replit AgentSweepSmol DeveloperDevinCavemanAmpAntigravity
How to use os-use on Cursor
AI-first code editor with Composer
1
Prerequisites
Before installing skills in Cursor, ensure your development environment meets these requirements:
›Cursor installed and configured on your development machine
›Node.js version 16.0+ with npm package manager (verify with node --version)
›Active project directory or workspace where you want to add os-use
2
Execute installation command
Execute the skills CLI command in your project's root directory to begin installation:
$npx skills add https://github.com/zrong/skills --skill os-usecopy
The skills CLI fetches os-use from GitHub repository zrong/skills and configures it for Cursor.
3
Select Cursor when prompted
The CLI will show a list of available agents. Use arrow keys to navigate and space to select Cursor:
◆ Which agents do you want to install to?
│
│ ── Universal (.agents/skills) ── always included ────
│   • Amp
│   • Antigravity
│   • Cline
│   • Codex
│ ●Cursor(selected)
│   • Cursor
│   • Windsurf
4
Verify installation
Confirm successful installation by checking the skill directory location:
.cursor/skills/os-use
Reload or restart Cursor to activate os-use. Access the skill through slash commands (e.g., /os-use) or your agent's skill management interface.
⚠
Security & Verification Notice
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your development environment. Always verify the publisher's identity, review recent commits, and test in isolated environments before production deployment.
Additional Resources
›View source code on GitHub›Skills CLI documentation›Learn more about Cursor›What are agent skills?