ESP32 iPhone Agent: A $7 Board Lets an AI Model Use a Real iPhone

A trending GitHub repo shows how little hardware an AI model needs to operate a phone. esp32-iphone-agent, created on October 11, 2026 and at about 2,575 stars gained within days, describes itself as a way to "let an AI model use a real iPhone with a $7 ESP32." Its README says the system needs no jailbreak and no developer account for daily use, and works with any OpenAI-compatible vision model. The Hacker News thread is small, but the repo's star count tells its own story.

This post explains how it works, how to set it up, where it falls short, and what to think about before pointing an agent at your own phone.

TL;DR: ESP32 iPhone Agent at a glance

How does it work?

Three parts, each with an unusual trick.

Eyes: AirPlay. The board creates its own Wi-Fi hotspot and acts as an AirPlay Screen Mirroring receiver. The H.264 video stream is passed to the phone app over Bluetooth Low Energy without being decoded on the board. In effect the iPhone mirrors its own screen to a fake Apple TV that hands the video straight back to the app.

Hands: USB HID. The board presents itself to the phone as a USB mouse and keyboard. With AssistiveTouch enabled on iOS, a mouse click becomes a tap and a drag becomes a swipe. That is how it drives a phone that offers no remote-control API.

Brain: the app. An iPhone app decodes the stream, sends screenshots to the vision model, and uses a planner and an actor to choose actions. It runs in the background and shows status in the Dynamic Island. After each action it checks whether the screen changed, re-homes the relative pointer, and corrects missed taps. Each run is saved on the phone for step-by-step replay.

How do you set it up?

According to the README:

1. Flash the ESP32-S3 with arduino-cli and the provided script.
2. Build and install the iOS app with Xcode. A free Apple ID works, though signing expires every 7 days.
3. In the app, pair the board over Bluetooth, join its Wi-Fi and enable Screen Mirroring, enable AssistiveTouch for calibration, and enter your model endpoint and key.
4. Type a task and press Start.

USB-C iPhones connect directly; Lightning models need an adapter. Prompts are plain text files in a prompts folder, so you can change the agent's behavior without recompiling. Because the phone's Wi-Fi is occupied by the board's hotspot while the agent runs, the model is reached over cellular data.

What are the limits?

The README is candid about them:

• BLE bandwidth caps the mirrored screen at 480 by 1040 and up to 5 fps, so each step takes a few seconds.
• iOS hides some screens, including passcode entry, Apple Pay and DRM video. Mirroring stops when the phone locks.
• The app uses a private iOS API, which must be replaced before any App Store release.
• Keyboard input is ASCII-only and landscape apps are unsupported.
• A legacy v0.1.0 needed a Mac or Linux computer running UxPlay and a Python server; the current version removes that.

The roadmap lists landscape support, non-ASCII input and faster multi-action steps.

How does it compare with other ways to automate a phone?

Most phone automation relies on accessibility services, debugging bridges or developer tools. On iOS those paths are tightly restricted, which is why this project is interesting: it avoids software permissions entirely by acting as hardware that the phone trusts, a mouse and a screen mirroring target. The cost is speed and resolution.

For desktop equivalents, see how tools let agents drive a logged-in browser, such as Tencent's BrowserSkill, or point at the screen with BigArrow. Small hardware doing AI work is a recurring theme; compare the 28.9M-parameter model on an $8 ESP32 and WiFi sensing on ESP32. Those use the chip as the compute; here it is just the input and output device.

Is it safe to point an agent at your phone?

Think about what the agent can reach. A phone holds messaging, banking, email, authentication codes and photos. The agent acts with your hands, and a vision model can misread a screen or follow instructions embedded in what it sees, such as a message that tells it to do something. The README asks users to automate only devices and accounts they own or are permitted to use, to respect the terms of the apps involved, to watch the agent, and to keep the stop button within reach.

Practical precautions:

1. Use a spare phone with no banking apps, not your main device.
2. Log in only to accounts you can afford to have actions taken in.
3. Keep tasks short and supervised.
4. Remember that screenshots go to whichever model endpoint you configure; self-host the model if the screen content is sensitive.
5. Check app terms: many forbid automated interaction.

For a broader look at how agents get misled by on-screen content and what happens when agents act with real accounts, see our analysis of a Grok bot and disputed agent permissions. AgentBeam, the agent security platform from the explainx.ai team, is aimed at stopping agents before they take dangerous actions.

What people are asking

What does it cost to build? The board is about $7 per the README's own framing. You also need an iPhone you already own, a cellular data plan (the phone's Wi-Fi is used by the board's hotspot), a one-time Mac with Xcode, and a vision model endpoint. A self-hosted open model costs only electricity; a cloud model bills per screenshot, and every step sends at least one.

Why does each step take seconds? Three delays stack: the mirrored stream is capped at 5 fps over BLE, the screenshot must reach the model, and the model must plan the next action. The app mitigates this by checking whether the screen changed after each action instead of blindly queuing taps, which also catches misses, but it cannot make a slow link fast. The roadmap's "faster multi-action steps" is the obvious next lever.

Why not just use accessibility APIs or Xcode tooling? On iOS, third-party control of other apps is deliberately limited, and developer tooling needs a computer and a signed build. The hardware route sidesteps both, at the price of low resolution and no access to hidden screens. It is closer to a robot finger than an API.

Will this keep working? It depends on iOS behavior that Apple controls: AirPlay receivers, USB mouse support through AssistiveTouch, and background behavior of the app. The README itself notes the private API used by the app. Treat the project as a research prototype that may break with an iOS update.

Is it a good benchmark target? Possibly. Because it uses real apps rather than emulators, results reflect real-world UI variation. But the low frame rate and capped resolution make it unfair to compare with desktop computer-use numbers.

What should you test first?

1. A read-only task. Ask the agent to open Settings and report the iOS version. This checks the whole loop without side effects.
2. A reversible task. Create and delete a note. Confirm it taps accurately and recovers from a missed tap.
3. A multi-app task on a spare account. For example, copy text from one app to another. Watch for drift when screens change unexpectedly.
4. A failure case. Lock the phone mid-task and see whether the agent pauses and resumes as the README says it should.
5. Review the replay. The app saves each run for step-by-step replay; reading it is the fastest way to learn where your prompts need tightening.

Keep a log of success rates by task type before trusting it with anything that matters. Small hobby projects often shine on a demo reel and struggle on the third unfamiliar app.

What about the license?

The main project is MIT. The exception is hardware/hid_bridge/src/airplay/, which is ported from UxPlay, an open-source AirPlay mirroring server, and licensed GPL-3.0; the README says firmware binaries linking it are distributed under GPL-3.0. If you plan to build a product on the firmware, that matters.

Who is this for?

Tinkerers, mobile QA teams who need to test on real devices without enterprise tooling, researchers studying computer-use agents on phones, and people building accessibility experiments. It is not a polished consumer product: setup needs Xcode, signing expires weekly on a free account, and a few seconds per step makes long tasks slow.

Related reading

• Tencent BrowserSkill: agents borrow your real browser
• BigArrow: AI agents point at the screen
• 28.9M parameters on an $8 ESP32
• RuView: WiFi sensing on ESP32
• Grok bot, bank balance and agent permissions

Sources: esp32-iphone-agent on GitHub and its README. Hacker News discussion linked above.

Figures are as shown in the README on October 16, 2026 and may change.