Barehands: Give AI Agents Hands With a Webcam

Barehands is a webcam-powered interface that lets an AI agent put things on a stage in front of you. Notes, images, and 3D models appear as glass cards. You pinch to move them, use two hands to scale them, drag to scrub an exploded model, and clap to clear the board.

The interesting part is not the hologram language. It is the separation underneath: Barehands supplies a body—camera, gestures, ring, and board—while the agent supplies the brain. The two meet through tiny state files and localhost commands rather than a vendor-specific model API.

The project's official repository describes the setup as a webcam plus Chrome, with a standard-library Python server. Its demo video shows the intended interaction: a person moves through a spatial interface while an assistant can use the same surface to present information. That makes Barehands a useful case study in what an agent interface looks like when the output is spatial and visible instead of another block of chat.

The companion Jarvis assistant video is useful context for the same design direction: the visual surface is not the model, but a way to give an assistant a presence, a state, and a place to put the result.

TL;DR — what you get and what you do not

The core idea: an agent body with two surfaces

Barehands has two visible surfaces and two correspondingly small control paths.

The ring is the face

The ring is a compact status display. The agent writes a word to state/state, and the interface reacts:

thinking
listening
speaking
idle

That is deliberately simpler than streaming an assistant's full internal trace into the UI. The ring communicates presence and mode, not a transcript. A user can see that the agent is listening or speaking without surrendering the whole board to a chat panel.

For Claude Code users, the repository describes hooks that mirror real sessions into the ring. This is the same general pattern covered in our Claude Code hooks guide: a lifecycle event updates a local side effect. The difference is the output—rather than running a formatter or blocking a command, a hook makes the agent's state legible in the room.

The board is the stage

The board is where the agent shows its work. A present action can move a card to center stage, enlarge it, and dim the rest. Other actions add or manipulate pieces of the scene:

The reverse path is just as important. bin/board-state.sh reports what is already on the board so the agent can inspect before it speaks or changes the scene. That read-before-write behavior is a small but meaningful agent design choice: the assistant gets an external state it can query instead of assuming the UI still matches its last action.

This is the spatial equivalent of an agent harness. The harness defines what the model can do, how it observes state, and what actions are valid. Barehands does not attempt to be a complete harness, but its ring and board protocols provide a narrow, inspectable interface that a harness can call.

Setup: the shortest path from clone to camera

The README's minimal path is intentionally small:

git clone https://github.com/jaredrhod/barehands
cd barehands
python3 server.py

Then open http://127.0.0.1:8794/stage.html in Chrome, approve camera access, and use the board. On Windows, the README uses python server.py rather than python3.

The Python server is described as standard-library only. The browser loads Google MediaPipe for hand tracking and three.js for 3D rendering from public CDNs on first run. That keeps the repository small and the initial install approachable, but it also creates two operational questions:

1. What exactly is loaded at runtime? Pinning or reviewing CDN versions matters if the interface is used in a controlled environment.
2. What does camera permission expose? The browser is receiving the camera stream, so the machine and browser profile should be treated as part of the trust boundary.

Barehands does not need an account, API key, hosted database, or model subscription to draw the scene. That makes it a useful local-first surface for a model you already operate, but it does not make the agent itself local or private automatically. The brain can be any program, including one that sends the content it reads to a remote service.

Give the board a vault and a media boundary

The notes surface is a configured folder of Markdown. An Obsidian vault works because the project treats the vault as files rather than as a proprietary database. The example configuration looks like this:

{
  "name": "Assistant",
  "orbs": [
    { "title": "Notes", "path": "~/MyVault", "kind": "notes" },
    { "title": "Props", "path": "media", "kind": "media" }
  ]
}

The Notes orb points at the knowledge a person wants to browse. The Props orb is more constrained: the README calls media/ a “props airlock” and lists separate locations for ordinary images, transparent effects, 3D models, and hologram rendering.

That split is a good default for agent interfaces. Notes are information to retrieve; props are things the agent is allowed to put on stage. Keeping the media surface in a named directory gives the server a place to enforce a boundary instead of allowing every arbitrary path to become a visual asset.

It is not a complete data-loss prevention system. A configured notes path can still contain sensitive information, and an agent that can call a local script may have other permissions outside the board. Treat the jail as one layer in a defense-in-depth design, not as proof that every connected model is safe.

For a deeper look at durable agent context, compare MemPalace's local memory architecture and benchmark caveats. Barehands handles the visible, spatial side of memory; a vault or memory system handles what the assistant can recall.

The gestures are a language, not a demo trick

The project documents a compact gesture vocabulary:

• Tap / quick pinch: open and close.
• Pinch-drag: move an object.
• Hold while carrying: rotate a 3D object.
• Two hands: scale.
• Flick: throw.
• Clap: sweep the board clean.
• Claw: pull a distant object toward your hand.
• Empty pinch dragged sideways: scrub an exploded model apart and back together.

This matters because a spatial agent interface needs verbs, not just a canvas. “Show me the plan” can map to present; “bring that model here” can map to yank; “take it apart” can map to explode. The human gesture vocabulary and the agent command vocabulary describe the same scene from opposite sides.

The README says the thresholds were tuned on real hands and use ratios rather than absolute hand size, so distance from the camera should not be the only calibration variable. That is a design claim, not a guarantee for every camera, lighting condition, skin tone, background, or hand pose. The included troubleshooting material is important precisely because gesture recognition fails at the edges: debug overlays and pose sampling turn a mysterious misfire into a measurable tuning problem.

How to connect an agent without making the board the agent

The safest conceptual model is to keep Barehands as a capability surface, not the decision-maker. A separate agent or harness should decide when to act, then use a small allowlisted vocabulary to express the result.

A simple workflow might look like this:

User request
    ↓
Agent reads notes and board state
    ↓
Agent chooses a bounded board action
    ↓
bin/board.sh receives JSON
    ↓
Server validates action and media path
    ↓
Board presents the result

The state file provides a lightweight heartbeat. The board-state script provides observation. The action allowlist provides a constrained output channel. Together, those are the beginnings of an agent loop: observe, decide, act, observe again.

The README also says a local LLM, cron job, or Stream Deck button can act as the brain. That is a useful portability test. If the interface depends on one assistant's proprietary computer-use API, it is a product integration. If it accepts files and localhost commands, it is closer to a protocol that many agents can implement.

That protocol is not automatically interoperable. Every agent still needs instructions for the action schema, JSON escaping, failure handling, and when to read board state. A good setup guide should tell the agent to validate paths, avoid shell interpolation, confirm destructive actions, and recover when the browser or server is unavailable.

Security: what the README's boundaries do—and do not—promise

Barehands includes several sensible constraints:

1. Local default: the documented URL binds the interaction to 127.0.0.1.
2. Action allowlist: the server recognizes a defined set of board verbs.
3. Media jail: displayed media is restricted to the configured media/ surface.
4. Explicit camera permission: Chrome asks before the page receives camera access.

Those reduce accidental exposure, but an AI-connected local server remains security-sensitive. Before handing it to an autonomous agent, review:

This is where Barehands differs from a polished computer-use demo. A demo can show a model clicking through an app. A reusable agent interface must define permissions, state, failure recovery, and what happens when the model is wrong. Our computer-use coverage explains the same distinction from the managed desktop-agent side.

What Barehands is good at today

The project is a strong fit for workflows where visual arrangement is part of the task:

• Planning: present a plan card while the agent speaks through the next steps.
• Research: place source notes and media props on a board for comparison.
• 3D explanation: show a model, rotate it, and expose its exploded structure.
• Teaching: let a learner manipulate a concept instead of reading a static answer.
• Streaming: mirror the transparent stage into OBS for a tutorial or short video.
• Ambient status: turn the ring into a visible “thinking/listening/speaking” signal during a voice or coding session.

It is less suited to unattended automation, high-stakes data, or a replacement for a full desktop sandbox. The interface is local, but the connected agent may not be. The gesture system is expressive, but camera quality and lighting still matter. The protocols are simple, but their safety depends on how carefully the scripts and calling agent are constrained.

AGPL-3.0-or-later: commercial use with a sharing condition

Barehands' README says it can be used, changed, and built on commercially. The repository is licensed under AGPL-3.0-or-later, which is not the same as a permissive MIT or Apache license.

The practical question is not “can a business use it?” The answer in the README is yes. The practical questions are:

• Are you passing a modified copy to another person or organization?
• Are you running a modified version as a service for other users?
• Are you preserving the license and providing the corresponding source required by the AGPL?
• Do you need a separate commercial license for a closed-source product?

The repository's LICENSE is the authority. This article is not legal advice, and a closed product embedding or offering a modified Barehands service should get a license review before shipping.

What to ask before building on it

Barehands' most promising contribution is its protocol shape, but a serious evaluation should go beyond the demo video:

1. Can you reproduce the setup on your camera and lighting? Test gestures at the distance and angle you will actually use.
2. What does the server expose? Read server.py, bin/, and stage.html; do not infer the security model from the README alone.
3. What is the failure mode? Disconnect the browser, corrupt the JSON, remove a media file, and send an unknown action.
4. Can the agent observe? Make it call board-state.sh before changing the scene and verify that its mental state matches the board.
5. What data leaves the machine? Inspect CDN requests, model-provider calls, logs, and the configured notes path.
6. What survives a restart? Decide which state is ephemeral and which notes/configuration should be backed up.
7. Who can trigger a destructive action? Add confirmation or undo around board-clearing gestures if the stage contains valuable work.

This is the right evaluation posture for any viral open-source agent project. MemPalace's post-launch README corrections are a useful reminder that a compelling README and a reproducible implementation are related, not interchangeable.

The larger idea: agents need bodies that are inspectable

AI agents are often presented as invisible workers: they read context, call tools, and return text. Barehands suggests another interface model. The agent can have a face, a stage, a set of gestures, and a visible world state without becoming a humanoid robot or a full virtual-reality system.

The best part is the inspectability. A user can see the object the agent is presenting, move it by hand, ask the agent to read the board, and interrupt the scene. The model remains powerful, but the interface gives the person a physical checkpoint between intention and action.

That is why the “body waiting for a brain” description works. The brain does not need to be one vendor's assistant. It can be a local model, a coding agent, a voice loop, or a small script—as long as it can observe and act through the documented boundary.

Barehands is not yet proof that holographic interfaces are the future of agent work. It is something more useful: a concrete experiment in making agent state and output spatial, while keeping the control surface small enough to inspect. The next question is not whether a model can make a card fly across the room. It is whether the protocol remains predictable when the agent is wrong, the camera is noisy, the board is full, and the data is private.

Related reading

• What is an agent harness? The scaffolding layer that makes AI reliable
• Claude Code hooks: automate actions before and after tool calls
• MemPalace and the reality of local AI memory benchmarks
• OpenAI Codex computer use on Windows and mobile steering
• What are AI agents? A complete beginner's guide
• Loop engineering for AI agents
• Barehands on GitHub
• Barehands demo video
• Barehands Jarvis assistant video

Repository behavior, setup commands, license language, and feature descriptions reflect the Barehands README and repository available on August 22, 2026. Dependencies, scripts, CDN versions, supported browsers, and security properties can change; inspect the current source before using it with private data or an autonomous agent.