Work2Skill: Turning Human GUI Demonstrations Into Reusable Agent Skills

Work2Skill is a research framework that watches a person use desktop software and turns that recording into a reusable agent skill, a SKILL.md file a computer-use agent can follow later without the original video. The paper, trending on Hugging Face Papers with 30 upvotes at the time of writing, comes from authors at Shanghai Jiao Tong University, Fudan University, Nanyang Technological University, Zhejiang University and Shanghai Qiji Zhifeng Co., Ltd., and it ships code and a dataset. The paper page lists the repository as china-qijizhifeng/Work2Skill on GitHub.

The core claim: computer-use agents struggle with unfamiliar software and workflows that need human domain knowledge, and the cheapest source of that knowledge is a person doing the task once. This post explains the method, the reported numbers, and what builders can take from it.

TL;DR

Why do GUI agents need skills from demonstrations?

Computer-use agents have become capable at general tasks, but a new application or an in-house workflow is different. The paper's framing is that a demonstration "conveys procedural knowledge through the same GUI interaction space" the agent works in, yet reusing it is hard because you must interpret each action in context and separate the workflow's rules from details specific to one task. Clicking a cell at row 4 is not knowledge; knowing that totals must be recomputed after a filter is.

This is the same gap that agent skills try to fill in coding: a short, human-readable file that tells an agent how a job is done in this environment. Our explainer on agent skills covers the format; Work2Skill asks whether the file can be written automatically from watching a person rather than by hand. A related approach for teams is Tencent's TeamAI CLI, which shares skills through git, and Headcount, a hand-authored skill pack. Work2Skill is the induction side of the same story.

Branching skill tree, illustrating a Work2Skill agent skill induced from a human demonstration

How does Work2Skill work?

According to the paper, there are three parts.

1. WorkRecorder. It captures screen video, mouse and keyboard events, and accessibility-tree information where available, all aligned on a single timeline. The alignment matters: a raw click coordinate is meaningless until it is tied to the widget that was under the cursor and the screen state around it.

2. WorkDemo-350. A dataset of 350 screened demonstrations, about 8.48 hours in total, covering 64 MyPCBench Bounded Action tasks and a 286-task subset of OSWorld-Verified. Screening means low-quality recordings were filtered out before skills were induced from them.

3. Agentic skill induction. Rather than feeding the whole recording to a model once, the system uses a tool-using agent that can inspect images and video on demand, drafts a SKILL.md, and revises it. The finished skills capture workflow rules, step dependencies and completion checks, and they run without the original recording. That last property is what makes them reusable: the agent at test time reads only the text.

The ablations back up the capture design. Removing indexed images, UI annotations or key logs lowered the Score, which suggests each signal contributes something the others do not.

What do the results show?

The authors compared four setups: No Skill, ShowUI-Aloha, Demo2Tutorial and Work2Skill, each paired with four execution models: GPT-5.5, Opus-4.8, Qwen3.5-35B-A3B and Qwen3.5-9B. Work2Skill had the highest Score and success rate among the compared methods.

Cursor reaching toward a button, representing a Work2Skill computer-use agent following an induced skill on a GUI task

Against No Skill:

• Mean Score improved by 4.67 to 10.54 points on MyPCBench and 3.11 to 7.50 points on the OSWorld-Verified subset, depending on the executor.
• On MyPCBench tasks whose requirements were changed from the demonstrated ones, Score improved by 3.93 to 8.64 points. That is the more interesting number, because it tests whether the skill captured the workflow's rules rather than memorizing one task.
• Skills transferred between models. A skill induced with one model helped GPT-5.5 and Qwen3.5-9B, although the smaller executor remained below the larger one.

These are benchmark deltas of a few to about ten points, not a transformation. We have not run the released code ourselves, so treat the figures as the authors' reported results. The paper's own comparison methods are also research baselines, not production tools.

What does this mean for people building agents?

Skills are becoming a learnable artifact. If a recording can yield a decent SKILL.md, then onboarding an agent to a niche app looks more like recording a screen share than writing prompts. For internal tools with no public documentation, that is the interesting use case.

Transfer across models is a practical win. A skill is plain text, so one induced with a strong model can plausibly be run by a cheaper one. The paper's GPT-5.5 to Qwen3.5-9B transfer supports that, with the caveat that the small model still underperforms. For cost-conscious teams, see how benchmarks of this kind are judged in our write-up on SWE-Game, where even strong coding agents stay below 60 percent.

Privacy and security need thought. A recorder that captures screen video and keystrokes will capture passwords and customer data too. Anyone adopting this pattern should scrub recordings and review induced skills before sharing them. If your agents act on a desktop, also look at how agents point at and read the screen in BigArrow.

It complements, not replaces, exploration. Other recent work lets agents learn by exploring on their own, for example Induction Labs' Intrinsic Discovery. Demonstrations carry human intent and domain rules that exploration struggles to discover; exploration scales without a human. Expect hybrids.

Limitations to keep in mind

• A human must demonstrate each workflow first, so the cost moves from prompt writing to recording and screening.
• The evaluation covers MyPCBench Bounded Action tasks and an OSWorld-Verified subset, not arbitrary enterprise software.
• Gains vary widely by executor, from under four points to over ten, and smaller models remain behind.
• The paper does not name a corresponding author on the page we read, and results have not been independently reproduced as far as we know.
• Skill induction adds its own agent loop, so there is an extra compute step before the benefit arrives.

What would a Work2Skill-style skill look like in practice?

The paper describes induced skills as capturing three things: workflow rules, step dependencies and completion checks. Translate that into a file and you get something like a short playbook. A rule might say that a report must be exported only after the date filter is applied. A dependency might say that a dialog has to be closed before the next menu responds. A completion check might say the task is only done when a confirmation banner appears, not merely when the last button was clicked.

That structure is useful for a reason beyond benchmarks. Agents on a GUI often fail silently: they click the right things in the wrong order, or stop one step early. Completion checks give the agent a way to verify itself, and step dependencies cut the number of wasted actions. These are the same properties that make hand-written skills valuable in coding agents, which is why the idea of generating them automatically is attractive.

A practical workflow for a team looks like this:

1. Pick one workflow that agents currently get wrong, ideally one with a clear success condition.
2. Record two or three clean demonstrations, with sensitive data removed or replaced.
3. Induce a draft skill, then read it as you would a pull request: are the rules correct, and does it avoid task-specific constants?
4. Test it on a variant of the task, since the changed-requirement results in the paper show that is where a good skill pays off.
5. Share it, version it, and retire it when the application changes.

Step four matters most. A skill that only works on the exact demonstrated task is a macro, not knowledge. The paper's separate reporting of changed-requirement tasks is a good reminder to test for that in your own setup.

How to try it

The authors point to the GitHub repository china-qijizhifeng/Work2Skill for code and data. We have not verified setup steps, so check its README for requirements. If you want to experiment without the recorder, the same idea works by hand: demonstrate a task, write down the rules and completion checks, and give the agent a SKILL.md. Our Claude Code commands reference shows where skills plug into a working agent environment.

Related reading

• What are agent skills? The complete guide
• Tencent TeamAI CLI: git-based team skills
• Headcount: a Claude skill for an org in a box
• Induction Labs: Intrinsic Discovery
• BigArrow: agents that point at the screen
• SWE-Game benchmark
• Primary source: Work2Skill on Hugging Face Papers

Figures are from the paper page as of October 12, 2026 and may change in later versions.