Mobile AI Factory asks a practical question: can an agent build the right small language model for your task and your exact phone without weeks of manual tuning? University of Washington researchers say yes — their arXiv 2611.13509 system orchestrates data mining, synthetic example generation, hyperparameter moves, and quantization search while measuring real Android devices across inference engines. Code lives at github.com/evanly-gh/SLM_Factory.
TL;DR
| Piece | What it does |
|---|---|
| Orchestrator LLM | Chooses whether to tune hyperparameters, mine user data, or synthesize examples |
| Guardrail rules | Roll back changes that hurt accuracy |
| Search loop | Walks model sizes and quantizations with adaptive accuracy targets |
| Phone probe | Benchmarks latency and quality on the target handset, not a datacenter surrogate |
| Output | Fine-tuned, quantized SLM ready for on-device deployment |
Why on-device SLMs still need factories
On-device chip inside a handheld device
Running models locally keeps messages, health notes, and enterprise tickets off cloud APIs — a story explainx.ai has tracked through Apple Intelligence disk fights, AirLLM memory tricks, and Qualcomm phone forecasts. The catch is per-task fine-tuning and per-device quantization: a 3B Q4 bundle that flies on a Pixel can stutter on a budget Moto.
Manual sweeps burn GPU weeks. Mobile AI Factory automates the sweep with an agentic controller — closer to Together Compiler agents optimizing kernels than to another leaderboard merge.
How the system works
Mobile AI Factory orchestrator loop diagram from the paper
Image: University of Washington (arXiv 2611.13509)
Authors describe a loop:
- Task spec in, phone spec in. You declare the behavior (entity extraction, summarization, etc.) and the Android target.
- Orchestrator decisions. A frontier-class LLM (paper evaluations include Claude Opus 5.5 as oracle and competitor) picks the next action: adjust learning rate, generate synthetic rows, or pull mined examples.
- Rollback rules. Hard-coded checks undo moves that drop accuracy — a guardrail pattern familiar from agent eval harnesses.
- Device-aware search. The factory benchmarks candidate models with the same inference backends you would ship, because best config changed per phone in their four-device study.
Numbers from the paper page
Hugging Face's summary — matching the paper page — states:
- 10 tasks, 4 Android phones
- Beats five-shot frontier models on five tasks (up to +0.34), matches on three
- Within 0.03 of exhaustive training over 27 variants on entity extraction with 19x less GPU time
- Claude Opus 5.5 with full hardware specs still mis-picks requirements on 25% easy or moderate and 60% hard cases without live phone measurements
Those figures are authors' experiment results, not independent audits. Reproduce on your hardware before betting a product roadmap.
Try it: SLM_Factory repo
The README-linked SLM_Factory repository is the builder entry point. Expect:
- Python tooling to define tasks and devices
- GPU resources for fine-tuning passes
- USB-connected Android phones or emulators for measurement passes — emulators may lie about NNAPI performance, so physical hardware is safer
Pair the repo with our Qwen Code terminal agent guide if you want an orchestrator CLI beside the factory — but keep secrets off-device when possible.
When a cloud model still wins
On-device SLMs lose when:
- Knowledge cutoff matters and you need fresh retrieval — see Bedrock agentic retrieve patterns.
- Context spans hundreds of pages — prefix caching for long agents targets cloud latency instead.
- Safety classifiers must stay centralized for policy updates, as with OpenAI CSAM classifiers.
Mobile AI Factory is for narrow, repeatable tasks where latency and privacy trump world knowledge — keyboard assist, field forms, offline inspectors.
Research lineage and open questions
The team sits in UW's ecosystem alongside Gollakota's wireless and sensing work; voice and on-device themes rhyme with Liquid AI audio decision models but Mobile AI Factory stays text-first.
Open engineering questions readers should track:
- Data consent when mining on-phone examples
- Drift after OS updates change NNAPI drivers
- Evaluation leakage if synthetic data mirrors test sets — the paper claims held-out tests, but your fork needs discipline
Orchestrator loop in more detail
The factory treats model building as a sequential decision process:
- Parse task spec into metrics (F1, latency ceiling, memory cap).
- Pick a base SLM from a small ladder (sub-3B to ~8B class — exact list in repo).
- Choose quantization (INT4, INT8, mixed) based on phone RAM telemetry.
- Generate or mine data until validation loss plateaus or budget expires.
- Measure on device across multiple inference backends (NNAPI, CPU, vendor GPU delegates).
- Rollback if accuracy drops below adaptive floor.
The orchestrator LLM is not magic; hard rules prevent catastrophic forgetting when synthetic data overwhelms real user examples. That hybrid — neural policy plus symbolic guardrails — mirrors Hugging Face PyBot swarms and industrial agent taxonomies.
Phone measurement: why emulators lie
Android emulators report ideal NNAPI latencies. The paper stresses real devices because:
- Thermal throttling changes clocks mid-inference.
- OEM NNAPI drivers ship months behind AOSP.
- Battery saver modes kill background batching.
If you replicate SLM_Factory, budget four physical phones minimum — flagship, mid, budget, and one older security-patch lag device common in field work.
Task-by-task intuition (from paper summary)
| Task class | Why on-device wins | Failure mode |
|---|---|---|
| Entity extraction on forms | Short context, repeatable labels | Missing rare kanji without locale data |
| Message summarization | Privacy | Drift when users switch languages |
| Lightweight classification | Tiny models suffice | Cloud API cheaper if volume low |
Authors report beating five-shot frontier models on eight of ten tasks. Read that as narrow tasks with good local data — not "replace Claude for coding."
Step-by-step: first SLM_Factory experiment
Prerequisites: Linux workstation with NVIDIA GPU, Android 14+ device, USB debugging enabled, Python 3.11+.
- Clone
https://github.com/evanly-gh/SLM_Factory. - Create a toy task JSON with 200 labeled examples you own — do not mine user chats without consent.
- Plug in one target phone; run the measurement harness once to cache latency tables.
- Start with the smallest base model and allow the orchestrator five search rounds only — watch costs.
- Export ONNX or TFLite artifact the repo supports; install via internal MDM, not Play Store, until QA signs off.
Log energy draw if your product markets sustainability; smaller models mean less heat, a selling point next to liquid AI audio efficiency narratives.
Privacy, compliance, and enterprise mobility
On-device fine-tuning with mined data touches GDPR purpose limitation, Japan APPI, and sector rules (HIPAA, PCI). Document:
- What leaves the device (ideally: only gradients or nothing)
- How long mined examples persist
- Whether the orchestrator calls cloud APIs during search (budget API keys accordingly)
MDM teams should treat shipped SLMs like custom apps: version, revoke, and audit.
When Mobile AI Factory is the wrong tool
Skip it if:
- You need RAG over giant corpora — use cloud retrieval stacks.
- Your task changes weekly — factory search cost amortizes only with stable metrics.
- You lack labeled examples — synthetic-only loops overfit, per paper rollback rules.
Use it when:
- You have stable field workflows (inspectors, warehouse scanners, clinic intake).
- Connectivity is unreliable.
- You already pay per-token at volume where a 1B local model pays back in months.
Connection to wider 2026 SLM trends
Mobile AI Factory sits between:
- Vendor SLMs like Tencent HY4 Flash (cloud-first) and
- Token factories like Sarvam (region-specific hosting).
It is a bring-your-own-task compiler for Android — not a hosted API. That positioning matters for sovereign and offline buyers.
Extended FAQ for mobile ML engineers
Does the factory run on iOS? Paper emphasizes Android phones; no iOS measurements in the summary we read.
Which frontier models did authors use as oracles? Hugging Face summary names Claude Opus 5.5 for misconfiguration tests; confirm in arXiv PDF before citing elsewhere.
Can I use SLM_Factory without an orchestrator API key? Check repo for local model options; budget cloud spend if orchestrator calls Opus-class APIs each search round.
How big are shipped models? Depends on search outcome; expect sub-8B quantized artifacts for phone RAM, not 70B downloads.
Will Google ban sideloaded models? Play Protect policies evolve; enterprise MDM distribution is safer for pilots.
Does 19x GPU savings generalize? Authors report one entity-extraction sweep; replicate on your task before promising finance teams savings.
Is this peer-reviewed? arXiv preprint as of November 2026; not a venue acceptance announcement.
How does it compare to Apple on-device intelligence? Apple's stack is consumer OS integrated; SLM_Factory is researcher tooling for custom tasks.
What about federated learning? Paper focuses on centralized fine-tune plus on-device measure; federated training is out of scope unless you extend the repo.
Subscribe to the arXiv comment thread on 2611.13509 if you want author replies on open issues before camera-ready. Star the GitHub repo to watch release tags. File issues upstream when phone measurements diverge from your OEM hardware.
Related reading
- AirLLM run 70B on 4GB GPU
- Baseten efficient frontier LLM inference guide
- MMFlash vision-language speculative decoding
- Nasiko open runtime agent token gateway
- DIYgod Pinky phone MCP human in the loop
- DeepSeek free chat daily limits
Paper metrics come from arXiv 2611.13509 and the Hugging Face paper page checked November 17, 2026.
