Lily: Perplexity's Custom Inference Engine for Apple Silicon

Perplexity just open-sourced Lily, a local inference engine it built from scratch for exactly one purpose: running Qwen3.6-35B-A3B as the local half of Hybrid Compute in Perplexity Computer, the company's Mac app. The engineering write-up, "Optimizing On-Device Inference for Apple Silicon," went up on Perplexity's official blog on September 1, 2026 — and unlike most model-launch posts, it's mostly ablation tables.

That matters. Hybrid Compute already ships to every Mac app user: it routes sensitive agent steps (bloodwork, tax documents, litigation files) to a model running locally on the user's own Apple Silicon chip, while everything else still hits the cloud. For that split to feel like one product instead of two, the local half has to keep pace with the cloud half — fast prompt processing and a token-generation rate that doesn't make the user wait. Lily is the engineering answer to that constraint: no PyTorch, no MLX in the execution path, just a single Rust runtime with hand-written Metal GPU kernels tuned to Qwen3.6-35B-A3B's exact shape.

TL;DR

Why the local model has to be fast, not just present

Hybrid Compute splits agentic tasks between frontier cloud models — for research and heavy reasoning — and a local model that works directly with a user's private files and apps, without that data leaving the machine. The design only works if switching between the two is invisible. A local model that's merely capable isn't enough; it needs prefill (processing the prompt) and decode (generating tokens one at a time) fast enough that a user mid-task doesn't notice which half of the system just answered.

That's a narrower, harder problem than "run an LLM on a Mac." MLX and MLX-LM — Apple's own open-source ML framework and its LLM library — are the standard way to do that today, and they're genuinely good: general-purpose, broad model support, solid default kernel choices. Lily's premise is that going further requires giving up generality entirely and optimizing around one model's exact dimensions instead.

The architecture that forces the issue: MoE plus Gated DeltaNet

Qwen3.6-35B-A3B isn't a conventional dense transformer, and that's the whole reason Lily exists. Two design choices in the model create three genuinely different computational shapes inside a single forward pass:

Sparse Mixture-of-Experts (MoE). The model has 35 billion total parameters, but only about 3 billion are activated per token. A router scores 256 expert subnetworks and picks 8, plus a shared expert that runs on every token regardless. This is efficient — most weights sit idle for any given token — but it's uneven: different experts get wildly different numbers of tokens routed to them on a given pass, so it isn't neat, uniform batch work the way a dense matmul is.

Hybrid attention. 10 layers use full grouped-query attention (GQA) — 16 query heads sharing just 2 key-value heads, which shrinks the KV cache — combined with 30 "Gated DeltaNet" layers. Instead of a growing attention cache, Gated DeltaNet compresses history into a small fixed-size recurrent state: a gate controls how much old state survives, and a "delta update" folds in each new token. These layers are sequential by nature during decode, but during prefill they can be restructured into parallelizable block computation — a very different execution path depending on which phase you're in.

Net effect: uneven MoE routing, attention over a growing cache, and fixed-size recurrence that runs two different ways — three shapes, one model. A general kernel library picks reasonable defaults for all three. Lily coordinates all three specifically around this model's dimensions and Apple Silicon's execution paths.

Why Apple Silicon isn't just a smaller datacenter GPU

Apple Silicon's unified memory means CPU and GPU share one physical pool — a 35B-parameter model can stay resident without a separate GPU copy — but reading that memory still costs real bandwidth. Perplexity's framing, stated directly in the post: Apple Silicon is "not a smaller datacenter GPU," it's a genuinely different platform with its own ceiling.

The prefill/decode split matters here specifically:

• Prefill processes many prompt tokens at once, reusing each block of model weights across many rows — GEMM (matrix-matrix multiply) work, well suited to the M-series GPU's Neural Accelerators (Metal 4 tensor operations).
• Decode generates one token at a time — the realistic single-user "batch 1" case — and barely reuses weights at all. That's GEMV (matrix-vector multiply) work: memory-bandwidth-bound, and better served by the GPU's regular vector ALUs than by matrix-optimized Neural Accelerators.

MLX-LM already picks reasonably optimized kernels for each. Lily's edge is coordinating expert routing, recurrent state, and KV cache access specifically around Qwen's exact dimensions and which of those two paths — matrix or vector — each operation should actually run on.

The measured wins — real ablations, not marketing numbers

Perplexity ablation-tested each optimization independently, which is the part worth taking seriously. These are the gains that actually moved the needle:

The two GEMM-side wins — in-kernel dequantization and GPU-resident MoE routing — dwarf everything else because MoE work dominates roughly 90% of prefill time. The 4-bit checkpoint itself is 19.4GB versus roughly 70GB in full bfloat16; doing the unpacking inside fast on-chip memory instead of writing the expanded weights back to memory first avoided a full extra read/write pass.

The long-context attention-layout switch is the most broadly useful finding for anyone running local chat sessions: the payoff scales up sharply with context — negligible at 32K, substantial at 128K — which argues for context-length-aware kernel selection generally, not just in this one engine.

What didn't work — and why that's the credible part

Perplexity's most useful disclosure is a negative result: speculative decoding made decode 18% slower, not faster. Speculative decoding normally works by having a small draft model propose several tokens, which the big model then verifies in one pass — usually a win because verification is cheaper than generation. On this workload it backfired for two concrete reasons:

1. Verifying multiple draft tokens at once creates small, irregular batches — an inefficient shape on Apple Silicon's memory-bandwidth-bound decode path.
2. Different draft tokens often route to different MoE experts, which increases how much expert-weight data has to be read from memory per step — directly undermining the point of speculation.

Perplexity is explicit that this is a local, single-GPU finding, not a universal one — their datacenter-scale Qwen deployment on Nvidia Blackwell hardware does use speculative decoding successfully, under different memory and batching conditions. Several other optimizations (broader kernel fusion, accelerating the router itself, more aggressive operator combining) also stopped paying off once the easy wins were captured. Perplexity reports their MoE matrix operations already hit 90–98% of the theoretical fastest possible memory-read rate for that access pattern — a real hardware ceiling, not a tuning gap.

Headline numbers

Perplexity tested Lily on one MacBook Pro — M5 Max chip, 40-core GPU, 128GB unified memory — across ten prompt/context lengths from 256 to 128,000 tokens.

The correctness check matters as much as the speed numbers. Perplexity ran a teacher-forced comparison — both engines predicting the next token from the same reference context at 192 positions, so early divergence can't compound into a misleading gap — and found Lily's output functionally equivalent to MLX-LM's. This is speed without a quality trade-off, at least on the tested benchmark.

How this compares to what else is happening on Apple Silicon

Lily lands in a stretch where Apple Silicon local-inference engineering has been unusually active. TurboFieldfare squeezed Gemma 4 26B into ~2GB of RAM on the same hardware class; DFlash-MLX pushed speculative decoding to ~189 tok/s on M5 Max for a different model — worth noting since Lily found the opposite result for Qwen3.6-35B-A3B, a reminder that speculative decoding's payoff depends heavily on model architecture and hardware, not just raw hardware capability. Community MLX tuning has also moved fast on its own: a recent effort got Gemma 4 26B A4B running 2x faster on Mac through profiling and kernel tuning inside the existing MLX stack, without abandoning it the way Lily does.

That contrast is the practical takeaway. MLX and MLX-LM remain the right starting point for most local-inference projects — broad support, active community, no need to write custom Metal kernels. Lily demonstrates there's real headroom past that starting point once you're willing to commit to one model's exact architecture, and two of its techniques generalize well beyond Qwen3.6-35B-A3B specifically: GQA packing (load each shared KV row once, not once per query head) and context-length-aware attention layout switching are both reusable ideas for any GQA-based model running long contexts locally.

Where this fits versus buying local hardware

If you're weighing local inference against dedicated hardware for agent workloads, Lily's numbers are a data point worth adding to that comparison — see MacBook vs. dedicated GPU for local LLMs for the broader hardware trade-off, and closed-source vs. local open-source alternatives for the cloud-vs-local framing this whole feature sits inside. The economics argument for smaller, more efficient models generally — why speed and efficiency increasingly matter as much as raw capability — is covered in small models have arrived.

Perplexity's stated thesis going forward

Perplexity frames this as more than a one-off optimization exercise. Their stated position: as open-weight models and consumer AI hardware both keep evolving, high-performance local inference will increasingly depend on model-and-hardware-specific engines like Lily, rather than one generic abstraction layer trying to cover everything well. They say future work will widen the approach to more models, more chips, and more serving setups — not just Qwen3.6-35B-A3B on Apple Silicon.

For now, Lily is a narrow, deep answer to a narrow, deep problem: making the local half of Hybrid Compute fast enough that switching between cloud and local never feels like switching at all.

Related reading

• Perplexity Mac Hybrid Compute: local models for sensitive agent steps
• Gemma Chat: offline vibe coding with Gemma 4 and MLX on Mac
• TurboFieldfare: Gemma 4 26B in ~2GB RAM on Apple Silicon
• DFlash-MLX: lossless speculative decoding on Apple Silicon
• Gemma 4 26B A4B runs 2x faster on Mac — community MLX optimization
• Qwen 3.6 27B local dev guide: llama.cpp, OpenCode, dense vs MoE
• What is AI model quantization? Running frontier AI locally
• Closed-source AI vs. local open-source alternatives
• MacBook vs. dedicated GPU for local LLMs

Official: Optimizing On-Device Inference for Apple Silicon — Perplexity Engineering

Benchmarks, architecture details, and optimization figures reflect Perplexity's September 1, 2026 engineering post and its stated M5 Max test configuration. Verify current Lily capabilities and supported hardware against the official GitHub repo and Perplexity's documentation before building on it.