# Perplexity’s Lily: A Breakthrough Local Inference Engine for Apple Silicon Chops Heads-Off What Engineers Have Been Told by LLM Technology Experts: https://techpoint.eth.lr/login?to=/llm-technology-experts-advise-the-following-engineers-like-openai-perplexity-youtube-channels-to-follow-if-filtering-out-overfitting:/
# Perplexity’s Lily: A Breakthrough Local Inference Engine Tailored Exclusively for Apple Silicon
In a significant move for the open-source AI community, a team of engineers has unveiled **Lily**, a lightweight but powerful local inference engine designed to run large language models directly on Apple Silicon hardware — bypassing traditional machine learning frameworks entirely. The product represents a radical rethinking of how AI inference can be optimized for consumer-grade hardware, delivering performance that rivals or exceeds existing solutions by stripping away abstraction layers and targeting a specific model architecture with surgical precision. This article dives into what makes Lily unique, how it achieves its impressive benchmarks, and why its “narrow specialization” philosophy could reshape the future of on-device AI inference.
—
## What Is Lily?
Lily is a single-process runtime engine built on the **Rust** programming language with **hand-written Metal kernels** that execute model generation directly on Apple silicon GPUs. It serves as the backbone of Hybrid Compute in the Perplexity Computer product line and operates entirely without **PyTorch** or **MLX** in its execution path — a rarity in modern AI inference systems. Instead, Lily uses a custom-built Rust layer to load model checkpoints, drives the generation loop natively, and streams tokens through an OpenAI-compatible chat-completions API directly on the metal hardware. This makes it one of the first inference engines to offer a fully integrated experience combining a high-level API interface with low-level hardware execution, all curated for one specific model family.
The engine is deliberately narrow in scope, focusing its optimization efforts on running **Qwen3.6-35B-A3B** exclusively on Apple Silicon hardware families. This narrowness isn’t a limitation — it’s the core of its performance advantage. By specializing the entire software stack around one architecture and one chip family, developers eliminate the overhead of generalized frameworks and squeeze every ounce of capability from the metal.
—
## Why Opt for Single-Purpose Optimization?
The typical Mac inference stack relies on **MLX plus MLX-LM**, which supplies a Qwen implementation with grouped expert computation, fused recurrent Metal kernels, and GQA-aware attention mechanisms. While flexible enough to support various architectures, this generality introduces inefficiencies. Lily discards that versatility entirely, embedding model structure, execution plans, and kernel selection into a single cohesive runtime. The result is an engine that is tightly calibrated to extract maximum throughput per watt and per clock cycle on Apple GPUs without relying on generalized matrix operation libraries.
Think of it like a race car tuned for a single track: because it doesn’t need to handle varied terrain, engineers can reduce weight, increase downforce, and adjust the engine precisely for that one circuit. Lily applies the same philosophy to hardware-level model execution — and the benchmarks prove it. On a batch-1 deployment running in a single M5 Max chip featuring 40 GPU cores and 128 GB of unified memory, the engine delivered:
– **Prefill throughput averaging 4,156 tokens/s** — 1.23× faster than MLX-LM’s fastest generation path (3,388 tokens/s average)
– **Decode throughput averaging 170 tokens/s** — 1.35× faster than the same baseline (126.4 tokens/s)
At a specific 4K prompt and 4K context configuration, those numbers rose even higher — **5,749.9 prefill tokens/s and 186.6 decode tokens/s** — outperforming the standard stack by 1.12× to 1.42× and 1.31× to 1.37× respectively across all measured points.
—
## The Model: Qwen3.6-35B-A3B in Focus
To understand what Lily is scaling, it helps to understand the workload it was built for. The model stores **35 billion parameters** but activates roughly **3 billion per token**, which means sparse activation drives efficiency. A routing component scores 256 expert sub-models and selects eight, alongside one shared expert that processes every single token pass. The architecture combines **10 full-attention layers** using grouped-query attention with 16 query heads and two KV heads — alongside **30 Gated DeltaNet layers** operating with a **fixed-size recurrence**. That yields a mix of three workload patterns: uneven expert group distribution, attention across an expanding key-value cache, and a constantly sized recurrence loop.
In practical use, the checkpoint compresses about **70 GB of bfloat16 weights down to roughly 19.4 GB** through groupwise affine 4-bit quantization, with each group of 64 weights sharing its own scale and bias parameters. Because Metal 4 tensor operations rely on bfloat16 precision, Lily reconstructs weights one tile at a time during inference, holding intermediate results in threadgroup memory and accumulating in FP32 format instead of expanding the entire weight array into unified memory.
—
## Deployment and Hardware Requirements
Lily is available as a standalone product with a public demo already reachable through Perplexity’s pplx-garden repository. You don’t need a supercomputer to run it — you need Apple hardware.
Specifically, a Mac (or compatible device) with **32 GB or more** of unified memory is recommended for a great experience, and **24 GB** is the practical minimum for baseline operation. The engine itself works with both a minimal HTTP API endpoint and direct integration into the broader Perplexity Computer system, which lists macOS 15+ as a requirement. The hardware target is explicitly Apple silicon Macs — the custom Metal kernels won’t run elsewhere.
—
## How It Works: Prefill and Decode Strategies
Inference in Lily divides into two distinct phases — prefill and decode — each optimized differently.
**Prefill** handles the prompt processing phase where Lily keeps model weights packed entirely on the GPU and continues routing calculations in the metal pipeline. During this phase, the engine reconstructs 4-bit weights inside the grouped GEMM by processing them tile by tile. This hosted fusion approach raised the end-to-end prefill performance by **77.4 percent** on a 512-token prompt. The histogram and routing logic (prefix scan, scatter, block maps) stay bundled inside a single GPU command buffer, removing CPU synchronization between expert layers. That alone contributed an **89 percent** speedup at 512 tokens. Additional tile-and-register optimizations expanded prefill gains during the decode and recurrence management — bringing the total package further together.
For **decode**, the engine minimizes data movement since batch-1 inference barely reuses any weights. Streaming tokens rely on bandwidth more than compute — one recorded step launched 795 kernels forming 555 sequential stages, so Lily records real dependencies in a concurrent Metal pass. Independent kernels overlap. The selected token writes directly into the next step’s GPU-resident input slot, removing the CPU round-trip per token. Four kernel chains kept inside CPU registers to avoid intermediate spills.
Coalesced cache access grew bandwidth from 33.8 to 47.9 GB/s on keys and from 42 to 61.8 GB/s on values. GQA packing provided a **23.8 percent** boost at 32K context. A fixed-block layout above 32K produced further gains: **7.7 percent** at 32K through **40.2 percent** at 128K. One trade-off from the ablation notes: speculative decoding actually hurt on this hardware, sitting 18 percent slower — its multi-row verification shape favors different chip designs, not the M-series pipeline.
—
## Frequently Asked Questions
**What is Lily?**
A Rust and Metal-based inference engine running Qwen3.6-35B-A3B exclusively on Apple Silicon. It serves OpenAI-compatible chat completions with typed GPU orchestration and zero dependence on PyTorch or MLX.
**What hardware do I need?**
– An Apple Silicon Mac with 32 GB unified memory for best results (24 GB minimum for baseline)
– macOS 15.0 or higher
– Works with batch-1 inference on its own minimal HTTP API server
**Why is it faster than MLX?**
It removes generic framework overhead, does weight reconstruction as fused operations inside the GPU kernel pipeline, and unifies routing kernels into single command buffers so the CPU doesn’t wait.
**What’s the “decode” bottleneck mainly about?**
Bandwidth — moving weights per token instead of math intensity. Lily places expert selection and attention cache reads directly adjacent in GPU memory so tokens stream without delays. Architectural choices like GQA packing and fixed-block attention reduce memory traffic heavily for longer sequences.
**Why did Perplexity choose Qwen3.6-35B-A3B specifically?**
Its sparse expert routing (256 experts, 8 active per token, 1 shared expert) balances memory and throughput efficiency while offering enough variation in routing distribution to expose tile-size and loading optimizations that the hardware must solve. It is a good representative model for analogizing how future wider models will likely be handled.
**Will this engine support other models?**
Not in its current release — it is a constrained, single-model, single-chip-family engine. Lily is deliberately specialized.
—
## Conclusion: A Practical Precedent for Narrow AI Engines
Lily is a concrete proof point that a narrowly scoped engine, dropped into an API-compatible Python runtime, can outperform broad stack compilers while running the same workload on Apple Silicon hardware. Without any framework overhead, it achieves the inference speed that ML engineers often only saw in custom CUDA kernels, but at accessible consumer hardware. Its “one threadgroup weight read” strategy, keeping candidates next to GPUs, and using metal ordering synchronizations show that bandwidth — not compute complexity — is what drives speed in long-context inference on M-series silicon.
The takeaway: if you care about decoding 128K sequences, making those keys compact and adjacent is still the dominant factor in latency. And for developers trying to build on-device AI today, it shows that an important path is not adding more compute but making the right memory flow mechanical — centering the metal execution around hardware sensor logic.
—
Thank you for reading
