## FreeToken: Making Frontier MoE Models Run Locally
Recent advances in large language models have introduced mixture‑of‑experts (MoE) architectures that deliver strong performance while activating only a fraction of their parameters per token. Models such as Kimi‑K3, GLM‑5.2, and DeepSeek‑V4‑Flash demonstrate that MoE design makes frontier capabilities arithmetically feasible on modest hardware. However, actually running these models has remained costly and infrastructure‑intensive, typically requiring datacenter‑class GPU clusters. FreeToken addresses this gap by rethinking how models are served rather than by shrinking the models themselves.
### What FreeToken Changes
FreeToken reframes the problem as a system‑level one. Instead of assuming that running large models requires expensive GPU servers, it treats a personal computer or workstation as a unified, elastic inference platform. The system continuously maps computation and model state across whatever GPUs, CPUs, memory, and interconnect bandwidth are actually available. This approach enables interactive speeds for 35‑billion‑parameter models on an 8GB laptop GPU, 284billion on a gaming desktop, and 753‑billion GLM‑5.2 on a single workstation card.
### Key Technical Mechanisms
Three core mechanisms make this possible:
1. **Bandwidth‑adaptive execution**
FreeToken splits each decoding step between GPU cache fills and CPU execution based on measured PCIe and host memory bandwidth (a policy called q∗). This avoids stalling on a saturated link and ensures that latent expert evaluations are handled where they are most efficient.
2. **Semantic‑aware caching**
During prefill, full‑layer double buffering hides data movement behind computation. A shared LRU expert cache tracks router behavior across all MoE layers, reducing redundant data transfers during decoding.
3. **Elastic memory management**
The GPU expert cache can be rebuilt at safe points under a new VRAM budget without restarting the engine or reloading the model. Experts are streamed directly from disk into host memory, eliminating GPU warmup overhead.
### Performance and Efficiency
On an RTX 5090, FreeToken sustains 77–83 tokens per second for Qwen3.6‑35B‑AF16 and 22–25 tokens per second for DeepSeek‑V4‑Flash (MXFP4), outperforming strong open‑source and proprietary baselines by 1.5–2.3×. Worst‑case time‑to‑first‑token stays below 44seconds across all tested workloads, while comparable systems such as llama.cpp, Ollama, and KTransformers reach up to 946seconds. On consumer hardware, an 8GB RTX 4060 laptop achieves 39.3 tokens per second for 35B, exceeding typical production decode speeds for Codex.
### Availability and Deployment
FreeToken is released under the Apache2.0 license and is available on PyPI as `freetoken` v0.1.2. It provides a CLI with OpenAI‑ and Anthropic‑compatible endpoints and ships as a one‑click desktop app for Windows and Linux. The tool targets Linux x86_64 systems with NVIDIA GPUs (driver r580+, CUDA 13), making it practical for developers, startups, and small teams who already own consumer GPUs.
### Data and Verification
An independent audit of the project’s claims shows that most arithmetic is consistent with published figures, and key results such as throughput and cache‑miss behavior are directly reproducible. Some flagged items highlight areas where prose may overstate hardware isolation (e.g., “single GPU”) or compare metrics with slight shifts in denominator definitions. Overall, the calculation integrity is sound, and the approach fills a clear gap between model accessibility and practical deployment.
### Key Takeaways
– FreeToken dynamically partitions MoE cache misses using measured host‑GPU bandwidth instead of static offloading rules.
– It preserves exact bitwise outputs of MoE routing without modifying model architecture.
– It delivers 1.5–2.3× decode throughput over leading open‑source stacks while keeping latency under 44seconds.
– It enables 35‑billion‑parameter inference on 8GB laptops and 753‑billion‑parameter inference on a single workstation GPU.
– It is open‑source, easy to install, and ready for local deployment today.
FreeToken represents a practical step toward making frontier MoE models accessible outside expensive datacenters, leveraging system innovation rather than model compression. For teams looking to run large language models locally without sacrificing quality or responsiveness, it offers a compelling, deployable solution.
Thank you for reading
