# Ternary Bonsai 2 27B: Running a 27B-Parameter Language Model in Under 6 GB
## Overview
A new entrant in the world of efficient language models has emerged with the release of Ternary Bonsai 2 27B. Developed by PrismML, this model is a ternary-weight adaptation of the Qwen3.8 27B architecture. By converting nearly all model weights into a three-value format — -1, 0, and +1 — the developers have dramatically reduced the model’s memory footprint from over 53 GB down to just 5.93 GB. This makes a model of this size accessible on consumer-grade laptops and single consumer GPUs that would otherwise be unable to run it.
Despite the aggressive compression, the model retains 98.2% of its parent model’s performance across a suite of 20 standardized benchmarks. It supports both text and image inputs with a context window extending to 262,000 tokens. Early demonstrations have shown the model powering coding agents and computer-use tasks on an RTX 5090 graphics card.
The model arrived roughly two months after the initial release of the first Bonsai 27B model, whose ternary variant had retained approximately 95% of its parent’s capabilities. This second iteration represents a significant step forward in both compression efficiency and quality retention.
## Technical Architecture
The model preserves the full Qwen3.8 27B architecture with 27.36 billion total parameters. These are distributed as follows:
– **Language backbone:** 24.35 billion parameters
– **Embeddings and language model head:** 2.54 billion parameters
– **Vision tower:** 0.47 billion parameters
The backbone employs a hybrid attention mechanism, combining approximately 75% linear attention layers with 25% standard full-attention layers. This design balances computational efficiency with the ability to capture long-range dependencies in text.
### The Ternary Weighting Scheme
The core innovation lies in the ternary quantization approach. Every weight in the model — across embeddings, attention projections, MLP projections, and the language model head — is converted to one of three values: -1, 0, or +1. Each group of 128 weights shares a single FP16 scale factor, which is used during the forward pass to recover approximate magnitude information.
Only 26.2 million parameters, representing 0.0976% of the total, remain stored in higher precision. These are the weights belonging to the recurrent state path and the normalization layers, which proved too sensitive to aggressive quantization. In the GGUF format, the vision tower is packaged separately as a 0.63 GB file and is loaded only when processing image inputs.
### Storage Format and Bit Efficiency
Each ternary value carries log2(3), or approximately 1.585 bits of information. When combined with the 16-bit FP16 scale shared across each group of 128 weights, the effective storage cost is about 1.71 bits per weight. Including the small set of high-precision tensors brings the practical figure to 1.72 bits per weight.
The whitepaper describes two different packing strategies in GGUF format. PTQ1_0 stores trits densely at 1.76 bits per weight, resulting in a file size of 5.93 GB. PQ2_0 stores each trit in a dedicated 2-bit slot at a larger 7.25 GB, which offers cheaper unpacking during inference at the cost of more disk space.
### Rotated Weight Basis
An additional optimization involves storing the weights in a rotated basis. PrismML applies a blockwise Hadamard rotation with a block size of 1,024 parameters before performing the ternary assignment. At runtime, the matching transformation is applied to activations before each matrix multiplication. This technique, drawn from prior research on quantization-aware training, helps distribute weight magnitudes more evenly and reduces quantization error. The rotation itself is folded into the stored weights offline and does not add any additional bit cost.
## Deployment Requirements
The model weights are released under the Apache 2.0 license and can be run today on a 16 GB laptop or a single 24 GB GPU. Users will need either PrismML’s fork of llama.cpp or the company’s MLX runtime to run the model. The vision tower component adds 0.63 GB on top of the base model when processing images.
## Benchmark Performance
The model was evaluated across multiple capability categories. In overall performance, it achieved 83.9 out of a possible 85.4 (the Qwen3.8 FP16 baseline), retaining 98.2% of the parent’s 20-benchmark average.
| Capability Area | FP16 Parent Score | Ternary Model Score | Retention |
|—|—|—|—|
| Knowledge & Reasoning | 86.66 | 83.95 | ~97% |
| Math | 97.06 | 96.57 | ~99.5% |
| Coding | 82.17 | 81.58 | ~99% |
| Agentic & Tool Calling | 79.74 | 77.57 | ~97% |
| Instruction Following | 81.25 | 82.66 | ~102% (outperforms parent) |
| Vision | 81.64 | 78.59 | ~96% |
| Long-horizon Agents (SWE-bench Verified) | 80.6 | 60.8 | ~75% |
The model notably outperforms its FP16 parent on instruction-following benchmarks, suggesting that the ternary compression process may have a regularizing effect on the model’s behavior. The biggest quality gap appears in long-horizon agent tasks, where retention drops to roughly 75%.
## Speed and Hardware Compatibility
Decode performance has been measured across a wide range of hardware. On an RTX 5090 using the PQ2_0 packing, the model achieves 134.4 tokens per second in token-generation mode (TG128) and 1,901 tokens per second in prefill mode (PP512). On an H100 NVL, the figures drop to 81.6 tok/s for generation and 1,098 tok/s for prefill. Apple Silicon laptops using Metal backend also support the model, with an M5 Max achieving 46.8 tok/s in generation mode.
The PTQ1_0 packing generally wins on Ada-generation GPUs and the L40S, while PQ2_0 performs better on Blackwell, Hopper, Ampere, and Apple Silicon hardware. The difference comes down to the cost of unpacking 2-bit slots versus the slightly denser but more expensive-to-decode ternary packing.
## Frequently Asked Questions
**Q: What does “ternary” mean in this context?**
A: Ternary refers to the use of three discrete values — -1, 0, and +1 — to represent every weight in the model. This is in contrast to standard floating-point formats like FP16 or FP32, which use a continuous range of values. The three-value system dramatically reduces memory usage while still allowing the model to represent both positive, negative, and zero activations.
**Q: Can this model run on a standard laptop?**
A: Yes. The model weights require only 5.93 GB of storage and can run on a laptop with 16 GB of RAM using the provided llama.cpp fork or MLX runtime. Users should note that additional memory is needed for the KV cache, activations, and runtime overhead, so a 16 GB system will be tight but feasible. A 24 GB GPU or a system with more RAM provides more comfortable headroom.
**Q: What is the difference between PTQ1_0 and PQ2_0 packings?**
A: PTQ1_0 packs ternary values densely at 1.76 bits per weight, resulting in a 5.93 GB file. PQ2_0 stores each ternary value in a 2-bit slot, leading to a larger 7.25 GB file. PQ2_0 is cheaper to unpack during inference, making it faster on certain hardware platforms, particularly those without native ternary support. PTQ1_0 is more storage-efficient but requires more compute to decode.
**Q: Why does the model retain over 98% of its parent’s average score despite such aggressive compression?**
A: The combination of the rotated basis, the shared per-group scale, and the preservation of a small set of high-precision parameters (recurrent state and normalization weights) helps maintain the model’s representational capacity. Additionally, the Qwen3.8 27B architecture itself is known for being relatively robust to quantization, and the two-month iteration gap from the first Bonsai release allowed the developers to refine their approach.
**Q: Does the model support image input?**
A: Yes. The model includes a 0.47 billion parameter vision tower that is loaded separately as a 0.63 GB file in GGUF format. When processing image inputs, this vision tower is activated and its outputs are integrated into the language model’s processing pipeline. The model supports a 262K-token context for multimodal inputs.
**Q: How does the instruction-following performance compare to the parent model?**
A: Interestingly, the ternary model slightly outperforms the FP16 parent on instruction-following benchmarks, scoring 82.66 versus the parent’s 81.25. This suggests that the quantization process may act as a form of implicit regularization that improves the model’s ability to follow instructions, possibly by reducing overfitting to noisy training patterns.
**Q: What hardware is needed for the fastest inference?**
A: The fastest reported generation speed is 142.5 tokens per second on an RTX 5090 using the PQ2_0 packing. For prefill operations, an RTX 5090 achieves 4,121 tokens per second. High-end Hopper and Blackwell GPUs such as the H200, B200, and RTX PRO 6000 Blackwell also deliver strong performance, though the RTX 5090 offers the best price-to-performance ratio among measured devices.
## Conclusion
Ternary Bonsai 2 27B represents a compelling demonstration that large language models can be compressed to a fraction of their original size with minimal quality loss. By leveraging ternary quantization, blockwise Hadamard rotation, and careful preservation of critical high-precision parameters, the developers have created a model that runs comfortably on consumer hardware while retaining nearly all of the capabilities of its full-precision counterpart.
The model’s ability to outperform its parent on instruction-following tasks and maintain near-parity on coding and math benchmarks suggests that ternary compression is maturing as a practical deployment strategy. The availability of two packing formats — one optimized for storage and one for speed — provides flexibility for different use cases, from local experimentation to production deployment.
As the ecosystem of efficient inference tools continues to grow, models like Ternary Bonsai 2 27B are helping to democratize access to powerful language models, making them available on the hardware that researchers, developers, and hobbyists already own.
Thank you for reading
