# AnyJev: Turning Large Language Models Into Reliable Decision Engines Without Training
## Introduction
In recent years, large language models (LLMs) have shown remarkable ability to generate text, answer open-ended questions, and synthesize information. However, a vast number of real-world production tasks require something far simpler: picking one answer from a fixed set of options. Routing a customer support ticket, classifying a transaction as fraudulent or legitimate, or assigning a priority level are all examples of decisions that involve selecting from a predefined list rather than generating a free-form response.
A team from Nokia’s applied research group has released **AnyJev**, an open-source Python library designed explicitly for this purpose. AnyJev transforms an open-source LLM into what the developers call a “decision model,” extracting calibrated probabilities directly from the model’s next-token distribution without any fine-tuning or training step.
The library is available on PyPI, licensed under Apache 2.0, and supports both the Hugging Face transformers backend and the vLLM inference engine with shared-prefix scoring for efficient batched serving.
—
## Why This Matters: The Gap Between Text Generation and Decision-Making
Most existing approaches to using LLMs for classification tasks rely on prompting the model to output a label and then parsing that output. This introduces fragility: the answer depends on formatting, parsing logic, and the model’s tendency to generate specific tokens. AnyJev takes a fundamentally different approach.
Instead of generating text and parsing it, AnyJev reads the raw probability distribution over the option labels from the model’s next-token logits. This means every prediction is grounded in the model’s internal confidence scores, and no intermediate text generation step introduces noise or errors.
The library supports three question types out of the box:
– **Choice questions** — selecting one option from a set of K candidates
– **Binary (noul) questions** — yes or no decisions
– **Score questions** — placing an answer into one of several ordered bins
—
## The Core Problem: Why Naive Logit Reading Falls Short
A common shortcut in the open-source community involves restricting the model’s next-token vocabulary to only the option labels and then reading the resulting scores. While this is computationally cheap, Nokia’s researchers identified two significant flaws with this approach.
### Position Bias
LLMs tend to favor certain positions in a list of options. If “Yes” appears first in the list, it may receive an inflated probability compared to when it appears last, regardless of the actual input content. This means that simply reordering the options can change the model’s output.
### Prior Bias
Models develop tendencies toward certain labels based on their training data. For example, a model might systematically prefer “Yes” over “No” or favor more common labels over rare ones, even when the input evidence points elsewhere.
These two biases compound each other, making raw logit readings unreliable for production deployment where consistent, calibrated probabilities are essential.
—
## How AnyJev Works: The L0 and L1 Pipeline
AnyJev introduces a two-level system for producing reliable decision outputs, with each level adding progressively more sophisticated corrections.
### Level 0 (L0): Zero Labels, Maximum Reliability
L0 is the default mode and requires no labeled training data. It applies two complementary fixes to the raw next-token distribution.
**Cyclic Shifts** — For a question with K options, the list is presented in K different rotations during inference. Every option appears in every position exactly once. The resulting logit distributions are then combined using a geometric mean computed in log space. If position bias operates additively in logit space, this procedure cancels it out completely.
For a yes/no question, only 2 rotations are needed. For a score question, just 1. The cost scales linearly with K, the number of options.
**Prior Correction** — After collecting predictions on real inputs, AnyJev maintains a running mean of the predicted distributions. At inference time, it divides out this learned prior at a configurable strength (default 0.75). The correction begins after the model has seen at least 8 requests, ensuring enough data has accumulated to form a meaningful prior estimate.
In terms of compute, L0 requires K prefill operations per decision, but these are batched over a shared prefix. The researchers report approximately 0.25 seconds per decision when processing batches of 32 items on a single H100 GPU, with K set to 20.
### Level 1 (L1): Temperature Scaling for Confidence Calibration
For applications demanding even better-calibrated probabilities, L1 builds on top of L0 by adding temperature scaling. A small number of labeled examples (between 100 and 500) are used to fit an optimal temperature parameter, which is then saved as a lightweight JSON artifact alongside the model.
Temperature scaling reshapes the confidence distribution without changing the ranking of answers. This means L1 preserves the model’s correct decisions while making its expressed confidence scores more trustworthy — a critical property when downstream systems use probability thresholds to trigger automated actions.
—
## Benchmark Results
The researchers evaluated AnyJev on the BANKING77 dataset, a 20-way classification task with 300 test items, using the Qwen3-8B model. The results demonstrate substantial improvements across multiple metrics.
| Metric | Raw Logits | AnyJev L0 | AnyJev L1 |
|—|—|—|—|
| Labels Required | 0 | 0 | 100–500 |
| Flip Rate (Reversed Options) | 0.230 | 0.073 | 0.077 |
| Accuracy | 0.747 | 0.803 | 0.807 |
| Calibration Error (ECE) | 0.240 | 0.184 | 0.095 |
| Auto-Decidable at 5% Error | 7.7% | 46.3% | 52.0% |
Several additional findings deserve attention. L0 reduced order-dependent flips across all nine model-and-task combinations tested in the full ablation study. On a separate typed-decisions benchmark, the Qwen3-32B model with L1 achieved an expected calibration error of just 0.036, compared with 0.144 reported in prior work for a dedicated fine-tuned system. Notably, while the L1 calibration is excellent, the fine-tuned Laya model still leads on raw accuracy.
The full ablation covers models from the Qwen, OLMo, Granite, Phi, and Mistral families, demonstrating broad applicability across different architectures and scales. Nokia also reported promising results when applying AnyJev to an internal routing problem within their own infrastructure.
—
## Getting Started with AnyJev
Installing the library is straightforward. The base package plus the Hugging Face backend can be installed with a single pip command:
“`
pip install “anyjev[hf]”
“`
A typical usage pattern involves creating a `Decider` instance backed by a Hugging Face model, defining one or more typed questions, and calling `decide` with an input dictionary:
“`python
from anyjev import Decider, Question
from anyjev.backends.hf import HFBackend
d = Decider(HFBackend(“Qwen/Qwen3-8B”))
route = Question.choice(
“Which team should handle this?”,
[“billing”, “technical”, “sales”, “other”],
name=”route”
)
r = d.decide({“conversation”: […]}, [route])
print(r[“route”].distribution) # probabilities per option
“`
For high-throughput serving scenarios, AnyJev includes a `VLLMBackend` that connects to a running vLLM instance with prefix caching enabled. This allows efficient batched inference across many concurrent decision requests, making the library suitable for production deployment at scale.
The Apache 2.0 license means there are no restrictions on commercial use, modification, or redistribution, which lowers the barrier for adoption in enterprise settings.
—
## Frequently Asked Questions
### What is the difference between AnyJev and simply prompting an LLM to pick an option?
When you prompt an LLM to choose an option, the model generates text, and you parse that text to extract the answer. This approach introduces variability based on formatting, parsing errors, and the model’s token generation behavior. AnyJev bypasses all of this by reading the decision directly from the model’s next-token probability distribution over the predefined options, eliminating parsing entirely.
### Does AnyJev require fine-tuning the LLM?
No. AnyJev works with off-the-shelf, pre-trained models without any training or fine-tuning step. Both L0 and L1 operate purely at inference time. L1 does use a small set of labeled examples (100–500) to calibrate a temperature parameter, but this is not training in the traditional sense — it is a post-hoc scaling of existing logits.
### How does the cost of L0 compare to a standard inference call?
L0 requires K inference calls per decision for a choice question with K options, because each cyclic shift constitutes a separate forward pass. However, these calls share a common prefix and are batched together, making the overhead manageable. The researchers measured roughly 0.25 seconds per decision at a batch size of 32 on a single H100 GPU with K = 20.
### Can AnyJev be used with any open-source LLM?
AnyJev is designed to work with any LLM that exposes next-token logits over a vocabulary that includes the option labels. The benchmarks cover models from several major families including Qwen, OLMo, Granite, Phi, and Mistral. As long as the model can be loaded through Hugging Face transformers or vLLM, it should be compatible.
### What happens when the model is uncertain?
AnyJev returns probability distributions rather than hard labels. Users can set a confidence threshold and route low-confidence decisions to a human reviewer. On the BANKING77 benchmark, L1 achieved 52% auto-decidability at a 5% error rate, meaning over half of all items could be handled automatically while maintaining acceptable accuracy.
### Is there a way to handle new options that weren’t in the original set?
AnyJev requires all possible options to be defined in advance as part of the question specification. If new options emerge, the question definition must be updated and the cyclic shift and prior correction procedures will apply to the expanded set. There is no support for dynamically adding labels at inference time without updating the question configuration.
### How does the prior correction in L0 avoid hurting accuracy?
The prior correction divides out the model’s tendency to favor certain labels, but if one label genuinely dominates the traffic distribution, removing that prior can reduce accuracy. The researchers advise measuring the impact on your own specific task before deploying prior correction in production. The strength parameter (default 0.75) allows users to dial the correction up or down based on their tolerance for calibration versus raw accuracy.
—
## Conclusion
AnyJev represents a pragmatic and well-engineered approach to a problem that affects countless production systems: how to reliably extract a single decision from a language model without the overhead of fine-tuning or the fragility of text generation and parsing. By combining cyclic shift-based debiasing, running prior correction, and optional temperature calibration, the library achieves meaningful improvements in accuracy, calibration, and order-invariance compared to naive logit reading.
The decision to open-source the library under an Apache 2.0 license, publish it on PyPI, and support both Hugging Face and vLLM backends signals a commitment to accessibility and production readiness. Whether you are building a customer support routing system, a transaction classification pipeline, or any other application that demands consistent, calibrated decisions from an LLM, AnyJev offers a compelling foundation that requires no training data and minimal setup.
As with any tool that reads directly from model logits, practitioners should validate performance on their specific task and data distribution, particularly when enabling prior correction or deploying in safety-critical contexts. But for the broad class of fixed-option decision problems, AnyJev provides a robust, zero-training alternative that merits serious consideration.
Thank you for reading
