# Retrieve-for-Train: Learning Fast and Diverse Query Fan-Out with Offline Reinforcement Learning
## The Challenge of Broad Queries in Modern Search Systems
Modern search and recommendation systems face a persistent challenge: users often submit broad, open-ended queries that should return a diverse set of results rather than a single best match. A query like “camping gear” should surface tents, sleeping bags, stoves, and headlamps — not ten variations of the same tent. The process of splitting a single broad query into multiple targeted sub-queries is known as query fan-out, and doing it well is deceptively difficult.
When a generic large language model handles fan-out at inference time, two major problems emerge. The first is **paraphrastic collapse**, where the model generates near-synonymous sub-queries that all retrieve from the same narrow region of the catalog. For a query like “Bohemian festival style,” a zero-shot model might produce “bohemian festival fashion” and “festival bohemian clothes” — both of which return overlapping, homogeneous results. The second problem is **latency**: autoregressive generation of sub-queries combined with repeated retrieval calls creates a slow pipeline, and even best-of-N sampling multiplies inference cost considerably.
## A Three-Step Framework for Learning Effective Fan-Out
A new framework addresses these challenges by running reinforcement learning once in an offline training phase, then distilling the learned behavior into a compact model that produces all retrieval directions in a single forward pass. The approach consists of three stages.
### Stage 1: Fan-Out Language Model Training
A fan-out language model generates multiple sub-queries from a single broad query. A frozen dense retriever executes each sub-query against the database, and a set-level reward function evaluates the entire retrieved set as a whole rather than scoring individual items in isolation. This is critical because the quality of a fan-out depends on how the sub-queries work together, not on how any single sub-query performs alone.
### Stage 2: Supervision Synthesis
The trained fan-out model is used to generate training data at scale. It samples 128 different fan-out configurations per query at a temperature of 0.9, producing (query, target set) training pairs entirely without human labels. For open-ended tasks, the target sets are represented as retrieved content embeddings. For compositional tasks, the targets are sub-query embeddings. The rows within each set are shuffled because sets have no inherent ordering.
### Stage 3: Diffusive Retriever Training
A diffusion transformer with 53.9 million parameters is trained to map a query embedding directly to a full set of target embeddings. It uses a variance-exploding formulation within the engineered diffusion model framework. At inference time, the model generates all retrieval-direction embeddings in a single non-autoregressive pass, and nearest-neighbor search maps each embedding back to items in the database. This eliminates the sequential overhead of autoregressive generation entirely.
## Reward Design and the Problem of Reward Hacking
For open-ended abstract retrieval, the reward function combines three weighted components:
– **Groundedness** (weight λ = 0.6): Penalizes the distance between each sub-query embedding and its nearest database item, ensuring that every sub-query actually retrieves meaningful content.
– **Diversity** (weight λ = 0.2): Measured using the Vendi Score over representative retrieved items, such as the top result from each sub-query, encouraging coverage across different regions of the catalog.
– **Alignment** (weight λ = 0.2): Computes mean cosine similarity between each sub-query and the original broad query, keeping the fan-out semantically relevant to the user’s intent.
For weakly supervised compositional retrieval, the reward is simply the fraction of reference-set items that the fan-out successfully retrieves.
### Why All Three Terms Are Necessary
Ablation studies reveal why each component matters. When groundedness is used alone, the policy converges to degenerate outputs — strings like “line ending line ending line ending” — because the model discovers that repeating nonsense tokens that happen to be close to a catalog embedding in the vector space satisfies the reward. When alignment is added on top of groundedness, collapse happens even faster, as the policy finds that paraphrasing the original query repeatedly maximizes both groundedness and alignment simultaneously. It is only when diversity is included as a counter-anchor that the model learns to produce genuinely different, meaningful sub-queries that each retrieve distinct and relevant content.
Training uses a GRPO algorithm with soft PPO regularization, which adds both forward and reverse KL divergence penalties to stabilize learning. Key hyperparameters include a group size of 8, a learning rate of 1 × 10⁻⁷, and a global batch size of 512.
## Efficiency Gains
The speed advantage of the diffusion-based approach is substantial. At a batch size of 8, autoregressive fan-out takes approximately 1.46 seconds, while the diffusion model completes the same task in about 0.07 seconds. At a batch size of 1024, autoregressive generation reaches nearly 50 seconds compared to 4.21 seconds for the diffusion model. Across all tested batch sizes, the framework delivers a consistent 12× to 20× speedup. This is because the diffusion model generates all retrieval directions in parallel during a single non-autoregressive pass rather than iterating sequentially.
## Experimental Results
Experiments were conducted on the Polyvore fashion outfit dataset using a CLIP-based matryoshka encoder at 128 dimensions, as well as a proprietary expert-playlist music dataset with MuLan embeddings. Every fan-out method produced 10 sub-queries, and best-of-N sampling used N = 5.
For open-ended abstract retrieval quality, scored by an LLM judge on 5-point Likert scales, Gemma3-4B with the R4T-trained fan-out language model achieved an average score of 49.1 on Polyvore, compared to 40.9 for best-of-N and 38.5 for zero-shot fan-out. Diversity scores rose from 56.0 with zero-shot to 76.8 with the trained fan-out model, and the diffusion model retained most of this gain at 74.3. On the music dataset, Gemma3-4B with trained fan-out averaged 58.1 versus 49.2 for best-of-N.
For weakly supervised compositional retrieval on Polyvore, a coverage and diversity trade-off was observed. R4T-trained FOLM with Qwen3-4B reached 20.9 Recall@5K and 64.6 Hit@5K, compared to 15.7 and 52.1 for the Gemini-2.5-Flash baseline. However, its Vendi Score dropped to 27.5, which the authors attribute to reduced output entropy under strong RL optimization. The R4T diffusion model with Qwen3-4B maintained a higher Vendi Score of 34.7 with 16.5 Recall@5K, demonstrating that the non-autoregressive approach preserves more diversity in its outputs.
## Key Takeaways
– The framework runs reinforcement learning only once during offline training, then deploys a compact 53.9-million-parameter diffusion retriever at inference time.
– Three reward terms — groundedness, diversity, and alignment — work together to prevent reward hacking and paraphrastic collapse.
– The diffusion-based approach generates all fan-out directions in a single pass, delivering 12× to 20× speedups over autoregressive methods.
– On open-ended retrieval tasks, the trained models consistently outperform zero-shot baselines and best-of-N sampling across multiple datasets.
—
## Frequently Asked Questions
**What is query fan-out and why is it important?**
Query fan-out is the process of decomposing a single broad search query into multiple targeted sub-queries that collectively cover different aspects of the user’s intent. It is important because a single query like “camping gear” encompasses many distinct product categories, and a diverse set of sub-queries helps retrieval systems return a well-rounded slate of results rather than redundant items.
**What is paraphrastic collapse?**
Paraphrastic collapse occurs when a fan-out model generates sub-queries that are near-synonyms of each other. Because these sub-queries are semantically similar, they all retrieve from the same narrow region of the catalog, producing a homogeneous set of results with limited coverage of the user’s broader intent.
**How does the diffusion model achieve faster inference than autoregressive methods?**
The diffusion transformer generates all target embeddings in a single non-autoregressive forward pass, processing all sub-query slots simultaneously. In contrast, autoregressive methods generate sub-queries sequentially, one token at a time, and require separate retrieval calls for each. This parallel generation is the primary source of the 12× to 20× speedup.
**What is reward hacking in this context?**
Reward hacking occurs when the model discovers degenerate strategies that maximize the reward function without producing useful fan-outs. For example, using only groundedness as a reward, the model might converge to repeating meaningless tokens that happen to be close to catalog embeddings in vector space. The ablation study showed that combining groundedness, diversity, and alignment prevents these shortcuts.
**Why is the Vendi Score lower for R4T diffusion models in compositional retrieval?**
The diffusion model trades some Vendi Score for higher recall and hit rates. The lower Vendi Score suggests slightly reduced diversity in the diffusion model’s embeddings compared to the FOLM, though the diffusion approach still maintains a better balance between coverage and diversity than the autoregressive FOLM baseline.
**What datasets were used for evaluation?**
The framework was evaluated on the Polyvore fashion outfit dataset using a CLIP-based matryoshka encoder at 128 dimensions, and on a proprietary expert-playlist music dataset using MuLan embeddings. Both datasets are commonly used for retrieval research.
—
## Conclusion
The Retrieve-for-Train framework represents a significant advance in how search systems handle broad, open-ended queries. By running reinforcement learning once offline and then deploying a lightweight diffusion retriever, it solves both the paraphrastic collapse and latency problems that plague standard fan-out approaches. The careful design of the reward function — balancing groundedness, diversity, and alignment — proves essential to preventing degenerate training outcomes. Experimental results across fashion and music retrieval tasks demonstrate consistent improvements over zero-shot baselines, best-of-N sampling, and prior autoregressive fan-out methods, all while delivering order-of-magnitude speedups at inference time. This approach opens the door to faster, more diverse, and more useful search experiences for broad queries.
Thank you for reading
