The research addresses a growing requirement in search and recommendation: returning a coherent, complementary set of results rather than a single top match. For example, a user querying "camping gear" expects a slate that covers complementary items (tent, sleeping bag, stove, headlamp), not many near-duplicate four-person tents.
Why off-the-shelf LLMs fall short for set retrieval
Using a generic LLM at inference to brainstorm complementary sub-queries faces two main issues:
- Paraphrastic collapse: without database-aware optimization, zero-shot LLMs often produce redundant, near-synonymous sub-queries (e.g., "bohemian festival fashion" vs. "bohemian festival clothes"), yielding a homogeneous and unhelpful result slate.
- Autoregressive latency bottlenecks: decomposing a complex query into complementary facets typically requires extensive chain-of-thought reasoning tokens produced sequentially. This autoregressive process creates large test-time computation and a latency floor that conflicts with production sub-second response requirements.
The Retrieve-for-Train approach (ICML 2026)
In the ICML 2026 paper "Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion", the authors propose a reward-to-data compilation framework called Retrieve-for-Train. Rather than incurring heavy online computation for every query, the method runs an offline reinforcement learning (RL) training phase once to discover reward-aligned fan-outs, compiles those behaviors into supervision data, and distills them into a compact diffusion retriever that produces full target sets in a single non-autoregressive pass.
The pipeline has three stages:
- Fan-out language model training: RL trains a fan-out language model to emit property-aligned sub-queries scored by a set-level property-check reward, evaluating the group as a whole.
- Supervision synthesis: the frozen fan-out model synthesizes (query → target-set) pairs offline for supervised learning without human labels.
- Diffusive retriever training: a 53.9M-parameter diffusion model learns to map a query embedding directly to a complete set of target embeddings in one non-autoregressive pass.
Defining set-level quality with a composite reward
A central insight is that quality for slate retrieval is inherently set-level and non-decomposable, so supervision must encode set-level objectives. The authors fine-tuned 4B open-source models (Gemma3-4B and Qwen3-4B) with reinforcement learning using a strict mathematical composite reward composed of:
- Groundedness: penalizes distance to the database manifold so generated sub-queries correspond to real, retrievable items.
- Diversity: measured via the Vendi Score over the entire sub-query set to encourage semantic breadth.
- Alignment: anchors candidates to the original broad prompt to prevent semantic drift.
These three terms act as mutual counter-anchors: optimizing only groundedness produces degenerate strings that map numerically to database coordinates; optimizing only alignment leads to parafrastic collapse. The Vendi Score closes those shortcut paths and forces the policy into a balanced region where valid, grounded and semantically distinct sub-queries must be found to achieve high reward.
Training details: Soft-GRPO and generation setup
The fan-out models were optimized using group relative policy optimization (GRPO) with soft proximal policy optimization (soft PPO) regularization, referred to as Soft-GRPO. During experiments, each main query was expanded into exactly 10 sub-queries by the 4B fan-out models.
Experimental setup
The framework was evaluated across two set-valued retrieval regimes:
- Open-ended abstract retrieval: no unique ground truth; quality is judged solely by set-level metrics (diversity, alignment, groundedness).
- Weakly supervised compositional retrieval: queries have weak reference target sets representing one plausible realization.
Multimodal embedding backbones spanned two domains: a large-scale fashion dataset of user-curated outfits evaluated with a CLIP-based retriever, and a proprietary industrial music playlist dataset evaluated with MuLan. Fan-out generation was driven by Gemma3-4B and Qwen3-4B.
Results
-
Retrieval quality: Retrieve-for-Train outperformed single-query search, zero-shot expansion, and a Best-of-N baseline across both tasks. Zero-shot baselines tended to produce redundant paraphrases, while Retrieve-for-Train generated diverse, database-grounded and semantically distinct sub-queries.
-
Inference speed: directly running the RL-tuned language model retained autoregressive latency. Distilling the learned behavior into the 53.9M-parameter diffusion model removed that bottleneck: because the diffusion model generates all target directions simultaneously in a single non-autoregressive pass in embedding space, it produced a 12–20× speedup compared to autoregressive approaches. At scale, autoregressive fan-out latency can grow linearly to nearly 50 seconds with large context batches; Retrieve-for-Train-Diffusion operates from sub-second to a few seconds, making it suitable for production use at much lower compute cost.
-
Ablation insight: without a diversity term during reward optimization, the fan-out model collapses to generating degenerate, vector-exploiting strings. Introducing the Vendi Score prevents this exploitation and stabilizes training toward expert-like search behavior.
Conclusion
The paper demonstrates that offline RL can act as a one-time objective transducer: heavy compute for reward-driven behavior discovery is performed during training, and the resulting behaviors are compiled into cheap, fast inference via a diffusion prior. This decoupling enables production retrieval systems to optimize higher-order set-level properties (diversity, alignment, groundedness) efficiently, yielding a scalable, data-efficient pipeline for set retrieval in specialized or multimodal domains where human-labeled, property-aligned pairs are scarce or expensive to collect. The full methodology and experimental details are provided in the ICML 2026 paper "Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion."
