Rohan Surana

We extend GRPO beyond verifiable settings via reference-relative contrastive comparisons: stratified conditional rollouts build positive/negative reference sets, a metric projection head scores candidates against these anchors, and the resulting alignment scores serve as contrastive advantages during policy optimization — achieving competitive results without task verifiers.

We introduce SOPHIA, which treats reasoning traces as sequences of latent states and uses steering vectors to detect when LLMs fall into repetitive reasoning loops and redirect them toward productive paths at inference time, improving both accuracy and token efficiency.

We cast multi-negative DPO under the Plackett-Luce model as a D-optimal design problem and develop a greedy, theoretically grounded selection strategy for informative negatives. This improves alignment efficiency and performance.

We extend Direct Preference Optimization to ranking, allowing LLMs to learn from sparse, in-context listwise feedback and directly optimize differentiable surrogates of ranking metrics. This connects preference optimization with practical IR settings like conversational recommendation and generative retrieval.

We extend multi-negative DPO to the multimodal setting, using a CLIP+SAE-based sampler and importance sampling under a Plackett–Luce objective to select semantically diverse visual negatives. This substantially improves multimodal alignment and reduces hallucinations compared to single-negative multimodal DPO baselines.

We address GRPO's dependence on dense step-wise rewards by learning to extract dense preference signals from sparse outcome supervision. WS-GRPO trains a preference model on trajectory-level outcomes, then leverages it to provide step-wise weakly-supervised rewards combined with terminal rewards during group-relative policy optimization. This enables effective reasoning model training without expensive step-by-step annotations.

We unify candidate generation and ranking for LLM-based retrieval into a single policy-optimized process. F-GRPO factorizes the objective with separate advantages for the generation and ranking phases, addressing the combinatorial output space and delayed list-level utility feedback. On sequential recommendation and QA, it outperforms prior GRPO variants and decoupled baselines.

We propose a reinforcement learning framework that optimizes reusable natural-language skills for agentic LLMs. Instead of updating the task model, a lightweight skill generator is trained via a bi-level group-relative policy optimization objective combining intra- and inter-generation advantages, enabling iterative skill refinement that improves multi-step reasoning across open- and closed-source models.

We provide the first comprehensive survey of Reinforcement Learning from Verifiable Rewards (RLVR), systematizing methods that train language models using verifier feedback. We introduce taxonomies organizing approaches by verification type, reward computation, and policy learning, establishing unified terminology for mathematical reasoning, code generation, and instruction following.

We survey rollout design for RL post-training of reasoning LLMs, organizing pipelines with a Generate-Filter-Control-Replay (GFCR) taxonomy: proposing trajectories and topologies, constructing signals with verifiers and judges, allocating compute for continuation and early exit, and replaying artifacts for curricula without weight updates. We synthesize methods from RLVR and tree rollouts to adaptive budgets and self-improvement, ground them in math, code, multimodal, and agent benchmarks, and map rollout pathologies to GFCR stages and mitigation levers.