The Challenge of Reasoning in Large Language Models

Recent progress in large language models (LLMs) has heavily relied on reinforcement learning with verifiable rewards (RLVR). This method rewards models solely on the accuracy of their final answer, encouraging iterative problem-solving. However, RLVR faces a significant hurdle: computational cost. Each attempt, or “rollout,” to solve a problem is resource-intensive, limiting the number of trials a model can undertake. This becomes particularly problematic with complex tasks where finding the correct solution is rare within a reasonable budget.

The all-or-nothing nature of RLVR also creates a learning bottleneck. A model might execute several correct steps before a single error derails the entire process, resulting in a negative reward and no learning from the partially correct work. Alternatively, supervised fine-tuning (SFT), while capable of imparting reasoning skills, often leads to overfitting – the model memorizes training examples rather than generalizing to new scenarios. Furthermore, high-quality, human-generated training data for SFT is both scarce and expensive.

This gap in effective training methods, particularly for smaller, open-source models, has spurred the development of SRL.

How Supervised Reinforcement Learning Works

SRL bridges the gap between outcome-based reinforcement learning and imitation learning by representing problem-solving as a “sequential decision-making process.” Instead of focusing exclusively on the final result or rigidly mimicking expert demonstrations, SRL trains models to replicate a sequence of key actions that underpin expert reasoning. This allows for the development of an internal reasoning style while still learning from established problem-solving strategies.

In practice, SRL breaks down expert solutions into a series of concrete, intermediate actions. For example, in a mathematical problem, an action might be an algebraic manipulation; in software engineering, it could be a specific code command. A powerful “teacher” model generates these solution trajectories, which are then used to train a smaller, more efficient model.

“SRL sits in the middle: It captures the structured flexibility of real-world problem solving, where there are multiple valid strategies but also clear notions of what ‘good reasoning’ looks like at each step,” explains I-Hung Hsu, a research scientist at Google and co-author of the paper. “This makes SRL suitable for domains like data science automation or probably supply chain optimization — tasks that reward sound intermediate reasoning rather than mere final answers.”

During training, the model generates an “inner monologue” – its internal reasoning process – before executing an action. SRL then provides a reward based on the similarity between the model’s predicted action and the expert’s action. This step-by-step feedback system, offering dense and granular guidance, allows the model to learn and improve even with imperfect overall solutions, effectively addressing the sparse reward problem inherent in RLVR.

SRL in Action: Demonstrating Superior Performance

The researchers’ experiments demonstrate that SRL significantly outperforms existing methods in both challenging mathematical reasoning and agentic software engineering tasks. Furthermore, SRL encourages more flexible and sophisticated reasoning patterns, such as interleaved planning and self-verification, leading to improved solution quality without simply increasing output length.

Importantly, SRL-trained models maintain efficiency. According to Hsu, “The gains come from better reasoning quality and structure, not from verbosity. In terms of efficiency, SRL-trained models are roughly on par with the base model in token usage… while SRL isn’t designed to reduce inference cost, it achieves stronger reasoning performance without increasing it.”

In mathematical tests, the team fine-tuned Qwen2.5-7B-Instruct on 1,000 difficult math problems, achieving a 3.0% average performance boost over models trained with SFT and RLVR (using the GRPO algorithm, common in models like DeepSeek-R1).

Extending SRL to agentic software engineering, the team trained Qwen2.5-Coder-7B-Instruct on 5,000 expert trajectories of agents interacting with a coding environment. SRL achieved a 14.8% task resolve rate, a 74% relative improvement over a SFT-based baseline (SWE-Gym-7B), demonstrating its ability to train more competent AI agents for complex programming tasks.

The Future of AI Reasoning: A Combined Approach

The most promising results emerged from combining SRL with RLVR: using SRL for foundational reasoning training, followed by RLVR for refinement. This curriculum learning strategy yielded a 3.7% average performance increase in experiments.

Could this represent a new blueprint for building specialized AI? Hsu believes SRL provides a strong foundation. “In a sense, SRL provides a curriculum — teaching models to think and act step by step — before we refine those behaviors with outcome-based reinforcement learning. This SRL-first approach not only stabilizes the later RL stage but also makes reasoning more interpretable and generalizable, which is critical for high-stakes applications.”

While scaling this pipeline presents challenges, particularly the cost and complexity of end-to-end RLVR for agentic tasks, Hsu remains optimistic. He envisions future advancements in automating the generation and filtering of expert trajectories, leveraging powerful teacher models and self-improving student models to bootstrap new data. What role will automated data generation play in the next generation of AI reasoning systems? And how will these advancements impact industries reliant on complex problem-solving?

Frequently Asked Questions About Supervised Reinforcement Learning