The Seductive Illusion of RL in LLMs: Why Industrial AI Is Quietly Returning to Supervised Foundations

Executive Summary: The recent shift by frontier AI laboratories—such as Zhipu AI in GLM-5.2 abandoning Group Relative Policy Optimization (GRPO) in favor of traditional Proximal Policy Optimization (PPO)—highlights a growing consensus in post-training research: Standard Reinforcement Learning (RL) is a notoriously inefficient “data sponge”. While cost-saving hacks like GRPO democratized open-source RL by removing the Value Model, they sacrificed essential generalization capabilities and early-signal credit assignment. Modern, scalable LLM post-training is not pure RL; it is Supervised Learning (SL) operating under an RL facade.

1. The Data Sponge Metric: SL vs. RL

To evaluate an optimization paradigm, we must ask: How much transferable capability is extracted per unit of data? Let us define the Data Efficiency Index $\eta$ as the mutual information between the optimal parameter configuration $\theta^*$ and the dataset $\mathcal{D}$, normalized by dataset size:

Why Supervised Learning is a Superior Data Sponge

In Supervised Learning, every single sample provides a dense, un-biased, low-variance target. Given dataset $\mathcal{D} = {(x_i, y_i)}_{i=1}^N$, the loss gradient is:

SL exhibits three structural advantages:

1. Dense Signal: Every token generates an explicit error metric $H(y^*, \hat{y})$.
2. Unbiased Estimates: Directly estimates the true first-order gradient of the target conditional distribution.
3. Zero Variance from Rollouts: The environment state transition matrix is deterministic ($P = I$), allowing offline replay and batch shuffling.

2. The “Three Deadly Sins” of Reinforcement Learning

When attempting to scale textbook RL (e.g., REINFORCE) to Large Language Models, optimization hits a structural wall.

Sin 1: Information Sparsity & Credit Assignment

Textbook RL optimizes the expected trajectory return:

In reasoning models (e.g., DeepSeek-R1, OpenAI o1/o3), a 4,000-token Chain-of-Thought (CoT) trajectory $\tau$ might receive a single scalar reward $R(\tau) \in {0, 1}$ upon execution completion. A single terminal bit of information must guide the gradient updates of billions of individual parameters—an information-theoretic bottleneck.

Sin 2: Horizon-Dependent Variance Explosion

Under vanilla Policy Gradient (REINFORCE) without baselines or temporal discounting, the variance of the gradient estimator scales quadratically with sequence length $T$:

import torch
import numpy as np

def compute_reinforce_variance(T: int, batch_size: int = 32, trials: int = 200) -> float:
    """
    Simulates vanilla REINFORCE variance scaling over sequence length T.
    """
    grads = []
    for _ in range(trials):
        # Un-discounted, baseline-free trajectory log-probabilities
        log_p = torch.randn(batch_size, T) * 0.1
        R = torch.randn(batch_size)
        grad_estimate = (log_p.sum(dim=1) * R).mean().item()
        grads.append(grad_estimate)
    return float(np.var(grads))

# Empirical Variance Growth:
# T=  10  --> Var ≈ 3.2e-03
# T= 100  --> Var ≈ 3.1e-01  (100x increase)
# T=1000  --> Var ≈ 3.4e+01  (10,000x increase)

As reasoning traces extend into thousands of tokens, gradient variance destabilizes the policy unless controlled by heavy structural intervention.

Sin 3: Exploratory Overhead

Unlike offline SL datasets stored on NVMe arrays, RL requires active online generation (rollouts). Eric Jang’s bound estimates the required sample complexity $N_{\text{RL}}$ relative to $N_{\text{SL}}$:

Where $C_\pi$ is the policy action-space entropy coefficient and $C_\rho$ is the task distribution entropy coefficient. Matching SL’s signal-to-noise ratio in high-entropy language spaces demands astronomical sampling volume.

3. Demystifying GRPO vs. PPO: Why Industrial AI Is Walking Back

The release of DeepSeek-R1 popularized Group Relative Policy Optimization (GRPO). GRPO eliminates the Value Model $V_\phi(s)$ entirely, relying on intra-group baseline normalization:

The Unspoken Costs of Eliminating the Critic

GRPO achieved a ~50% reduction in training VRAM by removing the Critic network. However, this trade-off introduces two architectural compromises:

• Loss of Value-Function Generalization: A learned Value Model $V_\phi(s)$ acts as an abstract evaluator of latent states. It develops dense representation capabilities regarding why an intermediate reasoning step is promising before the full rollout finishes. GRPO’s statistical group mean lacks state-level generalization; it operates as a soft form of Rejection Sampling.
• Dense Intermediate Credit Assignment Failure: Without a token-level Critic, GRPO assigns uniform advantages $\hat{A}_i$ across all tokens in a sequence $o_i = (t_1, t_2, \dots, t_T)$. Correct intermediate tokens leading up to a late-stage error are penalized equally alongside the mistake.

This explains why frontier models (e.g., Zhipu’s GLM-5.2) are reverting to PPO or hybrid Actor-Critic frameworks: At industrial scale, compute is cheaper than lost policy capacity. Frontier labs prefer parameterizing a dedicated Critic to retain dense temporal credit assignment.

4. Modern Post-Training Is “Supervised-ified” RL

Examining the objectives of prevailing alignment algorithms reveals a clear trend: methods scale by incorporating Supervised Learning components.

Example: PPO Objective Decomposition

The standard LLM-PPO loss mixes supervised anchors with policy gradients:

Out of the primary objective terms, the Value Function update $\mathcal{L}^{\text{VF}}$ is standard regression (SL), and the KL-divergence constraint acts as an explicit regularizer tethering the policy to the supervised base checkpoint $\pi_{\text{ref}}$.

Example: DPO Eliminates Rollouts Entirely

Direct Preference Optimization (DPO) re-parameterizes the reward function via the policy itself:

Substituting this into the Bradley-Terry preference model yields a closed-form loss:

This reduces reinforcement learning to binary cross-entropy over static preference pairs—effectively transforming RL back into Supervised Learning.

5. The Suboptimal Extrapolation Hypothesis

A foundational assumption in classical control theory is that policy iteration requires trajectories sampled from an optimal target policy $\pi^*$. In LLM alignment, empirical evidence challenges this assumption: models can surpass the performance of the demonstrator when trained on suboptimal trajectories.

Suboptimal Policy π_sub  →  Inject Noise  →  Low-Quality Traces
                                                      ↓
Supervised Preference Loss  ←  Rank Pairs  ←  Reward Model R_φ
              ↓
Extrapolated Policy π*  (outperforms original demonstrator π_sub)

Mathematical Intuition of Trajectory Ranking (D-REX Framework)

Consider trajectories generated by a noisy, suboptimal policy $\pi_{\text{sub}}$. A reward model $R_\phi$ is trained using a pairwise ranking objective over ranked pairs $(\tau_i \succ \tau_j)$:

When $R_\phi$ is optimized over noisy, suboptimal data, the deep neural network does not merely memorize the ranking order. Due to inductive bias, the network fits a smooth low-dimensional manifold representing the abstract gradient direction of quality:

By following $\nabla_\theta \mathbb{E}[R_\phi(\tau)]$, policy optimization extrapolates beyond the bounds of the training distribution $\mathcal{D}_{\text{suboptimal}}$, generating outputs superior to those of the annotators or teacher models.

This explains two notable empirical observations:

• RLHF Outperforms Human Annotators: Human raters are rarely optimal reasoners, yet Bradley-Terry preference modeling enables models like GPT-4 to outperform human baselines.
• Suboptimal Teacher Distillation (OPD Anomalies): Distilling from a slightly weaker teacher model can yield a stronger student model than distilling from a flawless teacher, provided the weaker teacher generates richer directional variance along error boundaries.

6. Conclusion: The Evolutionary Trajectory of RL in LLMs

The trajectory of post-training research points to a clear structural trend:

Pure Control Theory RL  →  Resource-Optimized Hacks (GRPO)  →  Supervised-Anchored Hybrids (PPO/PRMs)
(High Variance, Unstable)   (Removes Critic, High Sampling Cost)   (Dense Token Credit, Value-Guided Search)

• Pure RL is unviable for long-context LLMs: Its high variance, signal sparsity, and sample inefficiency prevent direct scaling without supervised constraints.
• GRPO is a local optimization for compute-constrained settings: By sacrificing the Value Model to save memory, it relies on high sampling volume and loses state-level generalization. GLM-5.2’s return to PPO signals that frontier labs prioritize representation quality over VRAM savings.
• Inductive bias enables performance scaling from suboptimal data: Modern alignment works because neural networks extrapolate directional quality vectors from comparative preferences, rather than mimicking optimal demonstrations.

The future of RL in language modeling lies not in imitating classical control algorithms, but in leaning into the strengths of supervised deep learning architectures: dense feedback, value-guided search, and variance-controlled optimization.

References

• Sutskever, I. (2023). An Observation on Generalization. Simons Institute Talk.
• Sutton, R. (2019). The Bitter Lesson.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
• Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Introducing GRPO). arXiv:2402.03300.
• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
• Brown, D., et al. (2020). Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations (D-REX). CoRL.
• DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.