Proximal Policy Optimization Ppo Intuitively Why I

You just built an actor-critic agent. It works. The critic tells the actor which actions were genuinely good, and the actor learns faster than plain REINFORCE ever could. But there’s a catch — a big one. And it’s the reason you’re about to learn about PPO.

Let’s get concrete.

1. Opening: The Vanilla Policy Gradient’s Fatal Flaw

Imagine you’re hiking on a narrow mountain trail. A thick fog limits your visibility to just a few meters. You can only see the path directly in front of you. Taking a small, careful step is safe. But what if someone told you to take a giant leap forward?

You’d probably fall off the cliff.

That’s exactly what happens with vanilla policy-gradient methods like REINFORCE and basic actor-critic. They take the gradient — the direction that looks best right now — and multiply it by a learning rate. If that learning rate is even slightly too large, the policy can collapse.

Here’s what that looks like in practice. Let’s say your agent is learning to play a game with two actions: left and right. After some training, it’s figured out that pushing right is usually better. The current policy chooses right with 60% probability. That’s a good, confident-but-not-too-confident strategy.

Now, a single update comes along. The advantage for ‘right’ was strongly positive. The gradient says: “Make right even more likely!” With a slightly-too-large learning rate, the update overshoots. The new policy assigns 98% probability to right. The agent has become overconfident. Next time it encounters a situation where left was actually correct, it’ll almost certainly fail.

But wait — it gets worse. With a truly bad step size, the probability of the best action can drop to near zero in a single update. The agent just unlearned its best strategy.

Let’s see this happen in code.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

# A tiny two-action environment simulation
# Action 0 is 'left', Action 1 is 'right'
# The true best action is 'right' (action 1)

class SimplePolicy(nn.Module):
    """A minimal policy network for a two-action problem."""
    def __init__(self):
        super().__init__()
        # Just one layer: state -> action probabilities
        self.fc = nn.Linear(4, 2)

    def forward(self, x):
        return F.softmax(self.fc(x), dim=-1)

# Create the policy
policy = SimplePolicy()
optimizer = optim.SGD(policy.parameters(), lr=100.0)  # Deliberately huge learning rate

# A random state (4 numbers, like CartPole)
state = torch.randn(1, 4)

# Before the update: let's see the probabilities
with torch.no_grad():
    probs_before = policy(state)
print(f"Before update:")
print(f"  Probability of action 0 (left):  {probs_before[0, 0].item():.3f}")
print(f"  Probability of action 1 (right): {probs_before[0, 1].item():.3f}")
print()

# Now simulate a vanilla policy gradient update
# Assume action 1 (right) was taken and had a positive advantage of +2.0
# The REINFORCE loss is: -log_prob(action) * advantage
action_taken = torch.tensor([1])  # Action 1 was chosen
log_probs = torch.log(policy(state))
log_prob_action = log_probs[0, action_taken]

# The loss: negative because we want to maximize the probability
loss = -log_prob_action * 2.0

# Take a gradient step
optimizer.zero_grad()
loss.backward()
optimizer.step()

# After the update: check the new probabilities
with torch.no_grad():
    probs_after = policy(state)
print(f"After one update with learning rate 100.0:")
print(f"  Probability of action 0 (left):  {probs_after[0, 0].item():.3f}")
print(f"  Probability of action 1 (right): {probs_after[0, 1].item():.3f}")
print()

# Interpret the disaster
print("Interpretation:")
print(f"  Before: the agent chose the best action (right) with {probs_before[0, 1].item()*100:.0f}% probability.")
print(f"  After:  that probability dropped to {probs_after[0, 1].item()*100:.0f}%.")
print(f"  The agent just unlearned its best strategy in a single update!")

When you run this, you’ll see something dramatic. Before the update, the policy might assign 60% probability to the best action. After one update with a huge learning rate, that probability could drop to 2%. The agent has effectively collapsed.

As the OpenAI blog post on PPO puts it: “With vanilla policy gradient methods, a step that is too large can cause a catastrophic drop in performance.” (OpenAI Blog, 2017)

This is the problem PPO was designed to solve.

2. Intuition: The ‘Trust Region’ — Why You Can’t Just Take a Big Step

So how do we prevent this collapse? The answer is a concept called a trust region.

Imagine you’re in that foggy landscape again. You can only see a few meters around you. The map in your hand shows the general area, but the details are fuzzy. You want to move toward your goal, but you know that any estimate of the terrain beyond the visible zone is unreliable.

A trust region is simply the area around your current position where you trust your approximations to be accurate. You’re allowed to move anywhere within that zone. But if you try to step outside it, you risk falling off a cliff.

In reinforcement learning terms, the “terrain” is the policy’s performance. The “visible zone” is the region of policy space where our gradient estimate is accurate. If we take too big a step in parameter space, the new policy might be completely different from the old one — and our estimate of how good that new policy will be is completely wrong.

This is the hardest part of policy optimization. The gradient tells you which direction looks good right now, but it doesn’t tell you how far you can safely go.

The first solution to this problem was TRPO — Trust Region Policy Optimization (Schulman et al., 2015). TRPO uses a complex mathematical constraint based on KL divergence to enforce the trust region. It works, but it’s hard to implement and computationally expensive.

Think of TRPO as having a GPS that constantly recalculates the safe zone using heavy calculus. It works, but it’s slow and finicky. Every update requires solving a constrained optimization problem.

PPO was invented to get the same safety guarantee with a much simpler, more intuitive trick. As the PPO paper (Schulman et al., 2017) states: “PPO is much simpler to implement, more general, and has better sample complexity” than TRPO.

So what’s the trick? It’s called the clipped surrogate objective.

3. The Core Idea: The Clipped Surrogate Objective — What It Actually Does

Here’s the key insight behind PPO: it doesn’t forbid large updates. It just makes them not worth doing.

Let’s build this up step by step.

The Probability Ratio

First, PPO defines a ratio:

r(θ) = π_new(a|s) / π_old(a|s)

In plain English: “How much more (or less) likely is the new policy to take this action compared to the old one?”

If r(θ) = 1.0, the new policy is equally likely to take this action. If r(θ) = 2.0, the new policy is twice as likely. If r(θ) = 0.5, it’s half as likely.

The Unclipped Objective

The vanilla policy gradient objective multiplies this ratio by the advantage:

L_unclipped = r(θ) * Â(s, a)

If an action had a positive advantage (it was a good move), the update wants to make it more likely. If the ratio becomes 100x, the objective becomes 100 * advantage — a huge number that drives a huge gradient step.

This is the problem. The ratio can explode.

The Clipping Trick

PPO caps the ratio to a safe range: [1-ε, 1+ε]. The standard value is ε = 0.2, meaning the ratio can only change by 20% in either direction.

r_clipped = clamp(r(θ), 1-ε, 1+ε)

“You can only make an action up to 20% more likely in one update, no matter how good it was.”

The Min Operation

PPO takes the minimum of the clipped and unclipped objectives:

L_PPO = min(r(θ) * Â, r_clipped * Â)

Why the min? PPO is pessimistic. When the ratio goes outside the safe zone, it assumes the clipped version is correct. This prevents the update from being fooled by an unreliable ratio.

Let’s see this work through a concrete example.

import torch

def clipped_surrogate_objective(log_probs_old, log_probs_new, advantages, epsilon=0.2):
    """
    Compute the PPO clipped surrogate objective for a batch of data.
    
    Args:
        log_probs_old: log probabilities under the old policy [batch_size]
        log_probs_new: log probabilities under the new policy [batch_size]
        advantages: advantage estimates [batch_size]
        epsilon: clipping parameter (default 0.2)
    
    Returns:
        loss: negative PPO objective (to minimize)
        diagnostics: dict with ratio, unclipped objective, clipped objective
    """
    # Compute the probability ratio: r(θ) = exp(log π_new - log π_old)
    ratio = torch.exp(log_probs_new - log_probs_old)
    
    # Unclipped objective
    unclipped_objective = ratio * advantages
    
    # Clipped ratio: clamp to [1-ε, 1+ε]
    clipped_ratio = torch.clamp(ratio, 1.0 - epsilon, 1.0 + epsilon)
    clipped_objective = clipped_ratio * advantages
    
    # Take the minimum (pessimistic bound)
    ppo_objective = torch.min(unclipped_objective, clipped_objective)
    
    # We return negative because we minimize loss
    loss = -ppo_objective.mean()
    
    diagnostics = {
        'ratio': ratio,
        'unclipped_objective': unclipped_objective,
        'clipped_objective': clipped_objective,
        'ppo_objective': ppo_objective,
        'clipping_active': (ratio < 1.0 - epsilon) | (ratio > 1.0 + epsilon)
    }
    
    return loss, diagnostics

# Let's test with a concrete example
# Two actions, one batch of data
batch_size = 4

# Old policy: action probabilities [0.5, 0.5] for both actions
log_probs_old = torch.tensor([-0.6931, -0.6931, -0.6931, -0.6931])  # log(0.5)

# New policy: varies from slightly different to very different
# Case 1: ratio = 1.0 (no change) — clipping not needed
# Case 2: ratio = 1.5 (50% more likely) — clipping kicks in
# Case 3: ratio = 0.5 (50% less likely) — clipping kicks in
# Case 4: ratio = 1.1 (10% more likely) — within safe zone
log_probs_new = torch.tensor([-0.6931, -0.4055, -1.3863, -0.6061])
# Corresponding ratios: 1.0, 1.5, 0.5, 1.1

# Advantages: mix of positive and negative
advantages = torch.tensor([1.0, 2.0, -1.0, 0.5])

loss, diag = clipped_surrogate_objective(log_probs_old, log_probs_new, advantages, epsilon=0.2)

print("PPO Clipped Surrogate Objective — Detailed Example")
print("=" * 60)
print(f"{'Case':<10} {'Ratio':<10} {'Adv':<10} {'Unclipped':<12} {'Clipped':<12} {'PPO':<12} {'Clipping?':<10}")
print("-" * 60)

for i in range(batch_size):
    clipping = "Yes" if diag['clipping_active'][i].item() else "No"
    print(f"{i+1:<10} {diag['ratio'][i].item():<10.3f} {advantages[i].item():<10.1f} "
          f"{diag['unclipped_objective'][i].item():<12.3f} {diag['clipped_objective'][i].item():<12.3f} "
          f"{diag['ppo_objective'][i].item():<12.3f} {clipping:<10}")

print()
print("Interpretation:")
print("  Case 1: No change in policy — ratio = 1.0, no clipping needed.")
print("  Case 2: Ratio = 1.5 (50% more likely) — clipping kicks in! The PPO objective")
print("          uses the clipped value (1.2 * 2.0 = 2.4) instead of the unclipped (3.0).")
print("  Case 3: Ratio = 0.5 (50% less likely) — clipping kicks in again.")
print("          The PPO objective uses 0.8 * (-1.0) = -0.8 instead of 0.5 * (-1.0) = -0.5.")
print("  Case 4: Ratio = 1.1 (10% more likely) — within the [0.8, 1.2] safe zone.")
print("          No clipping needed; both objectives are the same.")

What’s actually going on here? The clipping creates a “safe zone” around the current policy. If the update tries to make an action more than 20% more likely, the objective stops rewarding further increases. The gradient becomes zero for that action. The update simply stops pushing in that direction.

As the Hugging Face Deep RL Course explains: “Without clipping, the ratio can explode and cause catastrophic updates. Clipping ensures the update stays within a safe region.” (Hugging Face, 2023)

And as Lilian Weng notes in her blog post: “The clipped objective is a first-order approximation of the trust region constraint used in TRPO, making it much simpler to implement.” (Weng, 2018)

4. But Wait — Where Does the Advantage Come From? (Generalized Advantage Estimation)

You’ve seen the PPO objective uses Â(s, a), the advantage. If you only know Q-learning, you might think: “Isn’t that just Q(s, a) - V(s)?”

Yes, that’s the definition. But there’s a subtlety: how you estimate the advantage massively affects bias and variance.

The Bias-Variance Tradeoff

Let’s define advantage in plain English: “How much better was this action compared to the average action in this state?”

To estimate this, we need to know the total future reward from taking this action. But we don’t have that — we only have the rewards we actually observed.

Here’s the tradeoff:

• TD(0) advantage: Use just the immediate reward plus the critic’s estimate of the next state’s value. This is low-variance (it doesn’t depend on many random future rewards) but high-bias (it relies on the critic’s potentially inaccurate estimate).

• Monte Carlo advantage: Wait until the end of the episode and use the total actual return. This is low-bias (it uses real rewards) but high-variance (it depends on all the random actions taken during the rest of the episode).

GAE: The Best of Both Worlds

Generalized Advantage Estimation (Schulman et al., 2015/2016) gives you a dial between these two extremes. It takes a weighted average of all possible n-step returns, controlled by a single parameter λ.

In plain English: “GAE is the sum of all future TD errors, each discounted by γ and decayed by λ.”

• λ = 0: You only trust the immediate TD error (high bias, low variance).
• λ = 1: You wait for the full Monte Carlo return (low bias, high variance).
• λ = 0.95: A common middle ground that works well in practice.

Let’s see this in code.

import torch

def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
    """
    Compute Generalized Advantage Estimation.
    
    Args:
        rewards: tensor of rewards [horizon]
        values: tensor of state values from critic [horizon + 1] (includes bootstrap)
        dones: tensor of done flags [horizon]
        gamma: discount factor
        lam: GAE lambda parameter
    
    Returns:
        advantages: tensor of advantage estimates [horizon]
        returns: tensor of target returns for critic [horizon]
    """
    horizon = len(rewards)
    advantages = torch.zeros(horizon)
    gae = 0.0
    
    # Compute advantages backwards (from last step to first)
    for t in reversed(range(horizon)):
        # TD error: δ_t = r_t + γ * V(s_{t+1}) * (1 - done) - V(s_t)
        delta = rewards[t] + gamma * values[t + 1] * (1 - dones[t]) - values[t]
        # GAE: accumulate with decay λ
        gae = delta + gamma * lam * (1 - dones[t]) * gae
        advantages[t] = gae
    
    # Target returns for the critic: V(s_t) + A(s_t)
    returns = advantages + values[:-1]
    
    return advantages, returns

# Let's test with a small example
# A 4-step trajectory
horizon = 4

# Rewards observed at each step
rewards = torch.tensor([1.0, 0.0, 2.0, 0.5])

# Values predicted by the critic (includes bootstrap value for state after last action)
values = torch.tensor([0.5, 0.8, 1.2, 1.5, 1.8])

# Done flags (0 = not done, 1 = done)
dones = torch.tensor([0, 0, 0, 0])

# Compute advantages with different λ values
print("GAE Example: 4-Step Trajectory")
print("=" * 60)
print(f"Step | Reward | V(s) | V(s+1) | TD Error")
print("-" * 40)

for t in range(horizon):
    delta = rewards[t] + 0.99 * values[t + 1] * (1 - dones[t]) - values[t]
    print(f"{t+1:<5} | {rewards[t]:<6.1f} | {values[t]:<5.1f} | {values[t+1]:<6.1f} | {delta:<8.3f}")

print()

# Compare λ = 0 (TD(0)) and λ = 1 (Monte Carlo) and λ = 0.95 (typical)
for lam_val, label in [(0.0, "λ = 0 (TD(0))"), (1.0, "λ = 1 (MC)"), (0.95, "λ = 0.95 (Typical)")]:
    adv, ret = compute_gae(rewards, values, dones, gamma=0.99, lam=lam_val)
    print(f"\n{label}:")
    print(f"  Advantages: {[f'{a:.3f}' for a in adv.tolist()]}")
    print(f"  Returns:    {[f'{r:.3f}' for r in ret.tolist()]}")

print()
print("Interpretation:")
print("  With λ = 0: advantages are just the TD errors. High bias, low variance.")
print("  With λ = 1: advantages are the full Monte Carlo returns minus V(s). Low bias, high variance.")
print("  With λ = 0.95: a weighted average that balances bias and variance.")
print("  Most PPO implementations use λ = 0.95 and γ = 0.99.")

What’s actually going on here? The λ parameter controls how much you trust the critic’s value estimates versus the actual rewards. When λ is small, you rely heavily on the critic (which might be inaccurate early in training). When λ is large, you wait for more actual rewards (which reduces bias but increases variance because of randomness in the environment).

As noted in the “37 Implementation Details of PPO” paper (Huang et al., 2022), the standard settings are γ = 0.99 and λ = 0.95 for most environments.

5. The Full PPO Training Loop: Putting It All Together

Now let’s see how all the pieces fit together in a complete PPO training loop. We’ll use CartPole — the same environment from Part 5 — so you can compare directly with DQN.

Here’s the architecture:

1. Actor network: Takes state → outputs action probabilities.
2. Critic network: Takes state → outputs state value V(s).
3. Training loop: Collect a batch of experiences (a “rollout”), compute advantages using GAE, then do multiple epochs of minibatch SGD on the PPO objective.

The “multiple epochs” trick is key: PPO reuses each batch of data for several gradient steps (e.g., 4-10 epochs). This is what makes PPO sample-efficient compared to vanilla policy gradient, which uses each experience once.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import gymnasium as gym
import numpy as np

# --- Network Definitions ---

class ActorNetwork(nn.Module):
    """The actor: maps state to action probabilities."""
    def __init__(self, n_observations=4, n_actions=2):
        super().__init__()
        self.fc1 = nn.Linear(n_observations, 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, n_actions)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return F.softmax(self.fc3(x), dim=-1)

class CriticNetwork(nn.Module):
    """The critic: maps state to state value V(s)."""
    def __init__(self, n_observations=4):
        super().__init__()
        self.fc1 = nn.Linear(n_observations, 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

# --- PPO Agent ---

class PPOAgent:
    """A complete PPO agent with actor, critic, and training loop."""
    def __init__(self, n_observations=4, n_actions=2, lr=3e-4, gamma=0.99, lam=0.95, epsilon=0.2, epochs=4):
        self.actor = ActorNetwork(n_observations, n_actions)
        self.critic = CriticNetwork(n_observations)
        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=lr)
        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=lr)
        self.gamma = gamma
        self.lam = lam
        self.epsilon = epsilon
        self.epochs = epochs

    def get_action(self, state):
        """Sample an action from the policy."""
        state = torch.FloatTensor(state).unsqueeze(0)
        with torch.no_grad():
            probs = self.actor(state)
            dist = torch.distributions.Categorical(probs)
            action = dist.sample()
            log_prob = dist.log_prob(action)
            value = self.critic(state)
        return action.item(), log_prob.item(), value.item()

    def compute_gae(self, rewards, values, dones):
        """Compute GAE advantages and target returns."""
        horizon = len(rewards)
        advantages = torch.zeros(horizon)
        gae = 0.0
        
        for t in reversed(range(horizon)):
            delta = rewards[t] + self.gamma * values[t + 1] * (1 - dones[t]) - values[t]
            gae = delta + self.gamma * self.lam * (1 - dones[t]) * gae
            advantages[t] = gae
        
        returns = advantages + values[:-1]
        return advantages, returns

    def update(self, states, actions, old_log_probs, advantages, returns):
        """Perform multiple epochs of PPO updates on a batch of data."""
        # Normalize advantages (standard trick for stability)
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
        
        kl_divergences = []
        clipping_fractions = []
        
        for epoch in range(self.epochs):
            # Get new action probabilities and values
            probs = self.actor(states)
            dist = torch.distributions.Categorical(probs)
            new_log_probs = dist.log_prob(actions)
            values_pred = self.critic(states).squeeze()
            
            # Compute probability ratio
            ratio = torch.exp(new_log_probs - old_log_probs)
            
            # Clipped surrogate objective
            unclipped_objective = ratio * advantages
            clipped_ratio = torch.clamp(ratio, 1.0 - self.epsilon, 1.0 + self.epsilon)
            clipped_objective = clipped_ratio * advantages
            actor_loss = -torch.min(unclipped_objective, clipped_objective).mean()
            
            # Critic loss (MSE between predicted values and target returns)
            critic_loss = F.mse_loss(values_pred, returns)
            
            # Total loss
            total_loss = actor_loss + 0.5 * critic_loss
            
            # Update networks
            self.actor_optimizer.zero_grad()
            self.critic_optimizer.zero_grad()
            total_loss.backward()
            self.actor_optimizer.step()
            self.critic_optimizer.step()
            
            # Track diagnostics
            with torch.no_grad():
                kl = (old_log_probs - new_log_probs).mean().item()
                clip_frac = ((ratio < 1.0 - self.epsilon) | (ratio > 1.0 + self.epsilon)).float().mean().item()
                kl_divergences.append(kl)
                clipping_fractions.append(clip_frac)
        
        return kl_divergences, clipping_fractions

# --- Training Loop ---

env = gym.make('CartPole-v1')
agent = PPOAgent(n_observations=4, n_actions=2)

# Training parameters
horizon = 2048  # Steps to collect before each update
max_timesteps = 100000

# Storage for one rollout
states = []
actions = []
log_probs = []
rewards = []
dones = []
values = []

timestep = 0
episode_rewards = []
episode_reward = 0

state, _ = env.reset()

while timestep < max_timesteps:
    # Collect rollout
    for _ in range(horizon):
        if timestep >= max_timesteps:
            break
            
        action, log_prob, value = agent.get_action(state)
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated
        
        states.append(state)
        actions.append(action)
        log_probs.append(log_prob)
        rewards.append(reward)
        dones.append(done)
        values.append(value)
        
        episode_reward += reward
        timestep += 1
        
        if done:
            episode_rewards.append(episode_reward)
            episode_reward = 0
            state, _ = env.reset()
        else:
            state = next_state
    
    # Convert to tensors
    states_tensor = torch.FloatTensor(np.array(states))
    actions_tensor = torch.LongTensor(actions)
    old_log_probs_tensor = torch.FloatTensor(log_probs)
    rewards_tensor = torch.FloatTensor(rewards)
    dones_tensor = torch.FloatTensor(dones)
    
    # Get bootstrap value for the last state
    with torch.no_grad():
        if not dones[-1]:
            last_value = agent.critic(torch.FloatTensor(state).unsqueeze(0)).item()
        else:
            last_value = 0.0
    values_tensor = torch.FloatTensor(values + [last_value])
    
    # Compute advantages and returns
    advantages, returns = agent.compute_gae(rewards_tensor, values_tensor, dones_tensor)
    
    # Update policy
    kl_divs, clip_fracs = agent.update(states_tensor, actions_tensor, old_log_probs_tensor, advantages, returns)
    
    # Print diagnostics
    avg_reward = np.mean(episode_rewards[-10:]) if episode_rewards else 0
    print(f"Timestep {timestep:>6} | Avg Reward (last 10): {avg_reward:>5.1f} | "
          f"KL: {kl_divs[-1]:.4f} | Clip Frac: {clip_fracs[-1]:.3f}")
    
    # Clear rollout storage
    states = []
    actions = []
    log_probs = []
    rewards = []
    dones = []
    values = []

env.close()
print("\nTraining complete!")

What’s actually going on here? The agent collects 2048 steps of experience (a “rollout”), then uses that data to update the policy multiple times. The KL divergence tells us how much the policy changed during the update. If it’s small (say, < 0.01), the clipping is barely needed. If it grows larger (say, > 0.03), the clipping starts kicking in to prevent the policy from changing too much.

As the PPO paper (Schulman et al., 2017) explains: “PPO typically performs multiple epochs of minibatch SGD on each batch of data, which makes it much more sample-efficient than vanilla policy gradient.”

6. Why PPO Became the Default: Evidence from the Real World

So PPO is stable and sample-efficient. But why did it become the default reinforcement learning algorithm across the entire field?

Let’s look at the evidence.

OpenAI’s Own Statement

OpenAI’s blog post announcing PPO states: “PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use and good performance.” (OpenAI Blog, 2017)

This isn’t just marketing. When the leading AI research lab says an algorithm is their go-to choice, it carries weight.

OpenAI Five: Dota 2 at Massive Scale

In 2019, OpenAI trained a team of agents to play Dota 2 at a professional level. The project, called OpenAI Five, used PPO at massive scale: 1536 GPUs running for 10 months of continuous self-play (Berner et al., 2019).

This demonstrated that PPO scales to the largest reinforcement learning problems. It’s not just a toy algorithm for CartPole — it works on problems with massive state spaces and complex multi-agent dynamics.

RLHF: The Technology Behind ChatGPT

Here’s the connection that might surprise you: the chatbot you’re talking to was likely trained with PPO.

Reinforcement Learning from Human Feedback (RLHF) — the technique behind InstructGPT and ChatGPT — uses PPO in its third step. After training a reward model on human preferences, PPO fine-tunes the language model to maximize that reward (Ouyang et al., 2022).

As the InstructGPT paper states: “We use PPO to optimize the policy against the reward model.” (Ouyang et al., 2022)

An Honest Caveat

PPO isn’t magic. It has limitations. The PPO-CMA paper (2018) showed that PPO can prematurely shrink exploration variance in continuous control tasks. The policy becomes too conservative and stops exploring new strategies.

It’s the default, but it’s not the final answer. Newer methods like DPO (Direct Preference Optimization) are emerging, and some even integrate with PPO (the “DPO Meets PPO” paper, 2024).

But for now, if you’re starting a new reinforcement learning project, PPO is the safe bet. It’s simple, stable, and scales.

7. Recap: What You Learned

Let’s summarize the key concepts:

• Vanilla policy gradients are fragile: One bad step can collapse the policy. The gradient tells you direction, not distance.
• PPO’s clipped surrogate objective keeps updates safe: It caps how much any action’s probability can change in one update (typically 20%).
• Generalized Advantage Estimation (GAE) provides a tunable bias-variance tradeoff: The λ parameter lets you dial between trusting the critic (λ = 0) and waiting for actual rewards (λ = 1).
• PPO reuses data with multiple epochs of minibatch SGD: This makes it sample-efficient compared to methods that use each experience once.
• PPO is the default RL algorithm in industry: From Dota 2 to ChatGPT, it’s the go-to choice for stability and scalability.

8. Check Your Understanding

Test your understanding of PPO with these questions based on Bloom’s taxonomy:

Remember: What does the clipping parameter ε control in PPO?

Understand: Explain in your own words why clipping prevents catastrophic policy updates.

Apply: If you set ε = 0.0, what would happen to the PPO update?

Analyze: Compare the bias-variance tradeoff in GAE when λ = 0 vs. λ = 1. Which setting would you use for a task with very noisy rewards?

Evaluate: Why might PPO still fail on a continuous control task with highly variable action spaces? (Hint: think about exploration variance.)

Create: Sketch a modification to the clipped objective that could handle action spaces with very different scales (e.g., one action is “move 1 cm” and another is “move 1 km”).

9. Related Articles

• Part 5: Deep Q-Networks — The foundation of function approximation in RL, which PPO also uses (though with an actor-critic architecture).
• Part 4: Q-Learning From Scratch — The tabular method that motivated the need for function approximation.
• OpenAI Spinning Up’s PPO documentation — A more mathematical reference if you want the formal details.
• The 37 Implementation Details of PPO — A deep dive into the code-level tricks that make PPO work in practice.

10. Apply What You Learned

Task: Implement PPO from scratch on the CartPole environment, then experiment with the clipping parameter ε. Try ε = 0.0, 0.2, and 0.5. Report the final average reward and the KL divergence during training.

Deliverable: A single Python script that trains PPO for 100,000 timesteps and prints a table of (ε, final_reward, max_KL) for each run.

Self-check rubric:

• Does the ε = 0.0 run fail to learn (reward stays low)?
• Does ε = 0.2 learn stably?
• Does ε = 0.5 show signs of instability (reward spikes then crashes)?

If yes, your implementation is correct.