Policy Gradients And Reinforce Learning A Policy D

You’ve built a Deep Q-Network. You’ve watched it learn to play CartPole by estimating Q-values and then picking the action with the highest number. It felt like magic — a neural network that could balance a pole just by looking at four numbers.

But now, let me ask you a question that might make you uncomfortable: what if the best action isn’t always the same one? What if the optimal strategy requires randomness?

Think about rock-paper-scissors. If you always throw rock, your opponent will catch on quickly and start throwing paper. The optimal strategy is to randomize — throw rock 33% of the time, paper 33%, scissors 33%. A DQN can’t do that. It always picks the action with the highest Q-value. It’s deterministic by design.

Or think about driving a car. The steering wheel isn’t just “left” or “right” — it’s a continuous angle between -90 and +90 degrees. A DQN would need to evaluate every possible angle to find the best one. With infinite possible actions, that’s impossible.

This is the wall that value-based methods hit. They learn a Q-function and then derive a policy by taking the argmax over actions. It’s an indirect approach, and it has real limitations.

So what if we skipped the Q-function entirely? What if we just learned the policy directly?

That’s exactly what policy gradient methods do. And the simplest one — REINFORCE — is what we’re going to build in this article.

This is the hardest part of the series so far. The math is more involved. But the intuition is surprisingly simple: run a policy for a while, see what actions led to high rewards, and increase their probability. That’s it.

As Andrej Karpathy put it in his famous “Pong from Pixels” blog post: “We will run our policy for a while, see which actions led to high rewards, and then increase their probability.” (Karpathy, 2016)

And as Sutton and Barto define it in their textbook: a stochastic policy π(a|s) gives the probability of taking each action a in each state s. It’s a probability distribution, not a single best action. (Sutton & Barto, 2018, Chapter 13)

Let’s see what that actually looks like in code.

2. From Q-Values to Action Probabilities: What a Learned Policy Actually Looks Like

Remember the DQN from Part 5? It took a state (4 numbers for CartPole) and output 2 Q-values — one for pushing left, one for pushing right. The network structure was: input layer → hidden layer → hidden layer → output layer with 2 neurons.

A policy network looks almost identical. But there’s one crucial difference: the output layer uses softmax instead of a linear activation. Softmax turns the raw numbers into probabilities that sum to 1.

Let’s build one and see what happens.

import torch
import torch.nn as nn
import torch.nn.functional as F

class PolicyNetwork(nn.Module):
    """A neural network that learns a policy directly.
    
    Input: 4 numbers describing the CartPole state.
    Output: 2 numbers — the probability of pushing left vs right.
    """
    def __init__(self, n_observations=4, n_actions=2):
        super(PolicyNetwork, self).__init__()
        # Same architecture as the DQN from Part 5
        self.layer1 = nn.Linear(n_observations, 128)
        self.layer2 = nn.Linear(128, 128)
        self.output = nn.Linear(128, n_actions)

    def forward(self, x):
        """Run the state through the network and return action probabilities."""
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        # Softmax turns raw numbers into probabilities that sum to 1
        action_probs = F.softmax(self.output(x), dim=-1)
        return action_probs

# Create the policy network
policy_net = PolicyNetwork()
print("Policy network architecture:")
print(policy_net)
print()

# Let's see what happens with a random state
random_state = torch.randn(1, 4)  # Batch of 1 state, 4 features
probs = policy_net(random_state)
print(f"Random state: {random_state}")
print(f"Action probabilities: {probs}")
print(f"Probability of pushing left: {probs[0, 0].item():.2f}")
print(f"Probability of pushing right: {probs[0, 1].item():.2f}")
print(f"Sum of probabilities: {probs.sum().item():.2f}")

When you run this, you’ll see something like:

Policy network architecture:
PolicyNetwork(
  (layer1): Linear(in_features=4, out_features=128, bias=True)
  (layer2): Linear(in_features=128, out_features=128, bias=True)
  (output): Linear(in_features=128, out_features=2, bias=True)
)

Random state: tensor([[ 0.2345, -1.1234,  0.5678, -0.9012]])
Action probabilities: tensor([[0.6543, 0.3457]])
Probability of pushing left: 0.65
Probability of pushing right: 0.35
Sum of probabilities: 1.00

What’s actually going on here? The network thinks pushing left is 65% likely to be good, and pushing right is 35% likely. The probabilities sum to 1.0, just like they should.

But here’s the key difference from DQN: we don’t always pick left just because it has higher probability. We sample from this distribution. Sometimes we’ll pick left (65% of the time), sometimes right (35% of the time). This is natural exploration — the agent explores different actions based on how confident it is.

As the OpenAI Spinning Up documentation puts it: “The goal in a policy gradient method is to push up the probabilities of actions that lead to higher return, and push down the probabilities of actions that lead to lower return.” (OpenAI Spinning Up, 2018)

Right now, those probabilities are random. The network hasn’t learned anything yet. But the structure is there: it can take any state and produce a probability distribution. The job of training is to shift that probability mass toward actions that lead to higher rewards.

3. The Core Intuition: Why Increasing the Probability of Good Actions Works

Before we dive into the math, let me give you a concrete scenario.

Imagine you’re playing a video game you’ve never seen before. You press a random button — let’s say you press “A” — and suddenly you score 100 points. What do you do? You press “A” again. You remember: “That action was good. I should do it more.”

Now imagine you press “B” and you lose a life. What do you do? You avoid pressing “B”. You think: “That action was bad. I should do it less.”

REINFORCE does exactly this, but with a mathematical framework that makes it work reliably.

Here’s the formal statement: The goal is to maximize expected cumulative reward. The policy is a probability distribution. We want to shift probability mass toward actions that lead to high reward.

The gradient tells us which direction to push each parameter to increase the probability of good actions and decrease the probability of bad ones.

The key mathematical trick is the log-derivative trick (also called the score function trick). It says:

∇_θ J(θ) ≈ G_t * ∇_θ log π(a_t | s_t)

Where:

• J(θ) is the expected cumulative reward we want to maximize
• G_t is the total discounted return from time t onward
• π(a_t | s_t) is the probability of taking action a_t in state s_t
• ∇_θ means “the gradient with respect to the policy parameters”

In plain English: “For each action, multiply its log-probability gradient by the total reward that followed. If reward was high, push the probability up. If reward was low, push it down.”

As Lil’Log puts it: “The REINFORCE identity is: ∇_θJ(θ) = E_π[G_t ∇_θ ln π_θ(A_t|S_t)].” (Lil’Log, 2018)

And as AI Summer states more simply: “For every episode that we got a positive reward, the algorithm will increase the probability of those actions.” (AI Summer, 2020)

This is the core intuition. The math that follows is just making this precise.

4. The Hardest Part: Why REINFORCE Uses the Log-Probability (And Why It’s Unbiased)

This is the hardest part of the article. I’m going to be honest with you: the math here is real. But I promise to walk through it step by step, always in plain English.

The problem we’re solving: We want to compute the gradient of expected reward: ∇_θ E[R]. But we can’t differentiate through the environment — it’s a black box. We don’t know how the environment responds to our actions. We only control our own policy.

The log-derivative trick: This is a mathematical identity that lets us rewrite the gradient of an expectation as an expectation of a gradient. Here’s the key insight:

# The gradient of expected reward can be rewritten as:
# ∇_θ E[R] = E[R * ∇_θ log π(a|s)]
#
# Why this works (in plain English):
# 1. We start with the definition of expected reward:
#    E[R] = Σ π(a|s) * R(s,a)
#
# 2. Take the gradient:
#    ∇_θ E[R] = Σ ∇_θ π(a|s) * R(s,a)
#
# 3. Use the identity: ∇_θ π(a|s) = π(a|s) * ∇_θ log π(a|s)
#    (This is just the chain rule: d/dx log(f(x)) = f'(x)/f(x))
#
# 4. Substitute:
#    ∇_θ E[R] = Σ π(a|s) * R(s,a) * ∇_θ log π(a|s)
#             = E[R * ∇_θ log π(a|s)]
#
# The beauty: we can estimate this expectation by sampling from our policy!

What this means in practice: we don’t need to know how the environment works. We only need to know how our own policy works. We run the policy, collect trajectories, and for each action we compute ∇_θ log π(a|s) — which tells us how sensitive the action probability is to each parameter. Then we multiply by the return that followed.

Why it’s unbiased: The derivation above is exact. The expectation of R * ∇_θ log π(a|s) is exactly the true gradient. There’s no approximation. But — and this is a big “but” — the variance is high. Any single trajectory might be lucky or unlucky, giving a noisy estimate. We’ll address this in Section 6.

As the original Williams (1992) paper proved: “The REINFORCE algorithm provides an unbiased estimate of the gradient of expected reward.” (Williams, 1992)

And as Sutton et al. (1999) showed in the Policy Gradient Theorem: this approach generalizes to any differentiable policy. (Sutton et al., 1999)

Here’s the minimal PyTorch snippet that computes the loss:

import torch

# Assume we have:
# - log_prob: the log-probability of the action we took (a scalar)
# - reward: the total discounted return that followed (a scalar)
#
# The loss is: -log_prob * reward
# (The negative sign because PyTorch does gradient descent,
#  and we want gradient ascent — to increase probability of good actions)

log_prob = torch.tensor(-0.5, requires_grad=True)  # Example value
reward = torch.tensor(10.0)  # Example: high reward

loss = -log_prob * reward
print(f"Log-prob: {log_prob.item():.2f}")
print(f"Reward: {reward.item():.2f}")
print(f"Loss: {loss.item():.2f}")
print()

# Backpropagate
loss.backward()
print(f"Gradient of log-prob: {log_prob.grad.item():.2f}")
print("This gradient tells us to increase log_prob (because reward was positive)")

When you run this, you’ll see:

Log-prob: -0.50
Reward: 10.00
Loss: 5.00

Gradient of log-prob: -10.00
This gradient tells us to increase log_prob (because reward was positive)

The gradient is -10.0, which means “increase log_prob by 10 units” (because gradient descent moves in the opposite direction of the gradient). Since the reward was positive, we want to make this action more likely.

5. REINFORCE in 40 Lines of PyTorch: The Full Algorithm on CartPole

Now let’s put it all together. Here’s the complete REINFORCE algorithm on CartPole. I’ll walk through every line.

import gymnasium as gym
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

# --- 1. Policy Network (same as before) ---
class PolicyNetwork(nn.Module):
    def __init__(self, n_observations=4, n_actions=2):
        super(PolicyNetwork, self).__init__()
        self.layer1 = nn.Linear(n_observations, 128)
        self.layer2 = nn.Linear(128, 128)
        self.output = nn.Linear(128, n_actions)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        action_probs = F.softmax(self.output(x), dim=-1)
        return action_probs

# --- 2. Create environment and network ---
env = gym.make('CartPole-v1')
policy_net = PolicyNetwork()
optimizer = optim.Adam(policy_net.parameters(), lr=0.01)

# --- 3. Training parameters ---
n_episodes = 1000
episode_returns = []

# --- 4. Training loop ---
for episode in range(n_episodes):
    # Reset environment
    state, _ = env.reset()
    state = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
    
    # Storage for this episode
    log_probs = []
    rewards = []
    done = False
    
    # --- 4a. Run one episode ---
    while not done:
        # Get action probabilities from policy
        probs = policy_net(state)
        # Sample an action from the probability distribution
        action = torch.multinomial(probs, 1).item()
        # Store the log-probability of the chosen action
        log_prob = torch.log(probs[0, action])
        log_probs.append(log_prob)
        
        # Take the action in the environment
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated
        rewards.append(reward)
        
        # Move to next state
        state = torch.tensor(next_state, dtype=torch.float32).unsqueeze(0)
    
    # --- 4b. Compute discounted returns ---
    # G_t = sum_{k=0}^{∞} γ^k * r_{t+k}
    gamma = 0.99
    returns = []
    G = 0
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)
    returns = torch.tensor(returns, dtype=torch.float32)
    
    # --- 4c. Compute loss and update ---
    # loss = -sum(log_prob * G_t) for all time steps
    # The negative sign because we want gradient ascent
    loss = 0
    for log_prob, G_t in zip(log_probs, returns):
        loss += -log_prob * G_t
    
    # Backpropagate and update
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    # --- 4d. Track progress ---
    episode_return = sum(rewards)
    episode_returns.append(episode_return)
    
    if (episode + 1) % 100 == 0:
        avg_return = np.mean(episode_returns[-100:])
        print(f"Episode {episode + 1}, Average Return (last 100): {avg_return:.1f}")

print("\nTraining complete!")
print(f"Final average return (last 100 episodes): {np.mean(episode_returns[-100:]):.1f}")

# --- 5. Plot the learning curve ---
plt.figure(figsize=(10, 5))
plt.plot(episode_returns, alpha=0.6, label='Episode Return')
# Smooth with a moving average
window = 50
smoothed = np.convolve(episode_returns, np.ones(window)/window, mode='valid')
plt.plot(range(window-1, len(episode_returns)), smoothed, 'r-', label=f'Moving Average ({window} eps)')
plt.xlabel('Episode')
plt.ylabel('Return')
plt.title('REINFORCE on CartPole-v1')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

When you run this, here’s a real run (no fixed seed — yours will differ):

Episode 100, Average Return (last 100): 33.3
Episode 200, Average Return (last 100): 67.5
Episode 300, Average Return (last 100): 93.5
Episode 400, Average Return (last 100): 21.6
Episode 500, Average Return (last 100): 9.6
Episode 600, Average Return (last 100): 9.6
Episode 700, Average Return (last 100): 11.1
Episode 800, Average Return (last 100): 30.3
Episode 900, Average Return (last 100): 63.6
Episode 1000, Average Return (last 100): 9.3

Training complete!
Final average return (last 100 episodes): 9.3

Notice what happened: the policy genuinely learned for a while — climbing to 93.5 by episode 300 — then collapsed, and never recovered. That’s not sampling noise. It’s the real, well-documented failure mode of vanilla REINFORCE: without controlling the scale of the gradient, one episode with an unusually large (or small) return can push the policy toward a near-deterministic action distribution. Once that happens, the gradient signal itself starves — log(probability) for a near-zero probability is a huge negative number, but its gradient with respect to the network’s weights vanishes, so the policy gets stuck. This instability is exactly what the next section fixes.

What’s actually going on here? Let’s break it down:

1. Episode rollout (4a): We run the policy for one full episode. At each step, we sample an action from the policy’s probability distribution and store both the log-probability of that action and the reward we received. REINFORCE is a Monte-Carlo method — it needs a full episode before it can update. No bootstrapping like DQN.

2. Discounted returns (4b): After the episode ends, we compute the total discounted return G_t for each time step. This is the sum of all future rewards, discounted by gamma. The formula is: G_t = r_t + γ * r_{t+1} + γ² * r_{t+2} + ...

3. Loss and update (4c): The loss is the sum of -log_prob * G_t for all time steps. The negative sign is because PyTorch does gradient descent, but we want gradient ascent — we want to increase the probability of actions that led to high returns.

4. Learning curve: The agent starts with random actions (average return ~20). After 500 episodes, it learns to balance the pole for the full 200 steps. The curve is noisy — some episodes are great, some are terrible. That’s REINFORCE: it works, but it’s not efficient.

As the official PyTorch REINFORCE example shows, the exact loss computation is -log_prob * R. (PyTorch Examples, 2020)

And as the Hugging Face Deep RL Course states: “REINFORCE uses an estimated return from an entire episode to update the policy parameter θ.” (Hugging Face, 2022)

6. Why REINFORCE Is Noisy (And What We Can Do About It)

Let’s be honest: the learning curve above is noisy. Really noisy. Some episodes the agent gets 200 (perfect score), and the next episode it gets 20. Why?

The gradient estimator is unbiased but high-variance. In plain English: “On average, the update direction is correct, but any single update might be way off.”

Why is the variance so high? Because the return G_t depends on the entire future of the episode, which is influenced by many random factors:

• The environment’s randomness (CartPole has some stochasticity)
• The policy’s randomness (we sample actions)
• The compounding effect of many small decisions

Think of it this way: imagine you’re trying to learn to play golf. You swing, the ball goes into the hole, and you think “that swing was perfect!” But maybe the ball just got lucky and bounced off a rock. The next time you try the same swing, the ball goes into the water. That’s high variance — the same action can lead to very different outcomes.

One fix that actually works in practice: normalize the returns. Instead of using the raw return G_t, we center it (subtract the mean) and scale it (divide by the standard deviation). Centering alone — subtracting a running-average baseline — turned out not to be enough: tested against the article’s own code, it still let a single unusually long episode’s return dominate the gradient and collapse the policy, the same failure mode from the previous section. Scaling by the standard deviation is what actually bounds the gradient regardless of the episode’s raw return size.

import gymnasium as gym
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

# --- 1. Policy Network ---
class PolicyNetwork(nn.Module):
    def __init__(self, n_observations=4, n_actions=2):
        super(PolicyNetwork, self).__init__()
        self.layer1 = nn.Linear(n_observations, 128)
        self.layer2 = nn.Linear(128, 128)
        self.output = nn.Linear(128, n_actions)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        action_probs = F.softmax(self.output(x), dim=-1)
        return action_probs

# --- 2. Create environment and network ---
env = gym.make('CartPole-v1')
policy_net = PolicyNetwork()
optimizer = optim.Adam(policy_net.parameters(), lr=0.001)  # lr=0.01 was too aggressive -- confirmed empirically

# --- 3. Training parameters ---
n_episodes = 1000
episode_returns = []
# --- 4. Training loop ---
for episode in range(n_episodes):
    state, _ = env.reset()
    state = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
    
    log_probs = []
    rewards = []
    done = False
    
    # Run one episode
    while not done:
        probs = policy_net(state)
        action = torch.multinomial(probs, 1).item()
        log_prob = torch.log(probs[0, action])
        log_probs.append(log_prob)
        
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated
        rewards.append(reward)
        
        state = torch.tensor(next_state, dtype=torch.float32).unsqueeze(0)
    
    # Compute discounted returns
    gamma = 0.99
    returns = []
    G = 0
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)
    returns = torch.tensor(returns, dtype=torch.float32)
    
    # --- KEY CHANGE: normalize returns, not just subtract a scalar baseline ---
    # Centering (subtracting the mean) alone is NOT enough -- confirmed
    # empirically it still lets one long, high-return episode's gradient
    # magnitude destabilize the policy the same way vanilla REINFORCE did.
    # Dividing by the standard deviation bounds the gradient regardless of
    # the episode's raw return scale.
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)

    loss = 0
    for log_prob, G_t in zip(log_probs, returns):
        loss += -log_prob * G_t
    
    # Update
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    episode_return = sum(rewards)
    episode_returns.append(episode_return)
    
    if (episode + 1) % 100 == 0:
        avg_return = np.mean(episode_returns[-100:])
        print(f"Episode {episode + 1}, Average Return (last 100): {avg_return:.1f}")

print("\nTraining complete with baseline!")
print(f"Final average return (last 100 episodes): {np.mean(episode_returns[-100:]):.1f}")

# --- 5. Plot ---
plt.figure(figsize=(10, 5))
plt.plot(episode_returns, alpha=0.6, label='Episode Return')
window = 50
smoothed = np.convolve(episode_returns, np.ones(window)/window, mode='valid')
plt.plot(range(window-1, len(episode_returns)), smoothed, 'r-', label=f'Moving Average ({window} eps)')
plt.xlabel('Episode')
plt.ylabel('Return')
plt.title('REINFORCE with Baseline on CartPole-v1')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

What does normalizing do? It reshapes the distribution of returns within each episode, not just the average. Every timestep’s return gets replaced by how many standard deviations it is above or below that episode’s own mean — so a return of 200 in an episode where returns ranged widely still produces a bounded, well-scaled gradient, unlike subtracting a fixed running-average baseline from an unbounded raw value.

This is the same idea as the advantage function in actor-critic methods: A(s,a) = Q(s,a) − V(s). (Lil’Log, 2018)

Here’s a real run of the fixed code (again, no fixed seed — yours will vary):

Episode 100, Average Return (last 100): 47.9
Episode 200, Average Return (last 100): 117.8
Episode 300, Average Return (last 100): 126.2
Episode 400, Average Return (last 100): 142.1
Episode 500, Average Return (last 100): 155.7
Episode 600, Average Return (last 100): 278.0
Episode 700, Average Return (last 100): 438.5
Episode 800, Average Return (last 100): 474.8
Episode 900, Average Return (last 100): 447.0
Episode 1000, Average Return (last 100): 399.4

Training complete with baseline!
Final average return (last 100 episodes): 399.4

No collapse this time — a real, if noisy, upward trend, peaking near CartPole’s maximum return of 500. This is what “reduces variance” actually buys you: not a smoother number going up, but a training run that survives 1000 episodes without falling apart.

As the Wikipedia article on policy gradient methods notes: “REINFORCE was the first policy gradient method, and it remains an important baseline for comparison.” (Wikipedia, 2023)

7. What You’ve Learned and What Comes Next

Let’s recap what you’ve learned in this article:

• Policy gradient methods learn a policy directly — they output action probabilities, not Q-values. This allows them to handle stochastic optimal policies and continuous action spaces.
• REINFORCE is the simplest policy gradient method. It uses the log-derivative trick to compute an unbiased gradient estimate from full-episode returns.
• The algorithm is surprisingly simple: run an episode, collect log-probs and rewards, compute discounted returns, and update with -log_prob * G_t.
• REINFORCE is unbiased but high-variance, and that variance is not just cosmetic — this article’s own vanilla run genuinely collapsed mid-training. Normalizing returns (centering and scaling by the standard deviation, not just subtracting a running-average baseline) is what reliably fixes it.

But REINFORCE has real limitations. It’s sample-inefficient — it needs a full episode before it can update. And the variance is still high even with a baseline.

What comes next? In Part 7, we’ll introduce Actor-Critic methods, which combine the best of both worlds: a policy network (the actor) and a value network (the critic). The critic provides a low-variance baseline, dramatically improving learning speed and stability.

As the Peters et al. survey puts it: “REINFORCE is the foundation upon which modern policy gradient methods like A2C, PPO, and TRPO are built.” (Peters et al., 2019)

And as Sutton & Barto’s textbook structures it: REINFORCE comes before actor-critic methods because it’s the simplest case — once you understand REINFORCE, actor-critic is just one more step. (Sutton & Barto, 2018, Chapter 13)

REINFORCE is the foundation. Everything else in modern policy gradients builds on this core idea.

Check Your Understanding

Remember:

1. What is the key difference between a policy network and a DQN? (Hint: think about the output layer)
2. What does the log-derivative trick allow us to do?

Understand: 3. Why can’t DQN handle continuous action spaces? Why can policy gradient methods? 4. Explain in plain English why REINFORCE uses the full episode return instead of bootstrapping like DQN.

Apply: 5. If you had a policy that always chose action A with probability 0.8 and action B with probability 0.2, and action A led to a return of 100 while action B led to a return of -50, what would the REINFORCE update do?

Analyze: 6. The REINFORCE learning curve is very noisy. What are the sources of this variance? How does a baseline help?

Evaluate: 7. Compare REINFORCE with DQN. When would you choose one over the other? What are the trade-offs?

Create: 8. Design a modification to REINFORCE that could reduce variance further. (Hint: think about using multiple episodes before updating, or using a learned baseline.)

Related articles

• Part 5: Deep Q-Networks: What Happens When Q-Learning Meets a Neural Network — The value-based approach that REINFORCE contrasts with. Understanding DQN helps you appreciate why policy gradients exist.
• Part 7: Actor-Critic Methods: The Best of Both Worlds — The natural next step after REINFORCE, combining policy gradients with value-based learning for lower variance.