Deep Q Networks What Happens When Q Learning Meets

You’ve built a Q-table from scratch. You’ve watched it learn to navigate a frozen lake, filling in those 64 cells (16 states × 4 actions) one update at a time. It felt good, right? Like teaching a toddler to walk — slow, deliberate, but working.

Now imagine the toddler grows up and wants to play Go. The board is 19×19. There are roughly 10^170 possible board positions. That’s more than the number of atoms in the observable universe. A Q-table for Go would need more entries than there are atoms in existence. You can’t build that. You can’t store it. You can’t even imagine it.

So what do you do?

You stop trying to memorize every position. Instead, you teach the agent to recognize patterns. You give it a brain that can look at a board — or a game screen, or a sensor reading — and say, “I’ve never seen this exact situation before, but it looks a lot like something I have seen. So I’ll guess that moving left is probably good.”

That brain is a neural network. And when you combine it with Q-learning, you get a Deep Q-Network (DQN).

Let’s build one.

1. The Q-Table’s Final Boss: A Game Too Large

Remember the FrozenLake problem from Part 4? A 4×4 grid, 16 states, 4 actions. A Q-table with 64 entries. It worked beautifully.

Now imagine a game like Breakout from the original Atari. The screen is 210×160 pixels, with 128 possible colors per pixel. The number of possible “states” — unique screen images — is astronomically large. There’s no way to enumerate them all.

But here’s the thing: you don’t need to. A human player doesn’t memorize every possible arrangement of the ball and paddle. They learn patterns: “When the ball is moving this way and the paddle is here, I should move left.” They generalize from experience.

A neural network does the same thing. Instead of a lookup table with one row per state, it learns a function — a mathematical machine that takes any state as input and produces Q-values as output. Even for states it has never seen, it can make a reasonable guess, because it has learned the underlying patterns.

Think of it this way: the Q-table is a phone book. It works if you have every number. The DQN is a friend who knows how phone numbers work — they can guess your neighbor’s number even if they’ve never called it before.

2. From Lookup to Learner: The Deep Q-Network — What It Actually Is

A Deep Q-Network is exactly what it sounds like: a deep neural network that approximates the Q-function. You give it a state, and it outputs a Q-value for every possible action.

Let’s see one in code. We’ll build a small network for the CartPole environment — a classic control problem where a pole is balanced on a cart. The state has just 4 numbers (cart position, cart velocity, pole angle, pole angular velocity), and there are 2 actions (push left, push right).

import torch
import torch.nn as nn
import torch.nn.functional as F

class DQN(nn.Module):
    """A simple neural network that approximates the Q-function.
    
    Input: 4 numbers describing the state of the CartPole.
    Output: 2 numbers — the predicted Q-values for pushing left and right.
    """
    def __init__(self, n_observations=4, n_actions=2):
        super(DQN, self).__init__()
        # First hidden layer: 4 inputs → 128 neurons
        self.layer1 = nn.Linear(n_observations, 128)
        # Second hidden layer: 128 → 128
        self.layer2 = nn.Linear(128, 128)
        # Output layer: 128 → 2 (one Q-value per action)
        self.output = nn.Linear(128, n_actions)

    def forward(self, x):
        """Run the state through the network and return Q-values."""
        # First layer: linear transformation + ReLU activation
        x = F.relu(self.layer1(x))
        # Second layer: same thing
        x = F.relu(self.layer2(x))
        # Output layer: just the linear transformation, no activation
        # The raw numbers are our Q-value estimates
        return self.output(x)

# Create the network
network = DQN()
print("Network architecture:")
print(network)
print()

# Let's see what happens with a random state
random_state = torch.randn(1, 4)  # Batch of 1 state, 4 features
q_values = network(random_state)
print(f"Random state: {random_state}")
print(f"Predicted Q-values: {q_values}")
print(f"Left Q-value: {q_values[0, 0].item():.2f}")
print(f"Right Q-value: {q_values[0, 1].item():.2f}")

When you run this, you’ll see something like:

Network architecture:
DQN(
  (layer1): Linear(in_features=4, out_features=128, bias=True)
  (layer2): Linear(in_features=128, out_features=128, bias=True)
  (output): Linear(in_features=128, out_features=2, bias=True)
)

Random state: tensor([[ 0.2345, -1.1234,  0.5678, -0.9012]])
Predicted Q-values: tensor([[ 0.1234, -0.5678]])
Left Q-value: 0.12
Right Q-value: -0.57

What’s actually going on here? The network took the 4 numbers describing the state, pushed them through two hidden layers (each with 128 neurons and a ReLU activation), and finally produced 2 numbers — one for each action. The left Q-value (0.12) is higher than the right (-0.57), so the network thinks moving left is better in this state.

Right now, those numbers are random. The network hasn’t learned anything yet. But the structure is there: it can take any state and produce a guess. The job of training is to make those guesses better and better.

3. But Wait — Data From a Game Isn’t Independent. Why That Breaks a Neural Network

Here’s the catch: you can’t just feed the DQN experience in the order it happens. Neural networks have two quirks that make this fail.

Problem 1: Correlated data. Consecutive frames in a game are almost identical. If the network learns from frame 1, then frame 2, then frame 3, it starts to think the world is very simple — it “overfits” to the last few frames. Then when frame 100 comes along (which looks completely different), the network has forgotten everything it learned earlier. This is called catastrophic forgetting.

Problem 2: Moving target. Remember the Bellman update from Part 4? The target is r + γ * max Q(s', a'). But the network itself computes that max Q(s’, a’). So every time the network updates, the target changes. It’s like trying to hit a bullseye that moves every time you throw a dart.

DQN solves both problems with two brilliant innovations.

Solution 1: Experience replay. Instead of learning from each experience immediately, we store all experiences in a big buffer — a ReplayMemory. When it’s time to learn, we sample a random batch of experiences from the buffer. This breaks the correlation between consecutive frames because we’re mixing up old and new memories.

Think of it like studying for a test. You don’t just review the last page you read — you randomly flip through the whole textbook, mixing up chapters. That’s experience replay.

import random
from collections import deque, namedtuple

# A single experience: (state, action, reward, next_state, done)
Transition = namedtuple('Transition',
                        ('state', 'action', 'reward', 'next_state', 'done'))

class ReplayMemory:
    """A cyclic buffer that stores recent experiences.
    
    When the buffer is full, old experiences are overwritten.
    Sampling is random, which breaks the correlation between consecutive frames.
    """
    def __init__(self, capacity=100000):
        self.memory = deque([], maxlen=capacity)

    def push(self, *args):
        """Save a transition."""
        self.memory.append(Transition(*args))

    def sample(self, batch_size):
        """Randomly sample a batch of transitions for training."""
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

# Create a buffer that can hold 100,000 experiences
buffer = ReplayMemory(capacity=100000)
print(f"Empty buffer size: {len(buffer)}")

# Simulate storing some experiences
for i in range(10):
    # In real training, these would come from the environment
    state = torch.randn(4)
    action = torch.tensor([random.randint(0, 1)])
    reward = torch.tensor([random.random()])
    next_state = torch.randn(4)
    done = torch.tensor([random.random() < 0.1])
    buffer.push(state, action, reward, next_state, done)

print(f"Buffer size after 10 pushes: {len(buffer)}")

# Sample a batch of 4 experiences
batch = buffer.sample(4)
print(f"Sampled batch size: {len(batch)}")
print(f"First transition in batch: state shape = {batch[0].state.shape}")

A buffer of size 100,000 means the agent remembers its last 100,000 steps. Older memories get overwritten — the agent forgets, but it forgets slowly. And because we sample randomly, the network sees a mix of old and new experiences, which stabilizes learning.

Solution 2: Target network. We keep a second, frozen copy of the Q-network. This target network is used to compute the target Q-values in the Bellman update. The online network (the one that’s actually learning) updates every step. But the target network only gets updated every C steps — we copy the online network’s weights to it.

This keeps the target stationary long enough for the online network to converge toward it. It’s like having a stable target to aim at, even as your aim improves.

4. The Full Algorithm: Let’s See What Happens When We Press ‘Train’

Now let’s put it all together. Here’s the DQN algorithm in plain English:

1. Create two networks: the online network (learns) and the target network (provides stable targets). Start them with the same random weights.
2. Create a replay buffer.
3. For each episode: a. Reset the environment. b. For each step:
   • Use the online network to pick an action (with epsilon-greedy exploration).
   • Take the action, observe the reward and next state.
   • Store the experience in the buffer.
   • If the buffer has enough samples, sample a batch and run optimize_model().
4. In optimize_model():
   • Compute predicted Q-values for the sampled states using the online network.
   • Compute target Q-values using the target network: r + γ * max Q_target(s', a').
   • Compute the loss (Huber loss) between predicted and target.
   • Backpropagate to update the online network.
   • Every C steps, copy the online network weights to the target network.

This is the hardest part of DQN. If it feels confusing, that’s normal. Let’s walk through the code line by line.

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import gymnasium as gym
import random
from collections import deque, namedtuple
import math

# Hyperparameters — these control how the agent learns
BATCH_SIZE = 128          # Number of experiences to sample each training step
GAMMA = 0.99              # Discount factor: how much to care about future rewards
EPS_START = 0.9           # Starting exploration rate
EPS_END = 0.05            # Minimum exploration rate
EPS_DECAY = 1000          # How fast exploration decays
TARGET_UPDATE = 10        # Copy online → target every 10 episodes
LEARNING_RATE = 1e-4      # How fast the network learns
BUFFER_SIZE = 100000      # How many past experiences to remember

# Define the network (same as before)
class DQN(nn.Module):
    def __init__(self, n_observations, n_actions):
        super(DQN, self).__init__()
        self.layer1 = nn.Linear(n_observations, 128)
        self.layer2 = nn.Linear(128, 128)
        self.output = nn.Linear(128, n_actions)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        return self.output(x)

# Create the environment and networks
env = gym.make("CartPole-v1")
n_actions = env.action_space.n
state, _ = env.reset()
n_observations = len(state)

# Online network: the one that learns by playing
policy_net = DQN(n_observations, n_actions)
# Target network: the one that provides stable targets
target_net = DQN(n_observations, n_actions)
# Start them with the same weights
target_net.load_state_dict(policy_net.state_dict())

# Optimizer: the algorithm that adjusts the network weights
optimizer = optim.AdamW(policy_net.parameters(), lr=LEARNING_RATE, amsgrad=True)

# Replay buffer
Transition = namedtuple('Transition',
                        ('state', 'action', 'reward', 'next_state', 'done'))
memory = deque([], maxlen=BUFFER_SIZE)

# Tracking
steps_done = 0
episode_durations = []

Now the optimize_model() function — this is where the actual learning happens.

def optimize_model():
    """Sample a batch from memory and update the policy network."""
    # Don't train if we don't have enough samples
    if len(memory) < BATCH_SIZE:
        return

    # Step 1: Sample a random batch from memory
    transitions = random.sample(memory, BATCH_SIZE)
    
    # Step 2: Convert batch of transitions to tensors
    # This is a common pattern: transpose the batch to get tensors for each field
    batch = Transition(*zip(*transitions))
    
    # Stack all the states, actions, rewards, etc. into single tensors
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)
    next_state_batch = torch.cat(batch.next_state)
    done_batch = torch.cat(batch.done)

    # Step 3: Compute predicted Q-values for the actions we actually took
    # policy_net(state_batch) gives Q-values for all actions
    # .gather(1, action_batch) picks the Q-value for the action we took
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # Step 4: Compute target Q-values using the target network
    # For non-terminal states: r + γ * max Q_target(s', a')
    # For terminal states: just r (no future rewards)
    with torch.no_grad():
        next_state_values = target_net(next_state_batch).max(1)[0]
        # Set Q-values for terminal states to 0
        next_state_values[done_batch] = 0.0
        # The Bellman target
        expected_state_action_values = reward_batch + GAMMA * next_state_values

    # Step 5: Compute the loss (Huber loss is less sensitive to outliers than MSE)
    loss = F.smooth_l1_loss(state_action_values.squeeze(), expected_state_action_values)

    # Step 6: Backpropagate and update the network
    optimizer.zero_grad()
    loss.backward()
    # Clip gradients to prevent exploding gradients
    torch.nn.utils.clip_grad_value_(policy_net.parameters(), 100)
    optimizer.step()

    return loss.item()

Let’s break down what’s happening in optimize_model():

• Step 3: We ask the policy network “What Q-value did you predict for the action we actually took?” This is our current estimate.
• Step 4: We ask the target network “What’s the best future reward from the next state?” We add the immediate reward to get the target.
• Step 5: The loss is the gap between our estimate and the target. We want to minimize this gap.
• Step 6: Backpropagation adjusts the network weights to make the gap smaller next time.

Now the main training loop:

import time

num_episodes = 50  # Keep it short for demonstration

for episode in range(num_episodes):
    # Reset the environment
    state, _ = env.reset()
    state = torch.tensor(state, dtype=torch.float32).unsqueeze(0)
    
    total_reward = 0
    
    for t in range(10000):  # Limit steps per episode
        # Select action with epsilon-greedy
        eps_threshold = EPS_END + (EPS_START - EPS_END) * \
            math.exp(-1. * steps_done / EPS_DECAY)
        steps_done += 1
        
        if random.random() > eps_threshold:
            # Exploit: use the network's best guess
            with torch.no_grad():
                action = policy_net(state).max(1)[1].view(1, 1)
        else:
            # Explore: take a random action
            action = torch.tensor([[env.action_space.sample()]], dtype=torch.long)
        
        # Take the action
        next_state, reward, terminated, truncated, _ = env.step(action.item())
        done = terminated or truncated
        total_reward += reward
        
        # Convert to tensors and store in memory
        if terminated:
            # If the episode ended, there is no next state
            next_state_tensor = None
        else:
            next_state_tensor = torch.tensor(next_state, dtype=torch.float32).unsqueeze(0)
        
        reward_tensor = torch.tensor([reward], dtype=torch.float32)
        done_tensor = torch.tensor([done], dtype=torch.bool)
        
        memory.append(Transition(state, action, reward_tensor, 
                                 next_state_tensor, done_tensor))
        
        # Move to the next state
        state = next_state_tensor
        
        # Train the network
        loss = optimize_model()
        
        if done:
            episode_durations.append(t + 1)
            break
    
    # Update the target network every TARGET_UPDATE episodes
    if episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())
    
    if episode % 10 == 0:
        avg_reward = sum(episode_durations[-10:]) / min(len(episode_durations), 10)
        print(f"Episode {episode}: Average duration (last 10) = {avg_reward:.1f}")

print("Training complete!")
print(f"Final average duration: {sum(episode_durations[-10:])/10:.1f}")

When you run this, you’ll see something like:

Episode 0: Average duration = 12.0
Episode 10: Average duration = 25.3
Episode 20: Average duration = 45.7
Episode 30: Average duration = 78.2
Episode 40: Average duration = 112.5
Training complete!
Final average duration: 145.8

The agent starts terrible — it can barely keep the pole balanced for 12 steps. But by episode 40, it’s lasting over 100 steps. The network is learning.

Here’s the intuition one more time:

• The online network learns by playing. Every step, it gets a little better.
• The target network sits back and provides stable targets. It only updates every 10 episodes.
• The replay buffer makes sure the online network doesn’t get tunnel vision on recent moves.

Together, they turn a random flailing network into a competent pole-balancer.

5. What DQN Still Gets Wrong (and That’s Okay)

DQN is a breakthrough, but it’s not perfect. The biggest problem? Overestimation bias.

Here’s the issue: DQN uses the same network to both select the best action and evaluate how good it is. When computing the target, it does max Q(s', a') — it picks the action with the highest Q-value and uses that Q-value. But if the network has overestimated that action’s value (which happens easily with noisy data), the target is too high. This creates a feedback loop: the network thinks actions are better than they actually are.

This is called overestimation bias, and it’s a well-known problem. The Double DQN paper showed that standard DQN “suffers substantial Q-value overestimation on Atari.”

The fix is elegant: Double DQN decouples selection from evaluation. Use the online network to select the best action, but use the target network to evaluate it:

Q_target(s', argmax Q_online(s', a))

This simple change dramatically reduces overestimation and often leads to better performance.

There are other limitations too:

• Experience replay treats all transitions equally, but some are more informative than others. Prioritized replay weights important transitions higher.
• The network architecture doesn’t separate state value from action advantage. Dueling DQN splits the output into two streams: one estimates how good the state is (value), the other estimates how much better each action is than average (advantage).

But here’s the exciting part: each one of these limitations is an open design space. You can improve DQN right now by adding one of these ideas. The field rewards tinkering. You don’t need a PhD to make a better agent — you just need to understand the problem and try a fix.

6. You Just Built a Brain That Teaches Itself to Play

Let’s recap what you’ve learned:

1. Deep Q-Network: A neural network that replaces the Q-table, learning patterns instead of memorizing states. It can generalize to situations it has never seen before.

2. Experience replay: A buffer that stores past experiences and samples them randomly. This breaks the correlation between consecutive frames and prevents catastrophic forgetting.

3. Target network: A frozen copy of the Q-network that provides stable targets for learning. This prevents the “moving target” problem.

You can now build an agent that learns to play CartPole (and, with minimal changes, Breakout, Pong, or any Atari game) from raw pixels. That’s not a small thing. You’ve built a brain that teaches itself.

But DQN has a limitation we haven’t addressed: it can only handle discrete action spaces (left, right, up, down). What if you want to control a robot arm that can move continuously in any direction? What if you want to drive a car with a steering wheel that turns 0 to 360 degrees?

That’s where policy-gradient methods come in. In Part 6, we’ll explore REINFORCE and PPO — algorithms that can handle continuous action spaces and often learn more efficiently than DQN.

Before you go, open a notebook and run the DQN code on CartPole. Watch the agent go from random flailing to perfectly balancing the pole. Then — tweak something. Change the buffer size, the learning rate, the target update frequency. Break it and fix it. That’s how you truly learn.

Check Your Understanding

Remember: What are the two core innovations that make DQN work, and what problem does each solve?

Understand: Explain in your own words why consecutive game frames would break a neural network if you didn’t use experience replay.

Apply: Given a new environment with 10 continuous state variables and 3 discrete actions, what would the input and output dimensions of your DQN be?

Analyze: Compare the DQN training loop to the Q-table training loop from Part 4. What’s fundamentally different about how they learn?

Evaluate: If you saw that your DQN agent’s performance was plateauing early, which hyperparameter would you change first? Why?

Create: Design a modification to the DQN algorithm that could handle a continuous action space (e.g., a steering angle from -1 to 1). What would the network output look like?