Multi Agent Reinforcement Learning When Agents Hav

You trained a DQN agent in Part 5. It mastered CartPole. It balanced a pole on a cart like it was born doing it. You felt like a reinforcement learning wizard.

Now imagine dropping two of those agents into the same environment. They’re both learning. They’re both changing their behavior. What happens?

Chaos. Pure, oscillating, never-converging chaos.

You’ve asked the question: “Why can’t I just train each agent with the DQN from Part 5?” It’s a natural thought. Each agent has its own neural network, its own experience replay buffer, its own target network. They should learn together, right?

Wrong. Here’s why.

1. The Single-Agent Illusion

Think back to the DQN from Part 5. It worked because the environment was stationary. The transition probabilities — the chance of going from state A to state B given an action — didn’t change while the agent was learning. The reward function was fixed. The agent was the only thing changing in the world.

Now imagine two agents. Agent 1 is learning to navigate a grid. Agent 2 is also learning to navigate the same grid. When Agent 2 changes its policy — say, it starts moving left instead of right — the world Agent 1 sees changes. The transition function Agent 1 learned yesterday is wrong today. The environment is non-stationary from Agent 1’s perspective.

In plain English: imagine learning to play chess, but every time you make a move, your opponent’s brain rewires itself to counter your new strategy — and you have to learn from scratch again. That’s what MARL feels like.

This problem has a name: the moving-target problem. It was first identified in the foundational multi-agent literature. As the survey “Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications” (arXiv 1812.11794) explains, “the environment becomes non-stationary from the perspective of any individual agent because the policies of other agents are changing over time.” The original paper that showed this empirically was Tan 1993, “Independent versus Cooperative Agents.”

Here’s what naive independent DQN looks like in pseudocode:

# This is pseudocode — it won't run, but it shows the idea
# Each agent gets its own DQN
agent_1 = DQN(state_dim, action_dim)
agent_2 = DQN(state_dim, action_dim)

for episode in range(num_episodes):
    state = env.reset()
    done = False
    while not done:
        # Each agent acts independently
        action_1 = agent_1.act(state)
        action_2 = agent_2.act(state)
        
        # The environment steps with both actions
        next_state, reward_1, reward_2, done = env.step([action_1, action_2])
        
        # Each agent learns from its own experience
        agent_1.learn(state, action_1, reward_1, next_state, done)
        agent_2.learn(state, action_2, reward_2, next_state, done)
        
        state = next_state

This looks reasonable. But it’s broken. Each agent’s learning update assumes the world is stationary. But the world is changing because the other agent is learning. The result? The agents chase each other’s tails, and neither converges.

This article is about fixing that. We’ll build intuition for the three main MARL settings — cooperative, competitive, and mixed. Then we’ll code a simple multi-agent environment and a basic independent-learner baseline to watch it fail. Then we’ll fix it with Centralized Training with Decentralized Execution (CTDE).

2. What Even Is a Multi-Agent Game? (Formally, But Gently)

An MDP — which you learned in Part 4 — is a game against nature. You make a move, nature responds with a new state and a reward. There’s no other agent trying to maximize its own reward.

A Markov Game (also called a stochastic game) is the natural generalization. It’s a game against other agents who also have goals. The formalism was introduced by Littman in 1994 (“Markov Games as a Framework for Multi-Agent Reinforcement Learning”).

Here’s what a Markov Game consists of:

• States: The set of all possible configurations of the world.
• Per-agent action spaces: Each agent has its own set of actions it can take.
• Transition function: Given the current state and the joint action (every agent’s action), what’s the probability of moving to the next state?
• Per-agent reward functions: Each agent gets its own reward signal.

In plain English: a Markov Game is just an MDP where the “action” is actually a vector of actions — one per agent — and each agent gets its own reward.

There are three main settings:

1. Fully cooperative: All agents share the same reward. It’s a team sport — everyone gets the same score. Example: two robots moving a heavy box together.

2. Fully competitive: Two agents with opposite rewards. One agent’s win is the other’s loss. Example: chess, Go, or any two-player zero-sum game.

3. Mixed (general-sum): Agents have their own rewards that may align or conflict. This is the real world — sometimes you cooperate, sometimes you compete, and sometimes both at once. Example: traffic routing, where each car wants to minimize its own travel time, but collectively they want to avoid gridlock.

Let’s make this concrete with a running example. We’ll build a simple 2-agent cooperative Predator-Prey grid. Two predators must corner one prey. The predators share a reward: +10 if they catch the prey together, -1 per step otherwise. This is small enough to visualize and code, but rich enough to show non-stationarity and the need for coordination.

import numpy as np
import matplotlib.pyplot as plt

class PredatorPreyGrid:
    """
    A simple 2-agent cooperative Predator-Prey grid.
    Two predators (agents) must corner a single prey.
    The prey moves randomly. Predators share the reward.
    """
    def __init__(self, grid_size=5):
        self.grid_size = grid_size
        self.state_dim = 6  # (pred1_x, pred1_y, pred2_x, pred2_y, prey_x, prey_y)
        self.action_dim = 4  # 0: up, 1: down, 2: left, 3: right
        self.num_agents = 2
        
        # Initialize positions
        self.reset()
    
    def reset(self):
        """Reset the environment to a random initial state."""
        # Place agents and prey at random non-overlapping positions
        positions = np.random.choice(self.grid_size * self.grid_size, 3, replace=False)
        self.pred1_pos = [positions[0] // self.grid_size, positions[0] % self.grid_size]
        self.pred2_pos = [positions[1] // self.grid_size, positions[1] % self.grid_size]
        self.prey_pos = [positions[2] // self.grid_size, positions[2] % self.grid_size]
        
        return self._get_obs()
    
    def _get_obs(self):
        """Return the full state as a numpy array."""
        return np.array([
            self.pred1_pos[0], self.pred1_pos[1],
            self.pred2_pos[0], self.pred2_pos[1],
            self.prey_pos[0], self.prey_pos[1]
        ], dtype=np.float32)
    
    def step(self, actions):
        """
        Take a step with the given joint action.
        actions: list of two integers (0-3) for each predator.
        Returns: next_state, rewards (list of 2 floats), done (bool)
        """
        # Move predator 1
        self.pred1_pos = self._move(self.pred1_pos, actions[0])
        # Move predator 2
        self.pred2_pos = self._move(self.pred2_pos, actions[1])
        # Move prey randomly
        prey_action = np.random.randint(0, 4)
        self.prey_pos = self._move(self.prey_pos, prey_action)
        
        # Check if both predators are adjacent to the prey (cornered)
        pred1_adj = self._is_adjacent(self.pred1_pos, self.prey_pos)
        pred2_adj = self._is_adjacent(self.pred2_pos, self.prey_pos)
        
        # Cornered means both predators are adjacent to the prey
        cornered = pred1_adj and pred2_adj
        
        # Reward: +10 if cornered, -1 per step otherwise
        reward = 10.0 if cornered else -1.0
        
        # Both agents share the same reward (cooperative)
        rewards = [reward, reward]
        
        done = cornered
        
        return self._get_obs(), rewards, done
    
    def _move(self, pos, action):
        """Move a position by one step in the given direction."""
        new_pos = pos.copy()
        if action == 0:  # up
            new_pos[0] = max(0, pos[0] - 1)
        elif action == 1:  # down
            new_pos[0] = min(self.grid_size - 1, pos[0] + 1)
        elif action == 2:  # left
            new_pos[1] = max(0, pos[1] - 1)
        elif action == 3:  # right
            new_pos[1] = min(self.grid_size - 1, pos[1] + 1)
        return new_pos
    
    def _is_adjacent(self, pos1, pos2):
        """Check if two positions are adjacent (Manhattan distance = 1)."""
        return abs(pos1[0] - pos2[0]) + abs(pos1[1] - pos2[1]) == 1

# Let's test it with random actions
env = PredatorPreyGrid(grid_size=5)
obs = env.reset()
print(f"Initial state: {obs}")
print(f"  Predator 1 at ({obs[0]:.0f}, {obs[1]:.0f})")
print(f"  Predator 2 at ({obs[2]:.0f}, {obs[3]:.0f})")
print(f"  Prey at ({obs[4]:.0f}, {obs[5]:.0f})")
print()

# Take 10 random steps
for step in range(10):
    actions = [np.random.randint(0, 4), np.random.randint(0, 4)]
    next_obs, rewards, done = env.step(actions)
    print(f"Step {step}: actions={actions}, reward={rewards[0]:.1f}, done={done}")
    print(f"  Predator 1 at ({next_obs[0]:.0f}, {next_obs[1]:.0f})")
    print(f"  Predator 2 at ({next_obs[2]:.0f}, {next_obs[3]:.0f})")
    print(f"  Prey at ({next_obs[4]:.0f}, {next_obs[5]:.0f})")
    if done:
        print("  They caught the prey!")
        break
    obs = next_obs

When you run this, you’ll see two predators and one prey moving around a 5x5 grid. With random actions, they almost never catch the prey. The reward stays at -1 per step. That’s the baseline we need to beat.

3. The Naive Approach: Independent Q-Learning (And Why It Fails)

Now let’s try the most natural thing: give each agent its own DQN from Part 5 and let them learn independently. This is called Independent Q-Learning (IQL).

Here’s the architecture:

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random

# Set random seeds for reproducibility
torch.manual_seed(42)
np.random.seed(42)
random.seed(42)

class DQN(nn.Module):
    """
    A simple Deep Q-Network.
    Same architecture as Part 5.
    """
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        super(DQN, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )
    
    def forward(self, x):
        return self.net(x)

class ReplayBuffer:
    """
    Experience replay buffer.
    Stores (state, action, reward, next_state, done) tuples.
    """
    def __init__(self, capacity=10000):
        self.buffer = deque(maxlen=capacity)
    
    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))
    
    def sample(self, batch_size):
        batch = random.sample(self.buffer, batch_size)
        states = torch.FloatTensor([b[0] for b in batch])
        actions = torch.LongTensor([b[1] for b in batch])
        rewards = torch.FloatTensor([b[2] for b in batch])
        next_states = torch.FloatTensor([b[3] for b in batch])
        dones = torch.FloatTensor([b[4] for b in batch])
        return states, actions, rewards, next_states, dones
    
    def __len__(self):
        return len(self.buffer)

class IndependentDQNAgent:
    """
    A single DQN agent for independent Q-learning.
    """
    def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99, epsilon=1.0, epsilon_min=0.01, epsilon_decay=0.995):
        self.q_network = DQN(state_dim, action_dim)
        self.target_network = DQN(state_dim, action_dim)
        self.target_network.load_state_dict(self.q_network.state_dict())
        self.optimizer = optim.Adam(self.q_network.parameters(), lr=lr)
        self.replay_buffer = ReplayBuffer()
        
        self.gamma = gamma
        self.epsilon = epsilon
        self.epsilon_min = epsilon_min
        self.epsilon_decay = epsilon_decay
        self.action_dim = action_dim
        self.update_target_every = 100
        self.step_count = 0
    
    def act(self, state, training=True):
        """Epsilon-greedy action selection."""
        if training and np.random.random() < self.epsilon:
            return np.random.randint(0, self.action_dim)
        
        with torch.no_grad():
            state_tensor = torch.FloatTensor(state).unsqueeze(0)
            q_values = self.q_network(state_tensor)
            return q_values.argmax().item()
    
    def learn(self, batch_size=64):
        """Update the Q-network using a batch from the replay buffer."""
        if len(self.replay_buffer) < batch_size:
            return
        
        states, actions, rewards, next_states, dones = self.replay_buffer.sample(batch_size)
        
        # Compute current Q-values
        current_q = self.q_network(states).gather(1, actions.unsqueeze(1)).squeeze()
        
        # Compute target Q-values using target network
        with torch.no_grad():
            next_q = self.target_network(next_states).max(1)[0]
            target_q = rewards + self.gamma * next_q * (1 - dones)
        
        # Compute loss and update
        loss = nn.MSELoss()(current_q, target_q)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        
        # Decay epsilon
        self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
        
        # Update target network periodically
        self.step_count += 1
        if self.step_count % self.update_target_every == 0:
            self.target_network.load_state_dict(self.q_network.state_dict())
        
        return loss.item()

# Now let's run independent Q-learning on Predator-Prey
env = PredatorPreyGrid(grid_size=5)

# Create two independent agents
agent_1 = IndependentDQNAgent(state_dim=env.state_dim, action_dim=env.action_dim)
agent_2 = IndependentDQNAgent(state_dim=env.state_dim, action_dim=env.action_dim)

num_episodes = 500
max_steps = 50
batch_size = 64

# Track rewards
episode_rewards = []

for episode in range(num_episodes):
    state = env.reset()
    episode_reward = 0
    
    for step in range(max_steps):
        # Each agent acts independently
        action_1 = agent_1.act(state)
        action_2 = agent_2.act(state)
        
        # Environment step
        next_state, rewards, done = env.step([action_1, action_2])
        
        # Store experience for each agent
        agent_1.replay_buffer.push(state, action_1, rewards[0], next_state, done)
        agent_2.replay_buffer.push(state, action_2, rewards[1], next_state, done)
        
        # Each agent learns
        agent_1.learn(batch_size)
        agent_2.learn(batch_size)
        
        episode_reward += rewards[0]  # Both agents get same reward
        state = next_state
        
        if done:
            break
    
    episode_rewards.append(episode_reward)
    
    if (episode + 1) % 100 == 0:
        avg_reward = np.mean(episode_rewards[-100:])
        print(f"Episode {episode + 1}: Average reward = {avg_reward:.2f}, Epsilon = {agent_1.epsilon:.3f}")

# Plot the results
plt.figure(figsize=(10, 5))
plt.plot(episode_rewards, alpha=0.3, color='blue', label='Episode reward')
# Smooth with a moving average
window = 50
smoothed = np.convolve(episode_rewards, np.ones(window)/window, mode='valid')
plt.plot(range(window-1, num_episodes), smoothed, color='red', linewidth=2, label=f'{window}-episode moving average')
plt.xlabel('Episode')
plt.ylabel('Team Reward')
plt.title('Independent Q-Learning on Predator-Prey')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

When you run this, you’ll see something frustrating. The reward oscillates wildly. It never converges. Some episodes the agents get lucky and catch the prey (reward of 10 minus steps), but most episodes they don’t. The moving average bounces between -20 and 5.

What’s actually going on here? Each agent’s Bellman update assumes the transition function is fixed. But the transition function depends on what the other agent does. When Agent 1 changes its policy — say, it learns to move left more often — the world Agent 2 sees changes. Agent 2’s Q-values, which were based on the old policy, are now wrong. Agent 2 has to unlearn and relearn. But while Agent 2 is relearning, Agent 1’s world changes again. They’re chasing a moving target.

This is the hardest part of MARL: the very thing that made DQN work — the assumption of a stationary environment — is broken the moment you add a second learning agent.

The survey by Zhang, Yang & Başar (arXiv 1911.10635) explicitly calls this out as a core challenge: “The non-stationarity issue arises because the environment’s transition dynamics depend on the policies of all agents, which are changing during learning.” Tan’s 1993 paper showed empirically that independent learners underperform cooperative learners precisely because of this issue.

4. The Big Fix: Centralized Training with Decentralized Execution (CTDE)

So independent Q-learning fails. What’s the fix?

The core insight is this: during training, we can let each agent peek at everyone else’s information. But during execution, each agent acts only on its own local observations.

Think of a soccer team. During practice (training), the coach sees everything — all players’ positions, the ball, the opponents. The coach can give advice based on the full picture. But during a match (execution), each player only sees what’s in front of them and must decide on their own.

This is called Centralized Training with Decentralized Execution (CTDE). It was introduced in the MADDPG paper (arXiv 1706.02275) and the VDN paper (arXiv 1706.05296).

Here’s the architecture:

During Training:

    Agent 1          Agent 2
      |                |
   [Actor 1]       [Actor 2]
      |                |
   action_1        action_2
      \                /
       \              /
      [Centralized Critic]
            |
      (sees all states and actions)
            |
       TD Error
            |
     Updates both actors

During Execution:

    Agent 1          Agent 2
      |                |
   [Actor 1]       [Actor 2]
      |                |
   action_1        action_2
      |                |
   (acts on local obs only)

In plain English: CTDE solves non-stationarity by giving each agent a stable training signal from a global view, while keeping execution practical and scalable.

The centralized critic sees the full state and all agents’ actions. This means the critic’s estimate of the Q-value is stable — it doesn’t change just because another agent changed its policy. The critic provides a fixed target for each agent to learn from.

5. CTDE in Action: Value Decomposition Networks (VDN)

Let’s implement a simple CTDE algorithm for our cooperative Predator-Prey environment. We’ll use Value Decomposition Networks (VDN).

The VDN idea is elegant: the team Q-value is the sum of per-agent Q-values. Each agent has its own Q-network. During training, we compute the team Q-value by summing the individual Q-values. We then compute the TD error using the team reward. We backpropagate this error through all the agent networks.

In plain English: VDN says “the team’s total value is just the sum of each player’s individual value.” This is a strong assumption — it doesn’t capture complex interactions where one agent’s value depends on what another agent does. But it’s simple and it works for many cooperative tasks.

Here’s the VDN implementation:

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random
import matplotlib.pyplot as plt

# Set random seeds for reproducibility
torch.manual_seed(42)
np.random.seed(42)
random.seed(42)

# Reuse the PredatorPreyGrid class from earlier (redefined here for self-containment)
class PredatorPreyGrid:
    def __init__(self, grid_size=5):
        self.grid_size = grid_size
        self.state_dim = 6
        self.action_dim = 4
        self.num_agents = 2
        self.reset()
    
    def reset(self):
        positions = np.random.choice(self.grid_size * self.grid_size, 3, replace=False)
        self.pred1_pos = [positions[0] // self.grid_size, positions[0] % self.grid_size]
        self.pred2_pos = [positions[1] // self.grid_size, positions[1] % self.grid_size]
        self.prey_pos = [positions[2] // self.grid_size, positions[2] % self.grid_size]
        return self._get_obs()
    
    def _get_obs(self):
        return np.array([
            self.pred1_pos[0], self.pred1_pos[1],
            self.pred2_pos[0], self.pred2_pos[1],
            self.prey_pos[0], self.prey_pos[1]
        ], dtype=np.float32)
    
    def step(self, actions):
        self.pred1_pos = self._move(self.pred1_pos, actions[0])
        self.pred2_pos = self._move(self.pred2_pos, actions[1])
        prey_action = np.random.randint(0, 4)
        self.prey_pos = self._move(self.prey_pos, prey_action)
        
        pred1_adj = self._is_adjacent(self.pred1_pos, self.prey_pos)
        pred2_adj = self._is_adjacent(self.pred2_pos, self.prey_pos)
        cornered = pred1_adj and pred2_adj
        
        reward = 10.0 if cornered else -1.0
        rewards = [reward, reward]
        done = cornered
        
        return self._get_obs(), rewards, done
    
    def _move(self, pos, action):
        new_pos = pos.copy()
        if action == 0:
            new_pos[0] = max(0, pos[0] - 1)
        elif action == 1:
            new_pos[0] = min(self.grid_size - 1, pos[0] + 1)
        elif action == 2:
            new_pos[1] = max(0, pos[1] - 1)
        elif action == 3:
            new_pos[1] = min(self.grid_size - 1, pos[1] + 1)
        return new_pos
    
    def _is_adjacent(self, pos1, pos2):
        return abs(pos1[0] - pos2[0]) + abs(pos1[1] - pos2[1]) == 1

class QNetwork(nn.Module):
    """A simple Q-network for one agent."""
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        super(QNetwork, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )
    
    def forward(self, x):
        return self.net(x)

class VDNAgent:
    """
    A VDN agent for cooperative multi-agent settings.
    Each agent has its own Q-network, but they share a centralized training signal.
    """
    def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99, epsilon=1.0, epsilon_min=0.01, epsilon_decay=0.995):
        self.q_network = QNetwork(state_dim, action_dim)
        self.target_network = QNetwork(state_dim, action_dim)
        self.target_network.load_state_dict(self.q_network.state_dict())
        self.optimizer = optim.Adam(self.q_network.parameters(), lr=lr)
        self.replay_buffer = deque(maxlen=10000)
        
        self.gamma = gamma
        self.epsilon = epsilon
        self.epsilon_min = epsilon_min
        self.epsilon_decay = epsilon_decay
        self.action_dim = action_dim
        self.update_target_every = 100
        self.step_count = 0
    
    def act(self, state, training=True):
        if training and np.random.random() < self.epsilon:
            return np.random.randint(0, self.action_dim)
        
        with torch.no_grad():
            state_tensor = torch.FloatTensor(state).unsqueeze(0)
            q_values = self.q_network(state_tensor)
            return q_values.argmax().item()
    
    def push_to_buffer(self, state, action, reward, next_state, done):
        self.replay_buffer.append((state, action, reward, next_state, done))
    
    def sample_buffer(self, batch_size):
        batch = random.sample(self.replay_buffer, batch_size)
        states = torch.FloatTensor([b[0] for b in batch])
        actions = torch.LongTensor([b[1] for b in batch])
        rewards = torch.FloatTensor([b[2] for b in batch])
        next_states = torch.FloatTensor([b[3] for b in batch])
        dones = torch.FloatTensor([b[4] for b in batch])
        return states, actions, rewards, next_states, dones
    
    def __len__(self):
        return len(self.replay_buffer)

# Now let's train VDN
env = PredatorPreyGrid(grid_size=5)

# Create two VDN agents
agent_1 = VDNAgent(state_dim=env.state_dim, action_dim=env.action_dim)
agent_2 = VDNAgent(state_dim=env.state_dim, action_dim=env.action_dim)

num_episodes = 500
max_steps = 50
batch_size = 64

# Track rewards
episode_rewards = []

for episode in range(num_episodes):
    state = env.reset()
    episode_reward = 0
    
    for step in range(max_steps):
        # Each agent acts independently (decentralized execution)
        action_1 = agent_1.act(state)
        action_2 = agent_2.act(state)
        
        # Environment step
        next_state, rewards, done = env.step([action_1, action_2])
        
        # Store experience for each agent
        agent_1.push_to_buffer(state, action_1, rewards[0], next_state, done)
        agent_2.push_to_buffer(state, action_2, rewards[1], next_state, done)
        
        # VDN training: compute team Q-value and TD error
        if len(agent_1) >= batch_size and len(agent_2) >= batch_size:
            # Sample batches from both agents
            states_1, actions_1, rewards_1, next_states_1, dones_1 = agent_1.sample_buffer(batch_size)
            states_2, actions_2, rewards_2, next_states_2, dones_2 = agent_2.sample_buffer(batch_size)
            
            # Compute individual Q-values
            q_1 = agent_1.q_network(states_1).gather(1, actions_1.unsqueeze(1)).squeeze()
            q_2 = agent_2.q_network(states_2).gather(1, actions_2.unsqueeze(1)).squeeze()
            
            # Team Q-value is the sum of individual Q-values
            team_q = q_1 + q_2
            
            # Compute target team Q-value
            with torch.no_grad():
                next_q_1 = agent_1.target_network(next_states_1).max(1)[0]
                next_q_2 = agent_2.target_network(next_states_2).max(1)[0]
                next_team_q = next_q_1 + next_q_2
                
                # Team reward is the sum of individual rewards (they're the same here)
                team_reward = rewards_1 + rewards_2
                target_team_q = team_reward + agent_1.gamma * next_team_q * (1 - dones_1)
            
            # Compute loss and update both agents
            loss = nn.MSELoss()(team_q, target_team_q)
            
            agent_1.optimizer.zero_grad()
            agent_2.optimizer.zero_grad()
            loss.backward()
            agent_1.optimizer.step()
            agent_2.optimizer.step()
            
            # Decay epsilon
            agent_1.epsilon = max(agent_1.epsilon_min, agent_1.epsilon * agent_1.epsilon_decay)
            agent_2.epsilon = max(agent_2.epsilon_min, agent_2.epsilon * agent_2.epsilon_decay)
            
            # Update target networks
            agent_1.step_count += 1
            agent_2.step_count += 1
            if agent_1.step_count % agent_1.update_target_every == 0:
                agent_1.target_network.load_state_dict(agent_1.q_network.state_dict())
            if agent_2.step_count % agent_2.update_target_every == 0:
                agent_2.target_network.load_state_dict(agent_2.q_network.state_dict())
        
        episode_reward += rewards[0]
        state = next_state
        
        if done:
            break
    
    episode_rewards.append(episode_reward)
    
    if (episode + 1) % 100 == 0:
        avg_reward = np.mean(episode_rewards[-100:])
        print(f"Episode {episode + 1}: Average reward = {avg_reward:.2f}, Epsilon = {agent_1.epsilon:.3f}")

# Plot the results
plt.figure(figsize=(10, 5))
plt.plot(episode_rewards, alpha=0.3, color='green', label='Episode reward')
window = 50
smoothed = np.convolve(episode_rewards, np.ones(window)/window, mode='valid')
plt.plot(range(window-1, num_episodes), smoothed, color='darkgreen', linewidth=2, label=f'{window}-episode moving average')
plt.xlabel('Episode')
plt.ylabel('Team Reward')
plt.title('VDN on Predator-Prey')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

When you run this, you’ll see a very different picture. The team reward climbs steadily. By episode 300, the moving average is around 5 to 8, meaning the agents are consistently catching the prey. The reward doesn’t oscillate wildly — it converges.

What does this mean in plain English? The team reward went from oscillating between -20 and 5 to steadily climbing to 8. That’s the difference between agents fighting each other and agents cooperating.

Now here’s the catch: VDN assumes the team value is a simple sum. This doesn’t capture complex interactions. For example, what if the best team strategy requires one agent to sacrifice its own reward for the team? VDN can’t handle that because it assumes each agent’s value is independent. That’s where QMIX comes in — QMIX uses a monotonic mixing network that can capture more complex interactions. But that’s for another day.

6. When Agents Compete: Self-Play and the Autocurriculum

So far we’ve looked at cooperative settings. What about competitive settings — where one agent’s win is the other’s loss?

The most natural approach is self-play: an agent plays against copies of itself (or past versions of itself).

Imagine a chess player who only ever plays against their past self. Every time they get better, their opponent (their past self) is also better. They can never coast — they must keep innovating.

This creates an autocurriculum: the difficulty of the opponent automatically scales with the agent’s skill. The agent doesn’t need a hand-crafted curriculum of easy-to-hard opponents. The difficulty emerges naturally.

The most famous example is OpenAI’s Hide-and-Seek (arXiv 1909.07528). Agents learned to cooperate and compete in a simple hide-and-seek game. They self-organized into six escalating strategy phases — from simple hiding to building shelters to using ramps to escape. No reward shaping for tool use. Just competitive pressure.

But self-play has a failure mode: policy cycling. This is like Rock-Paper-Scissors dynamics. The agent learns to beat the current opponent, but in doing so, it forgets how to beat old strategies. The opponent then switches to a strategy that beats the new one, and the cycle continues.

DeepMind’s AlphaStar solved this with League training. Instead of playing against just one copy of itself, AlphaStar trained a population of agents: “main agents” that learned to beat the current best strategies, and “exploiters” that learned to find weaknesses in the main agents. This prevented policy cycling and produced a robust final policy.

7. Mixed Motives: When You Need Both Cooperation and Competition

The real world is rarely purely cooperative or purely competitive. Most multi-agent settings are mixed: you need to cooperate with teammates while competing against opponents.

DeepMind’s Capture the Flag (FTW) agent is the canonical example. The agent played a 2v2 version of the classic game. It needed to balance two conflicting goals: help your teammate score, but stop the opponent from scoring. A purely selfish agent would never pass the flag. A purely cooperative agent would never defend.

The FTW agent was trained with population-based training and learned internal reward signals (intrinsic motivation) to cooperate with teammates. It achieved a ~79% win rate against human players, even though humans had faster reaction times.

One concrete technique for mixed settings is Cooperative and Competitive Biases (arXiv 2101.06890). The idea is to bias each agent’s value updates with an “imaginary” joint action: assume cooperative teammates maximize the agent’s value, and competitive opponents minimize it. Then gradually anneal the bias away as the agent learns the true dynamics.

In plain English: you start by giving each agent a little nudge to assume the best of teammates and the worst of opponents. Over time, you remove the nudge and let the agent learn the true dynamics.

8. How Do You Actually Train These Things? Practical Tips

You understand the theory. You’ve seen the code. But how do you actually run a MARL experiment? Here are practical tips.

Use PettingZoo for Environments

PettingZoo is the Gymnasium-style API for multi-agent environments. It provides the same reset() and step() interface, but for multiple agents.

# This is a standalone example showing PettingZoo's API
# Note: You need to install pettingzoo: pip install pettingzoo

from pettingzoo.mpe import simple_speaker_listener_v4

# Create the environment
env = simple_speaker_listener_v4.env(render_mode='human')

# Reset the environment
observations, infos = env.reset()

print(f"Agents: {env.agents}")
print(f"Observation space: {env.observation_space}")
print(f"Action space: {env.action_space}")
print()

# Take a few steps with random actions
for step in range(5):
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminations, truncations, infos = env.step(actions)
    
    print(f"Step {step}:")
    for agent in env.agents:
        print(f"  {agent}: reward={rewards[agent]:.2f}, terminated={terminations[agent]}, truncated={truncations[agent]}")
    
    if all(terminations.values()) or all(truncations.values()):
        break

env.close()

Start Small

Start with 2-3 agents and a simple environment. The Predator-Prey grid we built is perfect. Don’t jump to 10-agent StarCraft battles on your first try.

Use Parameter Sharing

If your agents are homogeneous (same observation space, same action space, same reward structure), share the network weights. One network, multiple agent IDs. This is sample-efficient and stable.

# Parameter sharing: one network for all agents
class SharedDQN(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 64),
            nn.ReLU(),
            nn.Linear(64, action_dim)
        )
    
    def forward(self, x):
        return self.net(x)

# All agents use the same network
shared_network = SharedDQN(state_dim=6, action_dim=4)

# Each agent acts using the same network
state = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
with torch.no_grad():
    state_tensor = torch.FloatTensor(state).unsqueeze(0)
    q_values = shared_network(state_tensor)
    action_1 = q_values.argmax().item()
    
# Agent 2 uses the same network
    q_values = shared_network(state_tensor)
    action_2 = q_values.argmax().item()

print(f"Agent 1 action: {action_1}")
print(f"Agent 2 action: {action_2}")

Always Compare Against Independent Learning

It’s tempting to skip the baseline. Don’t. Always run independent Q-learning as a baseline to measure the value of your CTDE method. If your VDN doesn’t beat independent Q-learning, something is wrong.

Use Curriculum Learning

The Overcooked curriculum paper (arXiv 2312.11768) showed that training with a teammate of decreasing skill improves team reward but can slow individual skill growth. This is a non-obvious tradeoff. If you’re training a team, consider starting with a scripted teammate and gradually replacing it with a learned one.

9. Recap: What You Just Learned

Let’s summarize the key takeaways:

• Multi-agent environments are non-stationary — each agent’s learning changes the world for every other agent.
• Independent Q-learning fails because it ignores this non-stationarity. Each agent chases a moving target.
• Centralized Training with Decentralized Execution (CTDE) is the core fix: train with global information, execute with local information.
• VDN is a simple CTDE algorithm for cooperative settings: decompose the team value into per-agent values by summing them.
• Self-play creates an autocurriculum for competitive settings, but can lead to policy cycling — solved by diverse training populations (AlphaStar’s League).
• Mixed settings require balancing cooperation and competition — techniques like biased value updates can help.
• Practical tools: PettingZoo for environments, parameter sharing for efficiency, curriculum learning for smoother training.

10. Where to Go Next

You’ve built a working VDN. But VDN assumes the team value is a simple sum. What if the best team strategy requires one agent to sacrifice its own reward for the team? QMIX handles that with a monotonic mixing network.

What if you want to handle continuous actions? MADDPG extends the actor-critic idea to multiple agents.

Try implementing VDN on a PettingZoo environment. Then try breaking it by adding a third agent. See what happens.

11. Check Your Understanding

Remember: What is the non-stationarity problem in MARL?

Understand: Explain in your own words why independent Q-learning fails in a multi-agent environment.

Apply: Given a cooperative 2-agent environment, design a VDN architecture. What are the inputs and outputs of each network?

Analyze: Compare VDN and independent Q-learning on the Predator-Prey environment. Why does VDN converge while independent Q-learning oscillates?

Evaluate: In a mixed cooperative-competitive setting (e.g., Capture the Flag), what are the tradeoffs of using a purely selfish reward versus a team reward?

Create: Propose a modification to VDN that could handle a setting where agents have different roles (e.g., one predator chases, the other cuts off escape routes).

12. Apply What You Learned