Rlhf Explained How Chatgpt Was Actually Trained To

You’ve probably used ChatGPT and thought, “Wow, this thing actually understands me.” It doesn’t just complete your sentences — it follows instructions, refuses harmful requests, and tries to be genuinely helpful. But here’s the weird part: the base model that ChatGPT started from was just a next-token predictor. It was trained to guess the next word in a sentence, nothing more.

So how do you go from a model that predicts “bomb-making instructions” as a plausible next-token completion to one that says “I can’t help with that”? That’s the problem RLHF solves.

Think about it this way: a pretrained language model is like a parrot that’s memorized the entire internet. If you ask it “How do I make a bomb?”, it will happily complete the text because it’s seen bomb-making instructions in its training data. It doesn’t know that some things shouldn’t be said. It’s just predicting tokens.

Now, you might think supervised fine-tuning (SFT) fixes this. You show the model thousands of examples of helpful, harmless responses and train it to mimic them. And it works — sort of. The model learns the format of a good answer. But it still doesn’t understand why some responses are better than others. It’s like teaching a student to copy the teacher’s answers without understanding the subject. They can pass the test but fail in the real world.

This is where RLHF comes in. It’s a three-stage pipeline that teaches a model not just to imitate good responses, but to internalize human preferences. By the end of this tutorial, you’ll understand each stage, why it exists, and see annotated code for all three. We’ll build a tiny RLHF system from scratch using GPT-2, so you can see exactly what’s happening under the hood.

Stage 1: Supervised Fine-Tuning (SFT) — Teaching the Model to Follow Instructions

Before we can align a model, we need it to understand the format of a helpful response. This is the imitation learning step.

Here’s how it works in the real world (from the InstructGPT paper): human labelers write demonstrations of ideal assistant responses to prompts. They see a prompt like “Explain quantum computing to a 10-year-old” and write a clear, simple explanation. This creates a dataset of (prompt, ideal_response) pairs.

Then we train the model on this dataset using standard supervised learning — cross-entropy loss on the demonstration tokens. The model learns to predict the next token in the ideal response, given the prompt and previous tokens.

But here’s the catch: SFT alone makes the model mimic style, not internalize values. It can still produce harmful or unhelpful content because it hasn’t learned what “good” means beyond the surface. It’s like teaching a student to copy the teacher’s answers — they can pass the test but don’t understand the subject.

Let’s see this in code. We’ll fine-tune a small GPT-2 model on a tiny instruction dataset.

# Block 1: Supervised Fine-Tuning (SFT) on a small instruction dataset
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from datasets import Dataset
import numpy as np

# Load a small GPT-2 model and tokenizer
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Add padding token (GPT-2 doesn't have one by default)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Create a tiny instruction dataset (like a mini Alpaca)
data = {
    "prompt": [
        "What is 2+2?",
        "Explain gravity simply.",
        "Write a haiku about AI.",
        "What is the capital of France?",
        "Tell me a joke."
    ],
    "response": [
        "2+2 equals 4.",
        "Gravity is a force that pulls objects toward each other. On Earth, it makes things fall down.",
        "Silicon dreams wake,\nLearning patterns in the dark,\nThinking without thought.",
        "The capital of France is Paris.",
        "Why did the AI cross the road? To optimize the chicken's path!"
    ]
}

# Format as instruction-following pairs
def format_instruction(prompt, response):
    return f"### Instruction:\n{prompt}\n\n### Response:\n{response}"

formatted_texts = [format_instruction(p, r) for p, r in zip(data["prompt"], data["response"])]

# Tokenize
def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=128)

dataset = Dataset.from_dict({"text": formatted_texts})
tokenized_dataset = dataset.map(tokenize_function, batched=True)

# Set up training arguments (tiny model, quick run)
training_args = TrainingArguments(
    output_dir="./gpt2-sft",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    save_steps=10,
    logging_steps=5,
    learning_rate=5e-5,
    report_to="none",
)

# Define trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
)

# Train
print("Starting SFT training...")
trainer.train()
print("SFT training complete!")

# Save the model
model.save_pretrained("./gpt2-sft-final")
tokenizer.save_pretrained("./gpt2-sft-final")

# Test generation before and after (we'll use the saved model later)
print("\n--- Testing SFT model ---")
test_prompt = "What is machine learning?"
input_text = f"### Instruction:\n{test_prompt}\n\n### Response:\n"
inputs = tokenizer(input_text, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50, temperature=0.7)
print(f"Prompt: {test_prompt}")
print(f"Response: {tokenizer.decode(outputs[0], skip_special_tokens=True)}")

When you run this, you’ll see the model start to produce responses that follow the instruction format. But notice something: the model might still produce weird or off-topic responses. It’s learned the structure of a helpful answer, but it hasn’t learned what makes an answer good.

Stage 2: Training the Reward Model — Teaching a Judge What ‘Good’ Looks Like

Now we need a way to score responses that captures human preferences. This is the hardest part of RLHF.

Here’s the core insight: humans are much better at comparing two options than assigning an absolute score. If I show you two responses to the same prompt and ask “Which one is better?”, you can usually answer confidently. But if I ask “Rate this response on a scale of 1-10”, different people will give wildly different scores.

So the reward model is trained on pairwise comparisons. Human labelers see multiple model outputs for the same prompt and rank them from best to worst. The reward model learns to predict which response a human would prefer.

Mathematically, the reward model is a binary classifier (or regression model) that takes a response and outputs a scalar score. The loss function encourages it to rank the human-preferred response higher. If response A is preferred over response B, the loss is:

loss = -log(σ(reward(A) - reward(B)))

where σ is the sigmoid function. This pushes the reward model to give higher scores to preferred responses.

But here’s the hard truth: the reward model is a proxy for human judgment — it’s never perfect, and it’s where many alignment failures originate. Think of the reward model as a very opinionated critic. If the critic is biased, the model will learn that bias.

Let’s build a tiny reward model from scratch.

# Block 2: Training a Reward Model on synthetic preference data
import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModelForCausalLM
import numpy as np

# Load the same GPT-2 base (we'll add a regression head)
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
base_model = AutoModelForCausalLM.from_pretrained(model_name)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Create a reward model by adding a regression head to GPT-2
class RewardModel(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.base_model = base_model
        self.reward_head = nn.Linear(base_model.config.hidden_size, 1)
        
    def forward(self, input_ids, attention_mask=None):
        # Get the last hidden state
        outputs = self.base_model(input_ids, attention_mask=attention_mask, output_hidden_states=True)
        # Take the hidden state of the last token (or mean pooling)
        last_hidden = outputs.hidden_states[-1]  # [batch, seq_len, hidden]
        # Mean pooling over non-padding tokens
        if attention_mask is not None:
            mask = attention_mask.unsqueeze(-1).float()
            pooled = (last_hidden * mask).sum(dim=1) / mask.sum(dim=1)
        else:
            pooled = last_hidden.mean(dim=1)
        # Project to scalar reward
        reward = self.reward_head(pooled).squeeze(-1)
        return reward

reward_model = RewardModel(base_model)

# Create synthetic preference data
# For each prompt, we have two responses (A and B) and a label indicating which is better (1 = A is better, 0 = B is better)
prompts = [
    "What is 2+2?",
    "Explain gravity simply.",
    "Write a haiku about AI.",
]

responses_a = [
    "2+2 equals 4.",
    "Gravity makes things fall down.",
    "AI learns patterns.",
]

responses_b = [
    "I don't know.",
    "Gravity is a fundamental interaction that causes mutual attraction between masses.",
    "Beep boop I am a robot.",
]

# Labels: 1 means response_a is preferred, 0 means response_b is preferred
labels = [1, 1, 0]  # A is better for first two, B is better for third

# Tokenize pairs
def tokenize_pair(prompt, response):
    text = f"### Instruction:\n{prompt}\n\n### Response:\n{response}"
    return tokenizer(text, truncation=True, padding="max_length", max_length=128, return_tensors="pt")

# Training loop (very simplified)
optimizer = torch.optim.AdamW(reward_model.parameters(), lr=1e-5)
loss_fn = nn.BCEWithLogitsLoss()

print("Training reward model...")
for epoch in range(50):
    total_loss = 0
    for prompt, resp_a, resp_b, label in zip(prompts, responses_a, responses_b, labels):
        # Tokenize both responses
        tokens_a = tokenize_pair(prompt, resp_a)
        tokens_b = tokenize_pair(prompt, resp_b)
        
        # Get rewards
        reward_a = reward_model(tokens_a["input_ids"], tokens_a["attention_mask"])
        reward_b = reward_model(tokens_b["input_ids"], tokens_b["attention_mask"])
        
        # Preference loss: we want preferred response to have higher reward
        # If label=1 (A preferred), we want reward_a > reward_b
        # If label=0 (B preferred), we want reward_b > reward_a
        # We can frame this as: prefer_logits = reward_a - reward_b
        # Then target = 2*label - 1 (maps 1->1, 0->-1)
        prefer_logits = reward_a - reward_b
        target = 2 * torch.tensor(label, dtype=torch.float32) - 1  # 1->1, 0->-1
        
        # Loss: we want prefer_logits to be positive when A is preferred
        # Use a margin loss: max(0, -prefer_logits * target)
        loss = torch.max(torch.tensor(0.0), -prefer_logits * target)
        
        total_loss += loss
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Loss: {total_loss.item():.4f}")

print("Reward model training complete!")

# Test the reward model
print("\n--- Testing Reward Model ---")
test_prompt = "What is 2+2?"
test_responses = ["2+2 equals 4.", "I don't know.", "Maybe 5?"]
for resp in test_responses:
    tokens = tokenize_pair(test_prompt, resp)
    with torch.no_grad():
        reward = reward_model(tokens["input_ids"], tokens["attention_mask"])
    print(f"Response: '{resp}' -> Reward: {reward.item():.3f}")

When you run this, you’ll see that the reward model gives higher scores to responses that match our preferences. The correct answer “2+2 equals 4” gets a higher reward than “I don’t know.” This is the judge that will guide our PPO training.

Stage 3: Fine-Tuning with PPO — Optimizing the Policy Against the Reward Model

Now we get to the engine room. We use reinforcement learning to update the original SFT model so that it generates responses that score highly on the reward model — while not drifting too far from the original.

Let’s break down the key concepts:

• Policy: This is our language model. It takes a prompt and generates a response. In RL terms, it’s the thing we’re optimizing.
• Reward: This is the score from the reward model. It tells us how good the response is.
• PPO (Proximal Policy Optimization): This is the algorithm that updates the policy. It’s designed to make safe, incremental updates rather than big jumps that could break the model.

The PPO algorithm works like this:

1. Generate a response using the current policy.
2. Get the reward for that response from the reward model.
3. Calculate how much better or worse this response is compared to what we’d expect (the “advantage”).
4. Update the policy to make responses with positive advantage more likely, and responses with negative advantage less likely.
5. But here’s the key: we clip the update to prevent it from changing too much in one step.

Now here’s the crucial part: the KL penalty. We add a penalty if the model’s output distribution diverges too much from the original SFT model. This prevents the model from exploiting the reward model by generating gibberish that scores high.

The full loss looks like this:

Loss = -Reward_from_RM + β * KL_divergence

Where β controls how much we penalize divergence. Without this penalty, the model would quickly learn to generate text that the reward model likes but humans would find nonsensical.

Think of it this way: PPO is the engine. The KL penalty is the guardrails. Without it, the model would learn to cheat the reward model.

Let’s implement a simplified version of PPO using Hugging Face’s TRL library.

# Block 3: PPO Fine-Tuning with TRL
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from trl.core import respond_to_batch
import torch

# Load the SFT model (we'll simulate loading the one we trained earlier)
# In practice, you'd load from ./gpt2-sft-final
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# TRL requires a special wrapper that adds a value head (for the critic in PPO)
model = AutoModelForCausalLMWithValueHead.from_pretrained(model_name)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# We need a reward function. For this demo, we'll use a simple heuristic:
# Reward = length of response (longer = better) + 1 if it contains "helpful" or "good"
def reward_function(response_text):
    """Simple reward: longer responses with positive words get higher scores."""
    reward = len(response_text) / 100.0  # Normalize length
    if "helpful" in response_text.lower() or "good" in response_text.lower():
        reward += 1.0
    return reward

# Create a PPO config
config = PPOConfig(
    model_name=model_name,
    learning_rate=1e-5,
    batch_size=1,
    mini_batch_size=1,
    ppo_epochs=4,
    kl_penalty=0.1,  # The KL penalty coefficient (β)
    init_kl_coef=0.1,
    target_kl=0.1,
)

# Initialize PPOTrainer
ppo_trainer = PPOTrainer(
    config=config,
    model=model,
    tokenizer=tokenizer,
)

# Sample prompts for training
prompts = [
    "What is 2+2?",
    "Explain gravity simply.",
    "Write a haiku about AI.",
]

print("Starting PPO training...")
for epoch in range(3):
    for prompt in prompts:
        # Tokenize the prompt
        query_tensor = tokenizer.encode(prompt, return_tensors="pt")
        
        # Generate a response using the current policy
        response_tensor = ppo_trainer.generate(
            query_tensor,
            return_prompt=False,
            max_new_tokens=50,
            temperature=0.7,
        )
        
        # Decode the response
        response_text = tokenizer.decode(response_tensor[0], skip_special_tokens=True)
        
        # Get reward from our reward function
        reward = reward_function(response_text)
        
        # PPO step
        stats = ppo_trainer.step(
            [query_tensor[0]],  # List of query tensors
            [response_tensor[0]],  # List of response tensors
            [torch.tensor(reward, dtype=torch.float32)]  # List of rewards
        )
        
        print(f"Prompt: {prompt}")
        print(f"Response: {response_text}")
        print(f"Reward: {reward:.3f}")
        print(f"KL divergence: {stats['objective/kl']:.4f}")
        print("---")

print("PPO training complete!")

# Compare generations before and after
print("\n--- Comparing generations ---")
test_prompt = "What is machine learning?"
query_tensor = tokenizer.encode(test_prompt, return_tensors="pt")

# Before PPO (we don't have the original, but we can compare with a fresh generation)
response_tensor = ppo_trainer.generate(
    query_tensor,
    return_prompt=False,
    max_new_tokens=50,
    temperature=0.7,
)
response_text = tokenizer.decode(response_tensor[0], skip_special_tokens=True)
print(f"After PPO: {response_text}")

When you run this, you’ll see the reward increasing over time and the KL divergence staying bounded. The model is learning to generate responses that score higher on our reward function, but it’s not drifting too far from its original behavior.

Putting It All Together: The Complete RLHF Pipeline in Code

Now let’s combine all three stages into a single script. We’ll use a very small model (GPT-2 small) and a tiny dataset so the code runs in under 10 minutes on a laptop.

# Block 4: Complete RLHF Pipeline (SFT -> Reward Model -> PPO)
import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from datasets import Dataset
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
import numpy as np

print("=" * 50)
print("COMPLETE RLHF PIPELINE")
print("=" * 50)

# ============================================================
# STAGE 1: Supervised Fine-Tuning (SFT)
# ============================================================
print("\n--- Stage 1: SFT ---")

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Tiny instruction dataset
data = {
    "prompt": [
        "What is 2+2?",
        "Explain gravity simply.",
        "Write a haiku about AI.",
        "What is the capital of France?",
        "Tell me a joke."
    ],
    "response": [
        "2+2 equals 4.",
        "Gravity is a force that pulls objects toward each other. On Earth, it makes things fall down.",
        "Silicon dreams wake,\nLearning patterns in the dark,\nThinking without thought.",
        "The capital of France is Paris.",
        "Why did the AI cross the road? To optimize the chicken's path!"
    ]
}

def format_instruction(prompt, response):
    return f"### Instruction:\n{prompt}\n\n### Response:\n{response}"

formatted_texts = [format_instruction(p, r) for p, r in zip(data["prompt"], data["response"])]

def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=128)

dataset = Dataset.from_dict({"text": formatted_texts})
tokenized_dataset = dataset.map(tokenize_function, batched=True)

training_args = TrainingArguments(
    output_dir="./gpt2-sft",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    save_steps=10,
    logging_steps=5,
    learning_rate=5e-5,
    report_to="none",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
)

trainer.train()
print("SFT training complete!")

# Save SFT model
model.save_pretrained("./gpt2-sft-final")
tokenizer.save_pretrained("./gpt2-sft-final")

# ============================================================
# STAGE 2: Train Reward Model
# ============================================================
print("\n--- Stage 2: Reward Model ---")

# Load the SFT model as base for reward model
base_model = AutoModelForCausalLM.from_pretrained("./gpt2-sft-final")

class RewardModel(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.base_model = base_model
        self.reward_head = nn.Linear(base_model.config.hidden_size, 1)
        
    def forward(self, input_ids, attention_mask=None):
        outputs = self.base_model(input_ids, attention_mask=attention_mask, output_hidden_states=True)
        last_hidden = outputs.hidden_states[-1]
        if attention_mask is not None:
            mask = attention_mask.unsqueeze(-1).float()
            pooled = (last_hidden * mask).sum(dim=1) / mask.sum(dim=1)
        else:
            pooled = last_hidden.mean(dim=1)
        reward = self.reward_head(pooled).squeeze(-1)
        return reward

reward_model = RewardModel(base_model)

# Synthetic preference data
prompts = [
    "What is 2+2?",
    "Explain gravity simply.",
    "Write a haiku about AI.",
]

responses_a = [
    "2+2 equals 4.",
    "Gravity makes things fall down.",
    "AI learns patterns.",
]

responses_b = [
    "I don't know.",
    "Gravity is a fundamental interaction that causes mutual attraction between masses.",
    "Beep boop I am a robot.",
]

labels = [1, 1, 0]

def tokenize_pair(prompt, response):
    text = f"### Instruction:\n{prompt}\n\n### Response:\n{response}"
    return tokenizer(text, truncation=True, padding="max_length", max_length=128, return_tensors="pt")

optimizer = torch.optim.AdamW(reward_model.parameters(), lr=1e-5)

print("Training reward model...")
for epoch in range(50):
    total_loss = 0
    for prompt, resp_a, resp_b, label in zip(prompts, responses_a, responses_b, labels):
        tokens_a = tokenize_pair(prompt, resp_a)
        tokens_b = tokenize_pair(prompt, resp_b)
        
        reward_a = reward_model(tokens_a["input_ids"], tokens_a["attention_mask"])
        reward_b = reward_model(tokens_b["input_ids"], tokens_b["attention_mask"])
        
        prefer_logits = reward_a - reward_b
        target = 2 * torch.tensor(label, dtype=torch.float32) - 1
        
        loss = torch.max(torch.tensor(0.0), -prefer_logits * target)
        total_loss += loss
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Loss: {total_loss.item():.4f}")

print("Reward model training complete!")

# Save reward model
# In practice, you'd save the state dict. For simplicity, we'll keep it in memory.

# ============================================================
# STAGE 3: PPO Fine-Tuning
# ============================================================
print("\n--- Stage 3: PPO ---")

# Load SFT model with value head for PPO
model = AutoModelForCausalLMWithValueHead.from_pretrained("./gpt2-sft-final")

# Reward function using our trained reward model
def get_reward(prompt, response):
    tokens = tokenize_pair(prompt, response)
    with torch.no_grad():
        reward = reward_model(tokens["input_ids"], tokens["attention_mask"])
    return reward.item()

config = PPOConfig(
    model_name="./gpt2-sft-final",
    learning_rate=1e-5,
    batch_size=1,
    mini_batch_size=1,
    ppo_epochs=4,
    kl_penalty=0.1,
    init_kl_coef=0.1,
    target_kl=0.1,
)

ppo_trainer = PPOTrainer(
    config=config,
    model=model,
    tokenizer=tokenizer,
)

prompts = [
    "What is 2+2?",
    "Explain gravity simply.",
    "Write a haiku about AI.",
]

print("Starting PPO training...")
for epoch in range(3):
    epoch_rewards = []
    epoch_kls = []
    for prompt in prompts:
        query_tensor = tokenizer.encode(prompt, return_tensors="pt")
        
        response_tensor = ppo_trainer.generate(
            query_tensor,
            return_prompt=False,
            max_new_tokens=50,
            temperature=0.7,
        )
        
        response_text = tokenizer.decode(response_tensor[0], skip_special_tokens=True)
        reward = get_reward(prompt, response_text)
        
        stats = ppo_trainer.step(
            [query_tensor[0]],
            [response_tensor[0]],
            [torch.tensor(reward, dtype=torch.float32)]
        )
        
        epoch_rewards.append(reward)
        epoch_kls.append(stats['objective/kl'])
        
        print(f"Prompt: {prompt}")
        print(f"Response: {response_text}")
        print(f"Reward: {reward:.3f}")
        print(f"KL: {stats['objective/kl']:.4f}")
        print("---")
    
    print(f"Epoch {epoch}: Avg Reward = {np.mean(epoch_rewards):.3f}, Avg KL = {np.mean(epoch_kls):.4f}")

print("\nRLHF pipeline complete!")

# ============================================================
# COMPARISON: Base vs SFT vs RLHF
# ============================================================
print("\n" + "=" * 50)
print("COMPARISON: Base vs SFT vs RLHF")
print("=" * 50)

# Load base model for comparison
base_model = AutoModelForCausalLM.from_pretrained(model_name)
base_tokenizer = AutoTokenizer.from_pretrained(model_name)
if base_tokenizer.pad_token is None:
    base_tokenizer.pad_token = base_tokenizer.eos_token

test_prompt = "What is the meaning of life?"
input_text = f"### Instruction:\n{test_prompt}\n\n### Response:\n"

print(f"\nTest prompt: {test_prompt}")

# Base model
inputs = base_tokenizer(input_text, return_tensors="pt")
with torch.no_grad():
    outputs = base_model.generate(**inputs, max_new_tokens=50, temperature=0.7)
print(f"\nBase model: {base_tokenizer.decode(outputs[0], skip_special_tokens=True)}")

# SFT model (we'll use the PPO model's base for comparison)
# Note: In a real comparison, you'd load the SFT model separately
# For this demo, we'll note that the PPO model started from SFT
print(f"\n(Note: SFT model was the starting point for PPO)")

# RLHF model (after PPO)
query_tensor = tokenizer.encode(test_prompt, return_tensors="pt")
response_tensor = ppo_trainer.generate(
    query_tensor,
    return_prompt=False,
    max_new_tokens=50,
    temperature=0.7,
)
response_text = tokenizer.decode(response_tensor[0], skip_special_tokens=True)
print(f"\nAfter RLHF: {response_text}")

When you run this complete pipeline, you’ll see the reward score increase over PPO epochs while the KL divergence stays bounded. The SFT model learned the format of a helpful response. The RLHF model learned to prefer responses that score higher on our reward function. The difference is subtle but crucial.

What RLHF Actually Changes (and Doesn’t Change)

Now let’s be honest about what RLHF accomplishes — and what it doesn’t.

What RLHF changes:

• Output style: The model becomes more polite, more helpful in tone, and better at following instructions.
• Refusal rates: The model learns to refuse harmful requests.
• Helpfulness: Responses are more on-topic and directly address the user’s intent.
• Reduction of harmful content: The model is less likely to produce toxic or dangerous outputs.

What RLHF does NOT change:

• Factual accuracy: This is the big one. RLHF can actually increase hallucination because the model learns to be more confident and elaborate, even when it doesn’t know the answer. As Chip Huyen points out in her blog, “RLHF can make a model more sycophantic — it learns to tell users what they want to hear, not the truth.”
• Reasoning ability: RLHF doesn’t make the model smarter. It doesn’t improve its ability to reason through complex problems.
• Knowledge cutoff: The model still only knows what it was trained on.

The InstructGPT paper itself showed this clearly: the 1.3B parameter RLHF model was preferred by human labelers over the 175B parameter base model. But it still made up facts. The model was more polite, but not more correct.

Think of it this way: RLHF is a preference optimizer, not a truth optimizer. It makes the model more polite, not more correct. It’s like teaching a student to give the right answer on a multiple-choice test — they learn to pick what the teacher wants, not necessarily what’s true.

Recap and What’s Next

Let’s recap what we’ve learned:

1. SFT (Supervised Fine-Tuning): Teaches the model the format of a helpful response through imitation learning. The model learns to copy good examples.

2. Reward Model: Teaches a judge what “good” looks like by learning from human preferences. This is the hardest part because the reward model is a proxy for human judgment — it’s never perfect.

3. PPO (Proximal Policy Optimization): Optimizes the policy against the reward model with guardrails (KL penalty) to prevent exploitation. This is the engine that drives alignment.

Key takeaway: RLHF is the reason ChatGPT feels like a helpful assistant rather than a text autocomplete. But it’s not magic — it’s three carefully designed training stages. And it’s not a silver bullet — it optimizes for preference, not truth.

In the next part of this series, we’ll build a reward model from scratch and use it to align a small agent for a specific task. You’ll see how to design custom reward functions that capture exactly what you want your agent to do.

Check Your Understanding

Remember: What are the three stages of RLHF?

Understand: Why is SFT alone insufficient for alignment? What does it miss?

Apply: Given a dataset of human preferences (pairwise comparisons of responses), how would you train a reward model? Write the loss function in plain English.

Analyze: What happens if the KL penalty is set to zero during PPO fine-tuning? What would the model learn to do?

Evaluate: Is a model trained with RLHF guaranteed to be more truthful? Why or why not? Give an example from the InstructGPT paper.

Create: Design a small experiment to test whether RLHF reduces or increases hallucination on a specific domain (e.g., medical advice). What would you measure? How would you set up the comparison?

Related articles

• Part 5: Agentic RAG: Retrieval That an Agent Decides to Use — This article introduced the idea of giving an LLM control over its own retrieval process, which is a key capability for building agents that need to gather information before responding.
• Part 4: Multi-Agent Systems: When One LLM Isn’t Enough — This article explored how multiple agents can collaborate, which is relevant when you need different reward models for different aspects of alignment.