Borrowing Brains: A Beginner's Guide to Transfer Learning and Fine-Tuning

In our last few chapters, Lina has been building the Transformer from the ground up. We looked at how multi-head attention gives the model “eyes” and how positional encoding gives it a “GPS.” But try training one of these massive models from scratch on your laptop. You’d run out of battery—and patience—long before the model learned to say “Hello.”

Training a frontier language model like GPT-4 from scratch costs millions of dollars in electricity and hardware. Even a far more modest vision backbone like ResNet costs thousands of GPU-hours to train from scratch — days of compute you don’t have sitting around on a laptop. The good news? You don’t have to. Today we’ll learn how to “borrow” a brain that’s already been trained by a tech giant and tweak it for our own needs. This is called Transfer Learning.

The Store Manager’s Secret: Why We Don’t Start from Scratch

Lina wants to build a system that verifies book-cover photos in BookSight’s product listings—the same photo-verification idea from Part 4. She has two choices. She can teach a model what an edge is, what a texture is, what a book even looks like—teach it to see from zero. Or she can start with a model that’s already spent thousands of GPU-hours learning to see from millions of photos, and just teach it the one thing it doesn’t know yet: what her book covers look like.

Which one gets her feature live by Friday?

“Starting from scratch” in AI means starting with random weights—the model guesses blindly. Transfer learning means starting with a model like BERT or ResNet that has already spent thousands of GPU hours looking at millions of images or books. It already knows what a circle is, how grammar works, how to tell a book from a car.

Here’s the difference in setup time between starting from scratch and borrowing a brain, in Python.

import torch
import torchvision.models as models
import time

# Scenario A: Starting from scratch (Random weights)
start = time.time()
scratch_model = models.resnet18(weights=None)
print(f"Scratch model initialized in {time.time() - start:.4f} seconds.")

# Scenario B: Borrowing a brain (Pretrained weights)
start = time.time()
pretrained_model = models.resnet18(weights='IMAGENET1K_V1')
print(f"Pretrained model loaded in {time.time() - start:.4f} seconds.")

• import torch — Imports PyTorch, the framework powering every model in this series.
• import torchvision.models as models — Provides access to pretrained vision architectures (ResNet, VGG, MobileNet, etc.) bundled with PyTorch.
• import time — Standard Python module used here to measure how long each model takes to load.
• models.resnet18(weights=None) — Creates a ResNet18 network with randomly initialized weights. The architecture is there, but the weights know nothing — every convolutional filter starts as random noise.
• models.resnet18(weights='IMAGENET1K_V1') — Creates the exact same ResNet18 architecture, but loads weights that were trained on ImageNet (1.2 million images across 1,000 categories). The model arrives already knowing how to detect edges, textures, shapes, and object compositions.
• time.time() - start — Measures elapsed wall-clock time for each initialization. Both are fast (seconds), but the pretrained model’s “knowledge” took thousands of GPU-hours to produce — you’re just downloading it, not re-learning it.

What this actually means: Both models load fast. The pretrained_model arrives with knowledge already packed into its weights. The scratch model is an empty vessel. To match the pretrained model, you’d need to feed the scratch one 1.2 million images and wait days. With transfer learning, we start at the finish line.

Training from Scratch vs. Transfer Learning: Which Should You Reach For?

Approach	What it does	Best for	Tradeoff
Transfer learning (pretrained backbone + new head)	Start with weights trained on millions of images or billions of text tokens; swap the final layer; freeze, then fine-tune.	Most real-world tasks where the data type (images, text, audio) matches a known pretrained model. Fast, cheap, and usually accurate.	The backbone’s biases come with it. If your domain is truly novel (e.g., thermal imaging, spectrograms), the pretrained features may be unhelpful or even harmful.
Training from scratch	Start with random weights; learn every layer’s weights from your data alone.	Truly novel data types or architectures with no good pretrained match; research where you need to understand every layer’s behavior from the ground up.	Enormous data and compute requirements. Days or weeks of training vs. minutes. Often lower final accuracy because the model must rediscover basic features (edges, grammar) that a pretrained model already knows.

Lina’s takeaway for BookSight: She’s verifying book-cover photos — the same kind of image-classification task that ImageNet models already excel at. A pretrained ResNet backbone gives her high accuracy in minutes, not days. She’d only consider training from scratch if her images were in a fundamentally different modality (e.g., infrared scans of book bindings) that no pretrained model has ever seen.

The Anatomy of a Pretrained Model

Every deep learning model breaks into two parts: the Body and the Head.

1. The Body (The Backbone): The early and middle layers. In a vision model, these layers detect lines, textures, and shapes. In a language model, they capture word relationships and context. Think of this as the model’s “general knowledge.”
2. The Head (The Classifier): The very last layer. It takes that general knowledge and makes a specific guess — “This is a Golden Retriever” or “This sentence is angry.”

Transfer learning keeps the Body as-is. We chop off the old Head and attach a new one suited to the task.

The key insight: we aren’t changing the model’s eyes. We’re changing what it does with what it sees. Here’s how that looks when swapping the head of a ResNet model in code.

import torch.nn as nn

# Load a model trained to recognize 1,000 types of objects
model = models.resnet18(weights='IMAGENET1K_V1')

# Look at the old 'head' (the fc layer)
print(f"Old head output features: {model.fc.out_features}") # Usually 1000

# We only have 2 classes: 'Sneaker' or 'Not Sneaker'
model.fc = nn.Linear(in_features=512, out_features=2)

print(f"New head output features: {model.fc.out_features}")

• import torch.nn as nn — Imports PyTorch’s neural-network module API, which provides layer building blocks like nn.Linear, nn.Conv2d, and nn.Sequential.
• model.fc.out_features — Inspects the old classification head’s output width. For a ResNet18 trained on ImageNet, this is 1,000 (one logit per ImageNet class).
• model.fc = nn.Linear(in_features=512, out_features=2) — Replaces the old 1,000-class head with a brand-new nn.Linear layer. in_features=512 must match the backbone’s output width (ResNet18 produces 512-dimensional feature vectors). out_features=2 matches the number of classes in the new task.
• The old head’s weights are discarded entirely — the new nn.Linear is initialized with random weights and will be trained from scratch on the new task’s data.

Interpretation: We kept the 512 complex features the model uses to understand images, but replaced the final decision-maker. Instead of choosing between 1,000 categories, it now chooses between our 2.

Step 1: The ‘Frozen’ Phase (Feature Extraction)

The new head has random weights. Start training right now, and backpropagation will send huge error signals through the whole model — which can ruin the pretrained weights in the body. It’s like a student stumbling through a new dance and knocking over the teacher.

So we freeze the body. The instruction to Python: “Only update the weights in the new head. Leave the rest alone.”

# Freeze all layers in the 'Body'
for param in model.parameters():
    param.requires_grad = False

# Unfreeze ONLY the new head we just created
for param in model.fc.parameters():
    param.requires_grad = True

# Let's check if it worked
print(f"Body frozen? {not model.layer1[0].conv1.weight.requires_grad}")
print(f"Head unfrozen? {model.fc.weight.requires_grad}")

• for param in model.parameters(): — Iterates over every learnable parameter (every weight and bias) in the entire model — backbone and head alike.
• param.requires_grad = False — Sets the requires_grad flag to False, telling PyTorch’s autograd engine to skip gradient computation for this parameter. During backpropagation, no gradients will be computed or stored for these weights, so the optimizer won’t update them.
• for param in model.fc.parameters(): — Iterates only over the parameters of the newly created fc head (the nn.Linear(512, 2) layer from the previous code block).
• param.requires_grad = True — Re-enables gradients for the head’s weights only. Now backpropagation will compute gradients for the head but not for the frozen body.
• model.layer1[0].conv1.weight.requires_grad — Checks a specific weight inside layer1 (an early convolutional block in the ResNet backbone). After freezing, this should be False, confirming the body is frozen.
• model.fc.weight.requires_grad — Checks the head’s weight tensor. This should be True, confirming only the head is trainable.

What this actually means: During training, the model uses its existing “eyes” to look at your data. It won’t change how it sees shapes or colors. It only learns to map those shapes to your new labels. Fast and stable.

Step 2: The ‘Fine-Tuning’ Phase (The Gentle Nudge)

Once your new head has learned the basics and accuracy looks decent, you can move on to Fine-Tuning. Here you unfreeze the whole model and train it again—but with a catch.

The Golden Rule: Use a tiny learning rate. Think of it as a gentle nudge. We want to wiggle the body’s weights just enough that they specialize on your specific data—say, the cover style of a particular book edition—without forgetting how to see lines and edges.

# 1. Unfreeze everything
for param in model.parameters():
    param.requires_grad = True

# 2. Use a tiny learning rate (10x to 100x smaller than usual)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

print("Model is now in Fine-Tuning mode with a very small learning rate.")

• for param in model.parameters(): param.requires_grad = True — Re-enables gradients for every parameter in the model, including the backbone. Now the optimizer can update all layers.
• lr=1e-5 — Sets the learning rate to 0.00001, which is 100× smaller than the typical 1e-3 used for training from scratch. This tiny rate ensures that each gradient step only nudges the weights slightly, preserving the general visual features the backbone already learned.
• torch.optim.Adam(model.parameters(), lr=1e-5) — Passes ALL of the model’s parameters (not just the head) to the optimizer. Since everything is unfrozen, all layers will receive gradient updates — but the small learning rate keeps those updates gentle.

Check it against the data: Use a large learning rate here and your accuracy will often plummet. This is called “catastrophic forgetting.” The model’s brain gets scrambled by the new information.

Vision vs. Language: Is it the same thing?

Vision or Language, the workflow is identical.

• Vision: Use torchvision. The body sees shapes; the head identifies objects.
• Language: Use HuggingFace Transformers. The body sees grammar and context; the head identifies sentiment or topics.

The code for a Language Model (BERT) looks nearly the same:

# Conceptual comparison - The patterns are the same!
# Vision: model = models.resnet18(pretrained=True)
# Language: 
# from transformers import AutoModelForSequenceClassification
# model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

• This block is entirely commented out (# at the start of every line) — it’s a conceptual comparison, not runnable code.
• models.resnet18(pretrained=True) — The vision side: load a ResNet with pretrained ImageNet weights. (Note: the modern PyTorch API uses weights='IMAGENET1K_V1' instead of pretrained=True, as shown in the earlier code blocks.)
• AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2) — The language side: HuggingFace’s from_pretrained method loads a BERT model trained on masked-language-modeling over Wikipedia + BookCorpus, then automatically replaces the classification head with a new one matching num_labels=2. The body/head split and the swap-the-head pattern are identical to the vision workflow.

Here’s the catch: Language models often need more fine-tuning than vision models. Human language is specific to its domain. A BERT model trained on Wikipedia might need a good “nudge” to understand legal contracts or medical journals.

Wrap-Up: Your New Superpower

You don’t need a supercomputer to build world-class AI. Borrow a brain and stand on the shoulders of giants.

Here’s the workflow:

1. Pick a Pretrained Model: Find a “body” that matches your data type (images or text).
2. Swap the Head: Replace the final layer with one that matches your number of classes.
3. Freeze and Train: Train only the head to get a stable starting point.
4. Fine-Tune: Unfreeze the body and use a tiny learning rate to polish the results.

Interpret the result: Follow this and you can often hit 90%+ accuracy on a custom task in just a few minutes of training. That’s a borrowed brain, working.

Lina’s borrowed-brain model is now accurate and cheap to train. But as she pushes it further on BookSight’s data, she starts seeing training instability and overfitting. In the next part, we’ll address both with Batch Normalization and Dropout—regularization techniques that keep training stable when it gets noisy.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the “Body” and the “Head” of a pretrained model, and which one do you typically replace for a new task?

Understand In your own words, explain why we freeze the Body during the first phase of training instead of training the whole model at once from the start.

Apply The article says fine-tuning uses a learning rate “10x to 100x smaller than usual.” If a typical training learning rate is 1e-3, what range of learning rates does that recommendation produce, and does the article’s own example value of 1e-5 fall inside that range?

Analyze The article swaps the ResNet’s head from 1,000 output classes down to 2 (Sneaker / Not Sneaker) while keeping the same 512 input features. Walk through why keeping those 512 features unchanged is what makes the new head trainable with so little data, compared to training a 512-to-2 classifier from random image features.

Evaluate The article claims “catastrophic forgetting” happens if you fine-tune with too large a learning rate. Critique this warning: what’s the tradeoff a team faces if they’re overly cautious and use an extremely tiny learning rate instead—what do they risk losing by playing it too safe?

Create Design a transfer-learning plan for a new sneaker-marketplace task not covered in the article: detecting counterfeit logos in product photos. Specify which pretrained model type you’d start from, what you’d replace in the Head, and whether you’d freeze-then-fine-tune or fine-tune from the start, with a reason for your choice.

---

Related articles

• Building a Miniature Transformer From Scratch: The ‘Lego’ Approach to Deep Learning)
• Batch Normalization and Dropout: The Regularization Duo Every Model Needs)

References & Further reading

• Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). “How transferable are features in deep neural networks?” Advances in Neural Information Processing Systems (NeurIPS 2014). — The seminal study on transfer learning that systematically measured how well features learned on one task transfer to another, showing that early layers generalize broadly while later layers specialize — the empirical foundation for the freeze-the-body, swap-the-head strategy used in this article.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). “Deep Residual Learning for Image Recognition.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016). — Introduced the ResNet architecture used throughout this article’s transfer-learning examples, along with the residual connections that make deep backbones trainable and their pretrained weights widely available in libraries like torchvision.
• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” arXiv preprint arXiv:1810.04805. — Introduced BERT, the language model referenced in the article’s vision-vs-language comparison, demonstrating that pretraining on large unlabeled corpora followed by task-specific fine-tuning produces state-of-the-art results across dozens of NLP benchmarks.
• PyTorch Documentation: torchvision.models, Transfer Learning Tutorial — Official docs and tutorial for loading pretrained models and implementing the freeze-and-fine-tune workflow shown in this article.

---