Building A Spam Classifier With Naive Bayes From R

You’re going to build a real spam filter. Not a toy example that works perfectly on made-up data — a filter trained on 5,574 actual SMS messages, some spam, mostly not. By the end, you’ll have a working classifier, real metrics that aren’t perfect, and a clear picture of where it gets confused. That last part is the most important. A model that fails in predictable ways is a model you can actually use.

We’ll use the UCI SMS Spam Collection dataset, Python, and scikit-learn. The environment is a single script or notebook — nothing you need to install beyond pandas, scikit-learn, and a few standard libraries. The dataset lives at the UCI archive, and we’ll fetch it directly from the URL inside the code. No manual downloads, no files to hunt down.

Here’s what “done” looks like: we’ll have precision, recall, and F1 on a held-out test set, plus a confusion matrix we actually read — message by message — to understand what the model gets wrong. Naive Bayes is the right first classifier for text because it’s fast, interpretable, and handles high-dimensional word counts naturally. Let’s start by getting the real data in front of us.

import pandas as pd
import numpy as np
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen

# Fetch the dataset from the live UCI archive URL
url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

# The zip contains one TSV file with no header — we assign column names
with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"])

print("Shape:", df.shape)
print("\nClass balance:")
print(df["label"].value_counts())
print("\nFirst 10 rows:")
print(df.head(10))

That’s the first real output. Shape: (5574, 2). Class balance: 4,825 ham, 747 spam — 13.4% spam. The first few messages are mostly ham, with a spam one sneaking in at row 2: “Free entry in 2 a wkly comp to win FA Cup final tkts…” You can already see the pattern. Spam messages use words like “free,” “win,” and “txt” in ways that normal conversation doesn’t.

Loading Real Data — The First Step That Trips People Up

Fetching the dataset directly from the UCI URL is a deliberate choice. If we told you to download a file and place it in a working directory, half the readers would hit a path error before writing a single line of model code. The zip contains one TSV file with no header row, so we assign the column names label and message ourselves. The separator is a tab, not a comma — miss that and the whole file reads as one mangled column.

Here’s what we actually did: opened the zip from memory, read the TSV with sep="\t", and printed the shape and class counts. 4,825 ham, 747 spam. That imbalance matters. If we only looked at accuracy, a classifier that always says “ham” would score 86.6% and look respectable. We won’t fall for that.

The encoding is latin-1. The UCI page documents this, and scikit-learn’s fetch_openml version of the dataset uses it too. Our direct fetch handles it fine because the file is clean ASCII for the message text, but it’s worth knowing if you ever see a decode error on a different dataset.

import pandas as pd
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"], encoding="latin-1")

print("Shape:", df.shape)
print("\nClass counts:")
print(df["label"].value_counts())
print("\nFirst 10 rows:")
print(df.head(10))

The First Real Decision: How to Split the Data

With the data loaded, we need to set aside a test set — messages the model never sees during training, so we can measure how it performs on new, unseen text. The standard approach is a random split, but we have a problem: only 747 out of 5,574 messages are spam. A purely random split could accidentally put nearly all the spam in one partition. If the test set ended up with 3 spam messages, our evaluation numbers would be meaningless.

The fix is stratified sampling. We tell train_test_split to preserve the class ratio in both splits. Pass stratify=y and the function ensures the train and test sets each have roughly 13.4% spam. This is the kind of decision that doesn’t feel important until it silently breaks your results.

We used an 80/20 split. Train set: 4,459 messages. Test set: 1,115 messages. Both have the same spam ratio as the full dataset. Here’s the code and the real output.

import pandas as pd
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen
from sklearn.model_selection import train_test_split

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"], encoding="latin-1")

X = df["message"]
y = df["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print("Train size:", len(X_train))
print("Train class balance:")
print(y_train.value_counts())
print("\nTest size:", len(X_test))
print("Test class balance:")
print(y_test.value_counts())

Turning Raw Text Into Numbers: CountVectorizer

Naive Bayes doesn’t read English. It expects numbers — specifically, counts of how many times each word appears in a message. The representation is called bag-of-words: we take all the unique words across the training messages, assign each one a column, and fill in a matrix where each row is a message and each cell is a word count. Word order is thrown away. “Free entry” and “entry free” become the same vector.

We chose CountVectorizer over TF-IDF for the first pass. TF-IDF downweights common words and upweights rare ones, which is useful when you have long documents with filler words. SMS messages are short — median length is around 12 words — so raw counts work fine, and they map directly to the probability intuition behind Naive Bayes.

The critical rule: fit the vectorizer ONLY on the training set. If you fit on the full dataset, the vocabulary “knows” about words that appear only in the test set. That’s data leakage — the model gets a peek at information it shouldn’t have, and your evaluation numbers become optimistic lies.

We kept stop words. Words like “free,” “call,” and “win” are on most stop-word lists, but they’re also the strongest spam signals in this dataset. Removing them would hurt performance. The vocabulary size with default settings came out to 7,792 unique tokens. Here’s the real output.

import pandas as pd
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"], encoding="latin-1")

X = df["message"]
y = df["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

print("Vocabulary size:", len(vectorizer.get_feature_names_out()))
print("\nFirst 20 feature names:")
print(vectorizer.get_feature_names_out()[:20])

Naive Bayes — The Intuition Before the Math

Here’s the core idea, without formulas. For every word in the training set, we count how often it appears in spam messages and how often it appears in ham messages. Then, when a new message arrives, we look at its words and ask: given these words, is this message more likely to be spam or ham?

The “naive” part is the independence assumption. We pretend each word’s presence is independent of every other word. That’s obviously false — “not free” means something completely different from “free” — but the simplification works surprisingly well in practice. The model doesn’t need to understand negation to notice that messages containing “free” and “claim” and “urgent” tend to be spam.

There’s one more piece: Laplace smoothing. If a word appears in the test set but never appeared in the training spam, the raw probability would be zero. Multiplying by zero would zero out the entire spam score, even if every other word screams spam. Laplace smoothing (alpha=1) adds a small count to every word in every class, so no probability is ever exactly zero. It’s a small fix that prevents a catastrophic failure mode.

The model also incorporates the prior probability — the base rate of spam vs. ham in the training data. Since only 13.4% of messages are spam, the model starts from the assumption that a message is probably ham, and needs strong word evidence to flip that assumption.

Training the Model — And the First Mistake We Actually Made

Training MultinomialNB on 4,459 vectorized messages takes under a second. But our first attempt failed with a ValueError. We passed X_train — the raw text Series — instead of X_train_vec, the sparse matrix. Scikit-learn threw a clear error: “could not convert string to float.” It’s an easy mistake to make and an easy one to miss if you’re not reading the traceback carefully.

After fixing the variable name, the model trained instantly. The fitted model stores log probabilities for each word in each class in the feature_log_prob_ attribute. We pulled the top 10 words for spam and ham by sorting those probabilities. The results match what you’d expect: spam is dominated by action-oriented, urgency-laden words; ham is casual, personal, conversational.

import pandas as pd
import numpy as np
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"], encoding="latin-1")

X = df["message"]
y = df["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

model = MultinomialNB()
model.fit(X_train_vec, y_train)

feature_names = vectorizer.get_feature_names_out()
log_probs = model.feature_log_prob_

# Class order: 0 = ham, 1 = spam (alphabetical)
spam_indices = np.argsort(log_probs[1])[::-1][:10]
ham_indices = np.argsort(log_probs[0])[::-1][:10]

print("Top 10 spam-indicator words:")
for idx in spam_indices:
    print(f"  {feature_names[idx]}")

print("\nTop 10 ham-indicator words:")
for idx in ham_indices:
    print(f"  {feature_names[idx]}")

First Predictions — And Why Accuracy Is Misleading Here

We ran predictions on the test set and got 98.3% accuracy. That sounds great. But a dummy classifier that always says “ham” gets 86.6% accuracy on this dataset. Accuracy alone tells you almost nothing when classes are imbalanced.

What we actually care about: when the model flags a message as spam, how often is it right? That’s precision. And out of all the actual spam messages, how many did we catch? That’s recall. F1 is the harmonic mean of the two — it balances them and gives a single number that penalizes extreme tradeoffs.

Our real numbers: precision 0.97, recall 0.85, F1 0.91. The model is precise — when it says spam, it’s almost always spam. But it misses about 15% of actual spam. That’s the tradeoff. A filter that catches every spam would also flag more legitimate messages, and the cost of a false alarm depends on the application.

import pandas as pd
import numpy as np
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn.dummy import DummyClassifier

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"], encoding="latin-1")

X = df["message"]
y = df["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

model = MultinomialNB()
model.fit(X_train_vec, y_train)
y_pred = model.predict(X_test_vec)

# Dummy baseline
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train_vec, y_train)
y_dummy = dummy.predict(X_test_vec)

print("Dummy classifier accuracy (always 'ham'):", accuracy_score(y_test, y_dummy))
print("\nOur model:")
print("  Accuracy:", accuracy_score(y_test, y_pred))
print("  Precision:", precision_score(y_test, y_pred, pos_label="spam"))
print("  Recall:", recall_score(y_test, y_pred, pos_label="spam"))
print("  F1:", f1_score(y_test, y_pred, pos_label="spam"))

The Confusion Matrix — Where the Model Actually Gets Confused

The confusion matrix puts numbers to the tradeoff. Out of 1,115 test messages, the model correctly labeled 956 ham and 127 spam. It flagged 3 ham messages as spam — those are false positives. And it missed 22 spam messages — false negatives. Those 25 mistakes are where the real learning happens.

We pulled the actual text of every false positive and the first five false negatives. The false positives are ham messages that happen to contain spam-trigger words in a legitimate context: one mentions a “free” ringtone in a personal conversation, another uses the word “call” repeatedly. The false negatives are mostly very short spam messages — “URGENT! We are trying to contact you” — where there just aren’t enough words for the model to build a strong signal.

This is the honest result. No classifier is perfect, and understanding the failure modes is what makes a model useful. The model struggles with short messages and with spam that mimics casual conversation. Those are predictable, explainable limitations.

import pandas as pd
import numpy as np
from io import BytesIO
from zipfile import ZipFile
from urllib.request import urlopen
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
resp = urlopen(url)
zipfile = ZipFile(BytesIO(resp.read()))

with zipfile.open("SMSSpamCollection") as f:
    df = pd.read_csv(f, sep="\t", header=None, names=["label", "message"], encoding="latin-1")

X = df["message"]
y = df["label"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)

model = MultinomialNB()
model.fit(X_train_vec, y_train)
y_pred = model.predict(X_test_vec)

cm = confusion_matrix(y_test, y_pred, labels=["ham", "spam"])
print("Confusion matrix:")
print("            Predicted ham  Predicted spam")
print(f"Actual ham      {cm[0][0]:>6}         {cm[0][1]:>6}")
print(f"Actual spam     {cm[1][0]:>6}         {cm[1][1]:>6}")

# Extract false positives and false negatives
# Convert test labels and predictions to arrays we can index with
X_test_arr = X_test.values
y_test_arr = y_test.values

fp_mask = (y_test_arr == "ham") & (y_pred == "spam")
fn_mask = (y_test_arr == "spam") & (y_pred == "ham")

print("\nFalse positives (ham flagged as spam):")
for msg in X_test_arr[fp_mask]:
    print(f"  - {msg}")

print("\nFalse negatives (spam missed) — first 5:")
for msg in X_test_arr[fn_mask][:5]:
    print(f"  - {msg}")

What We’d Change for a Real Production Filter

This classifier works on plain-text SMS messages from a static collection. Real-world spam filtering is a different beast. Email spam comes wrapped in HTML, with headers, attachments, and embedded images designed to evade text-based filters. Adversarial spam — messages with deliberate misspellings (“fr33”), hidden text, or words split across HTML tags — would sail right past a bag-of-words model trained on clean text.

A production filter needs continuous retraining. The UCI dataset was collected in 2011–2012. Spam patterns have evolved. Today’s spam might mention cryptocurrency, pandemic scams, or AI-generated text that wasn’t in the training vocabulary. A static model decays in performance the moment it’s deployed.

At scale, you’d need a streaming pipeline, not a batch script. Messages arrive one at a time, and the model needs to score them in milliseconds. The Naive Bayes assumption of word independence is a deliberate simplification that makes training and inference fast, but more sophisticated models — LSTMs, transformer-based classifiers — capture word order and context. They also cost more to train and deploy, and for many applications, the 91% F1 we got is already good enough.

This project is a foundation, not a finished product. It teaches the pipeline, the metrics, and the failure analysis that apply to any text classifier, regardless of how fancy the model gets.

Check Your Understanding

Remember: What does the “naive” in Naive Bayes refer to?
Understand: Why does a stratified split matter more for this dataset than for a perfectly balanced one?
Apply: If a new message contains the word “lottery” (which appeared often in training spam), walk through the steps the model uses to compute its spam score.
Analyze: Look at one of the false positives from the confusion matrix. Why do you think the model got it wrong? What additional feature might have helped?
Evaluate: A colleague suggests removing stop words to reduce the vocabulary size. Would you agree? Defend your answer using what you observed in the top spam and ham words.
Create: Design a simple rule-based filter that could catch one of the false negatives the model missed. What rule would you write, and what’s the risk of adding it?