Chi-Square Tests: When Your Data Is Categorical, Not Continuous

1. Opening: The Coffee Shop Problem

Imagine you run a cozy little coffee shop. Every day, you track two things: what day of the week it is, and whether each customer orders a hot or an iced coffee. After a month, you have a pile of data. You look at it and wonder: Does the day of the week actually affect what people drink? Maybe weekends are for iced lattes, and weekdays are for hot drip coffee.

Your first instinct might be to run a t-test. But wait — can you? A t-test compares the average of a continuous measurement (like temperature or height) between two groups. Your data isn’t like that. You have counts: how many hot coffees on Monday, how many iced coffees on Saturday. There’s no average to compare. You have buckets, not a number line.

So what do you do? You reach for a chi-square test of independence. It’s the perfect tool for this job. You’ll learn exactly when and how to use it, and — more importantly — how to interpret the results in plain English.

2. What Is Categorical Data, and Why Does It Need Its Own Test?

Categorical data is data that falls into buckets. There are two main types:

• Nominal: categories with no natural order (e.g., drink type: hot or iced; day of week: Monday, Tuesday, …).
• Ordinal: categories with a natural order (e.g., cup size: small, medium, large).

This is different from continuous data like height, time, or temperature, where you can measure an average and a standard deviation.

With categorical data, the fundamental question is: Are the observed counts different from what we’d expect if there were no relationship? In other words, if the day of the week didn’t matter, how many iced coffees would we expect to sell each day? That’s the key idea behind the chi-square test.

Let’s load our coffee shop data and see what it looks like.

import pandas as pd
import numpy as np

# Create a realistic coffee shop dataset
# Days of the week and drink types
np.random.seed(42)
days = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']
drink_types = ['Hot', 'Iced']

# Generate counts: assume weekends have more iced drinks
data = []
for day in days:
    if day in ['Saturday', 'Sunday']:
        hot_count = np.random.poisson(30)
        iced_count = np.random.poisson(50)
    else:
        hot_count = np.random.poisson(40)
        iced_count = np.random.poisson(25)
    data.append({'day': day, 'drink_type': 'Hot', 'count': hot_count})
    data.append({'day': day, 'drink_type': 'Iced', 'count': iced_count})

df = pd.DataFrame(data)
print(df.head(10))

# Create a contingency table (pivot table) of counts
contingency_table = pd.crosstab(index=df['day'], columns=df['drink_type'], values=df['count'], aggfunc='sum')
print(contingency_table)

The output shows a table where each cell is the count of drinks sold on that day. For example, on Monday, we sold 38 hot coffees and 24 iced coffees. This is our observed data.

3. The Intuition: Observed vs. Expected

Here’s the core idea of the chi-square test: we compare what we actually saw (the observed counts) to what we would expect to see if there were no relationship between the variables (the expected counts).

Step 1: Calculate expected counts.

The formula in plain English is:

Expected count for a cell = (Row total * Column total) / Grand total

Let’s do this manually with pandas.

import numpy as np

# Calculate row totals, column totals, and grand total
row_totals = contingency_table.sum(axis=1)
col_totals = contingency_table.sum(axis=0)
grand_total = contingency_table.values.sum()

# Calculate expected frequencies
expected = np.outer(row_totals.values, col_totals.values) / grand_total
expected_df = pd.DataFrame(expected, index=contingency_table.index, columns=contingency_table.columns)
print("Expected frequencies:")
print(expected_df)

Step 2: Compare observed to expected.

If the observed and expected are close, there’s likely no relationship. If they’re far apart, something is going on.

Step 3: Compute the chi-square statistic.

The formula is:

χ² = Σ ( (Observed - Expected)² / Expected )

Let’s compute it step by step.

# Compute the chi-square statistic manually
observed = contingency_table.values
expected = expected  # from above

# Calculate (Observed - Expected)^2 / Expected
chi_sq_components = (observed - expected) ** 2 / expected
chi_sq_statistic = chi_sq_components.sum()

print("Chi-square components (each cell's contribution):")
print(pd.DataFrame(chi_sq_components, index=contingency_table.index, columns=contingency_table.columns))
print(f"\nChi-square statistic: {chi_sq_statistic:.2f}")

To calculate the chi-square components, we take the difference between the observed and expected counts for each cell, square that difference to remove negative signs, and then divide by the expected count to normalize the result based on the size of the expectation. Summing these values across all cells gives us the final chi-square statistic.

Why square? Why divide by expected?

Think of it like this: imagine you’re a waiter comparing your actual tips to your expected tips. If you expected $10 and got$12, that’s a $2 difference. But if you expected$100 and got $102, that's also a$2 difference — but much less surprising. Dividing by the expected value normalizes the difference, so a $2 difference on a$10 expectation counts more than a $2 difference on a$100 expectation. Squaring ensures that positive and negative differences don’t cancel each other out.

4. The Chi-Square Distribution: Where Does the ‘p-value’ Come From?

So we have a chi-square statistic of, say, 15.32. Is that big or small? To answer that, we need to compare it to a chi-square distribution.

The chi-square distribution is a family of curves that change shape based on degrees of freedom (df). For a test of independence:

df = (number of rows - 1) * (number of columns - 1)

In our case: (7 days - 1) * (2 drink types - 1) = 6 * 1 = 6 degrees of freedom.

Plain English: Degrees of freedom is roughly the number of cells that are free to vary once we know the row and column totals. It’s like having a fixed budget: once you decide how much to spend on rent, utilities, and food, the amount left for everything else is determined.

Let’s plot the chi-square distribution and see where our statistic falls.

import matplotlib.pyplot as plt
import scipy.stats as stats

# Define degrees of freedom
df = (contingency_table.shape[0] - 1) * (contingency_table.shape[1] - 1)

# Generate x values for the chi-square distribution
x = np.linspace(0, 30, 500)
y = stats.chi2.pdf(x, df)

# Plot the distribution
plt.figure(figsize=(10, 6))
plt.plot(x, y, 'b-', label=f'Chi-square distribution (df={df})')

# Shade the p-value area (area to the right of the statistic)
p_value = 1 - stats.chi2.cdf(chi_sq_statistic, df)
plt.fill_between(x, y, where=(x >= chi_sq_statistic), color='red', alpha=0.3, label=f'p-value = {p_value:.4f}')

# Mark our statistic
plt.axvline(x=chi_sq_statistic, color='red', linestyle='--', linewidth=2, label=f'χ² = {chi_sq_statistic:.2f}')

plt.xlabel('Chi-square statistic')
plt.ylabel('Probability density')
plt.title('Chi-Square Distribution and Our Test Statistic')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

If the p-value is low (typically below 0.05), we reject the null hypothesis that the variables are independent. But remember: the p-value is not the probability that the null hypothesis is true. It’s the probability of observing a chi-square statistic as extreme as ours (or more extreme) assuming the null is true.

5. Running the Test in Python: scipy.stats.chi2_contingency

Now let’s let Python do the heavy lifting. We’ll use scipy.stats.chi2_contingency and compare its output to our manual calculation.

from scipy.stats import chi2_contingency

# Run the chi-square test
chi2, p, dof, expected_scipy = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.2f}")
print(f"p-value: {p:.4f}")
print(f"Degrees of freedom: {dof}")
print("\nExpected frequencies (from scipy):")
print(pd.DataFrame(expected_scipy, index=contingency_table.index, columns=contingency_table.columns))

But wait — the chi-square statistic from scipy might not match our manual calculation! Why? Because scipy applies Yates’ continuity correction by default for 2x2 tables. Yates’ correction subtracts 0.5 from each |observed - expected| difference before squaring, to reduce bias for small samples.

Let’s check this.

# Manual calculation WITHOUT Yates' correction (as we did earlier)
chi_sq_no_correction = ((observed - expected_scipy) ** 2 / expected_scipy).sum()
print(f"Chi-square WITHOUT Yates' correction: {chi_sq_no_correction:.2f}")

# Manual calculation WITH Yates' correction (for 2x2 tables)
# Yates' correction: subtract 0.5 from each |observed - expected| before squaring
chi_sq_with_correction = ((np.abs(observed - expected_scipy) - 0.5) ** 2 / expected_scipy).sum()
print(f"Chi-square WITH Yates' correction: {chi_sq_with_correction:.2f}")

print(f"\nScipy's chi-square: {chi2:.2f}")

What’s happening? Our table is 7x2, not 2x2, so Yates’ correction doesn’t apply. That’s why scipy’s result matches our manual calculation without correction. But if you have a 2x2 table, scipy applies Yates’ correction by default, and you need to know this to avoid confusion.

6. Interpreting the Results: What Does a Significant Chi-Square Actually Tell Us?

A significant chi-square test (p < 0.05) tells us there’s evidence of a relationship between the variables. But it doesn’t tell us:

• How strong the relationship is
• Which categories are driving the relationship
• Whether the relationship is causal

Effect size: Cramer’s V

Cramer’s V is a measure of association strength that ranges from 0 (no association) to 1 (perfect association).

# Compute Cramer's V
n = contingency_table.values.sum()
min_dim = min(contingency_table.shape) - 1
cramers_v = np.sqrt(chi_sq_statistic / (n * min_dim))
print(f"Cramer's V: {cramers_v:.3f}")

A Cramer’s V of 0.3 or above is generally considered a moderate to strong association.

Post-hoc analysis: Standardized residuals

Standardized residuals tell us which cells contribute most to the chi-square statistic. A residual with absolute value greater than 2 is considered a major contributor.

# Compute standardized residuals
residuals = (observed - expected_scipy) / np.sqrt(expected_scipy)
residuals_df = pd.DataFrame(residuals, index=contingency_table.index, columns=contingency_table.columns)
print("Standardized residuals:")
print(residuals_df)

# Highlight cells with |residual| > 2
print("\nCells with |residual| > 2 (major contributors):")
major_contributors = residuals_df[abs(residuals_df) > 2].dropna(how='all')
print(major_contributors)

Plain English: A big chi-square means something is going on, but you need more digging to know what. The standardized residuals help you pinpoint which days and drink types are driving the effect.

7. When NOT to Use a Chi-Square Test: Small Expected Frequencies and Fisher’s Exact Test

The chi-square test has a rule of thumb: expected frequencies should be at least 5 in each cell. If they’re smaller, the chi-square approximation to the distribution is poor, and your p-values can be misleading.

Specifically:

• No more than 20% of expected cells should be below 5
• No expected cell should be below 1

When this happens, use Fisher’s exact test instead. It calculates the exact p-value by enumerating all possible tables with the same margins.

Let’s create a scenario with small expected frequencies.

# Create a subset with rare drink types (e.g., a new seasonal drink)
rare_data = [
    {'day': 'Monday', 'drink_type': 'Hot', 'count': 45},
    {'day': 'Monday', 'drink_type': 'Iced', 'count': 2},
    {'day': 'Tuesday', 'drink_type': 'Hot', 'count': 50},
    {'day': 'Tuesday', 'drink_type': 'Iced', 'count': 1},
    {'day': 'Wednesday', 'drink_type': 'Hot', 'count': 48},
    {'day': 'Wednesday', 'drink_type': 'Iced', 'count': 3},
]
rare_df = pd.DataFrame(rare_data)

# Create contingency table
rare_contingency = pd.crosstab(index=rare_df['day'], columns=rare_df['drink_type'], values=rare_df['count'], aggfunc='sum')
print("Contingency table with small expected frequencies:")
print(rare_contingency)

# Run chi-square test
chi2_rare, p_rare, dof_rare, expected_rare = chi2_contingency(rare_contingency)
print(f"\nChi-square p-value: {p_rare:.4f}")
print("Expected frequencies:")
print(pd.DataFrame(expected_rare, index=rare_contingency.index, columns=rare_contingency.columns))

Notice that some expected frequencies are below 5. Now let’s compare with Fisher’s exact test.

from scipy.stats import fisher_exact

# Fisher's exact test works on 2x2 tables
# We'll use the first two days as an example
small_table = rare_contingency.iloc[:2, :].values
odds_ratio, p_fisher = fisher_exact(small_table)
print(f"Fisher's exact test p-value: {p_fisher:.4f}")
print(f"Chi-square p-value for same subset: {chi2_rare:.4f}")

The p-values might differ noticeably. When expected frequencies are small, Fisher’s exact test is more reliable.

8. Common Pitfalls and How to Avoid Them

There are several common mistakes to watch for when performing chi-square tests.

Pitfall 1: Using chi-square on continuous data that has been binned

Binning continuous data throws away information and can create artificial patterns. The choice of bin boundaries can dramatically change your results.

# Generate continuous data (age)
ages = np.random.normal(40, 15, 1000)

# Bin into different categories
bins_1 = [0, 30, 50, 100]
bins_2 = [0, 35, 45, 100]

age_binned_1 = pd.cut(ages, bins=bins_1, labels=['Young', 'Middle', 'Old'])
age_binned_2 = pd.cut(ages, bins=bins_2, labels=['Young', 'Middle', 'Old'])

# Create a second categorical variable
purchase = np.random.choice(['Yes', 'No'], 1000)

# Create contingency tables
table_1 = pd.crosstab(age_binned_1, purchase)
table_2 = pd.crosstab(age_binned_2, purchase)

# Run chi-square tests
chi2_1, p_1, _, _ = chi2_contingency(table_1)
chi2_2, p_2, _, _ = chi2_contingency(table_2)

print(f"Chi-square with bins {bins_1}: p = {p_1:.4f}")
print(f"Chi-square with bins {bins_2}: p = {p_2:.4f}")

Notice how different bin boundaries can lead to different conclusions. Avoid binning continuous data for chi-square tests if possible.

Pitfall 2: Ignoring Yates’ correction for 2x2 tables

As we saw earlier, scipy applies Yates’ correction by default for 2x2 tables. If you’re doing manual calculations, make sure to apply it.

Pitfall 3: Interpreting a significant chi-square as a strong relationship

Always check Cramer’s V. A large sample can make even a tiny effect statistically significant.

Pitfall 4: Running multiple chi-square tests without correction

If you’re testing multiple pairs of variables, adjust your significance level (e.g., Bonferroni correction: divide 0.05 by the number of tests).

Pitfall 5: Using chi-square when the data is paired

If you have before/after measurements on the same subjects (e.g., did customers switch from hot to iced after a promotion?), use McNemar’s test instead.

from statsmodels.stats.contingency_tables import mcnemar

# Paired data: same customers before and after a promotion
# Each customer is classified as 'Hot' or 'Iced' before and after
before = np.array(['Hot', 'Iced', 'Hot', 'Iced', 'Hot'])
after = np.array(['Iced', 'Iced', 'Hot', 'Iced', 'Iced'])

# Create the 2x2 contingency table for McNemar's test
# Table: (before, after) pairs
#        After Hot   After Iced
# Before Hot   a          b
# Before Iced  c          d

a = sum((before == 'Hot') & (after == 'Hot'))
b = sum((before == 'Hot') & (after == 'Iced'))
c = sum((before == 'Iced') & (after == 'Hot'))
d = sum((before == 'Iced') & (after == 'Iced'))

mcnemar_table = [[a, b], [c, d]]

# Wrong: chi-square test
chi2_wrong, p_wrong, _, _ = chi2_contingency(mcnemar_table)
print(f"Chi-square (wrong for paired data): p = {p_wrong:.4f}")

# Correct: McNemar's test
result = mcnemar(mcnemar_table, exact=True)
print(f"McNemar's test (correct): p = {result.pvalue:.4f}")

9. Recap: What You’ve Learned

Let’s bring it all together:

• Chi-square test of independence: Use when you have two categorical variables and want to know if they’re related.
• The test compares observed counts to expected counts under the assumption of independence.
• A large chi-square statistic (small p-value) suggests a relationship, but always check effect size (Cramer’s V) and standardized residuals.
• Yates’ correction is applied by default in scipy for 2x2 tables — be aware of it.
• If expected frequencies are too small, use Fisher’s exact test instead.
• Watch out for common pitfalls: binning continuous data, ignoring multiple comparisons, using the wrong test for paired data.

10. Check Your Understanding

Test yourself with these questions, ordered from basic recall to creative application.

Remember: What is the null hypothesis of a chi-square test of independence?

Understand: Explain why we divide by the expected frequency in the chi-square formula.

Apply: Given a contingency table with row totals [100, 150], column totals [120, 130], and grand total 250, calculate the expected frequency for the cell in row 1, column 1.

Analyze: A chi-square test gives p = 0.03. What does this mean? What doesn’t it mean?

Evaluate: Compare chi-square and Fisher’s exact test. When would you choose one over the other?

Create: Design a study where a chi-square test would be appropriate, and one where it would not. Explain your reasoning.

11. Related Articles

• Covers the continuous-data counterpart to chi-square tests.
• Extends the idea of modeling categorical outcomes with predictors.
• Applies chi-square tests in a real-world experimental context.

12. Apply What You Learned