Python Generators: How to Process Massive Datasets Without Crashing Your Computer

The ‘Memory Full’ Wall

Have you ever tried to open a massive CSV file, only to watch your computer turn into a very expensive paperweight? Your cursor freezes, the fan whirrs like a jet engine, and Python finally throws that dreaded message: MemoryError.

Here’s what’s happening underneath. When you create a standard Python list, you’re asking your computer to build the entire finished product and store it in your RAM. It’s like cooking a meal for 1,000 people and putting every single plate on your kitchen counter at the same time. Unless you have a massive kitchen, you’ll run out of space.

Now let’s see what happens when we create a list with 50 million numbers.

import sys

try:
    # Creating a list of 50 million integers
    massive_list = [i for i in range(50000000)]
    print(f"List size: {sys.getsizeof(massive_list) / (1024**2):.2f} MB")
except MemoryError:
    print("Your computer just ran out of RAM!")

On a standard laptop, this list might take up close to 400 MB of RAM just for the structure itself — not counting the actual integers. Scale this to 500 million, or try loading a 10GB log file, and your system crashes. That’s the ‘Memory Full’ wall.

Instead of a list (the finished meal), we need a recipe — a generator. A generator doesn’t hold the data. It holds the instructions to make the data, one piece at a time.

Meet the ‘yield’ Keyword: Your Code’s Pause Button

In a normal function, you use return. Python hits return, the function is done — it hands over the result and clears its memory. yield works differently. Think of it as a ‘pause’ button.

When a function uses yield, it becomes a generator. Call it, and nothing runs yet. It just stands by. Ask for a value, and it runs until it hits yield, hands you that one value, then stays right there, waiting for the next request.

The tricky part: the function ‘remembers’ its state. Let’s compare a standard function to a generator.

def get_numbers_list(n):
    result = []
    for i in range(n):
        result.append(i)
    return result

def get_numbers_generator(n):
    for i in range(n):
        yield i

# The list function builds everything first
numbers = get_numbers_list(5)
print(f"List: {numbers}")

# The generator function just waits
gen = get_numbers_generator(5)
print(f"Generator object: {gen}")

# We get values one by one
print(f"First value: {next(gen)}")
print(f"Second value: {next(gen)}")

What this means in practice: the generator function only needs enough memory to hold one item at a time. The list function needs enough memory for all of them.

The Generator Expression: A One-Line Superpower

You probably know list comprehensions like [x for x in data]. They’re handy, but “eager”—the entire list gets built immediately.

Swap those square brackets [] for parentheses (), though, and you get a generator. That’s a generator expression.

import sys

# A list comprehension
list_comp = [i for i in range(1000000)]

# A generator expression
gen_exp = (i for i in range(1000000))

print(f"List size: {sys.getsizeof(list_comp)} bytes")
print(f"Generator size: {sys.getsizeof(gen_exp)} bytes")

On my machine, the list takes 8,448,728 bytes. The generator? 112 bytes.

Ten items or ten billion—the generator stays at 112 bytes. Why? Because it isn’t storing the numbers. It’s storing the logic of how to count to a million.

Real-World Data: Streaming a Massive CSV

Let’s apply this to a real data science problem. Say you have a CSV with 10 million rows of sales data, and you want only the rows where ‘price’ is over $100. pandas.read_csv() loads the whole file into RAM.

A generator lets you stream the file instead. RAM usage stays flat — even at 100GB.

import csv

def stream_expensive_items(filename):
    with open(filename, mode='r') as f:
        # csv.DictReader is a generator! It reads one line at a time.
        reader = csv.DictReader(f)
        for row in reader:
            if float(row['price']) > 100:
                yield row

# Let's imagine 'huge_data.csv' exists
# expensive_items = stream_expensive_items('huge_data.csv')
# for item in expensive_items:
#     print(item)

Here’s the idea: we crack the file open, grab one row, check the price. Toss it if it’s cheap; hand it over if it’s expensive. Then the next row. Never more than one row in memory at a time. The upshot: you can process files larger than your RAM without slowing your computer.

The Catch: When Generators Aren’t the Answer

Generators sound like magic, but they have two real limitations.

First, they’re one-and-done. Loop through a generator and it’s exhausted. Think of a roll of film—once you’ve developed it, you can’t play it again without starting over and re-reading the source.

Second, no indexing. Want the 500th item? The generator has to run the logic for the first 499 to get there.

gen = (i for i in range(3))

# First pass
for val in gen:
    print(val)

# Second pass - this will print NOTHING
for val in gen:
    print("This won't run!")

try:
    print(gen[0])
except TypeError as e:
    print(f"Error: {e}")

If you need to sort, shuffle, or access specific rows multiple times, stick with a list (or a Pandas DataFrame). But for cleaning, filtering, or aggregating, generators are hard to beat.

import csv
import logging

logger = logging.getLogger(__name__)

def stream_rows(filename):
    with open(filename) as f:
        reader = csv.DictReader(f)
        for row in reader:
            yield row

def run_inference(filename, model):
    rows = stream_rows(filename)

    # Production monitoring: log total row count
    total = sum(1 for _ in rows)
    logger.info(f"Processing {total} rows from {filename}")

    # Score every row
    predictions = [model.predict(row) for row in rows]
    return predictions