Dask for Out-of-Core Data: When Your Dataset Doesn't Fit in Memory

1. Introduction: The 20GB Dataset on an 8GB Laptop

You’ve just received a 20GB CSV file. Your laptop has 8GB of RAM. You fire up a Jupyter notebook and type pd.read_csv('huge_file.csv'). The kernel churns for a few seconds, then you see it: MemoryError. Your session is dead. Everything you had in memory is gone.

Sound familiar? If you’ve been doing data science for more than a few months, you’ve probably hit this wall. The standard workaround is chunking:

import pandas as pd

# Chunking: read 100,000 rows at a time
chunks = pd.read_csv('huge_file.csv', chunksize=100_000)
results = []
for chunk in chunks:
    # Do something with the chunk
    results.append(chunk['value'].mean())
overall_mean = sum(results) / len(results)
print(f"Overall mean: {overall_mean:.4f}")

But that’s clunky. What if you need to do a groupby across the whole dataset? Or a join with another table? Chunking forces you to write manual accumulation logic, and it’s easy to make mistakes.

What if there were a way to write the same pandas code you already know, but have it work on data that lives on disk — without crashing your memory? That’s exactly what Dask does.

By the end of this article, you’ll be able to process datasets larger than your RAM using Dask. You’ll understand the intuition behind partitions, lazy evaluation, and task graphs. And you’ll know when Dask is the right tool — and when it’s not.

2. What is Dask? (Intuition Before Formalism)

Think of Dask as a team of workers, each handling a chunk of your data, coordinated by a manager. The manager builds a to-do list (the task graph) and then assigns tasks to workers. The workers don’t need to see the whole dataset at once — they just need their chunk.

Dask’s tagline is “Scale the Python tools you love.” It’s not a brand-new framework. It’s a parallel version of pandas, NumPy, and scikit-learn that can handle data that doesn’t fit in memory. You write the same code you’d write for pandas, but Dask runs it on chunks that live on disk.

Here’s the key: Dask is lazy. When you write operations, they’re recorded, not executed immediately. Only when you call .compute() does the work actually happen. This lets you chain many operations without memory pressure.

Let’s see this in action with a built-in dataset from Dask itself.

import dask.dataframe as dd

# Create a Dask DataFrame from a CSV file (or use a built-in example)
# For this demo, we'll use dask.datasets.timeseries() which generates random data
import dask.datasets

df = dask.datasets.timeseries(
    start='2020-01-01',
    end='2020-01-10',
    freq='1s',
    dtypes={'x': float, 'y': float}
)

print("Dask DataFrame object (lazy):")
print(df)
print(f"Number of partitions: {df.npartitions}")

You’ll see something like:

Dask DataFrame object (lazy):
Dask DataFrame Structure:
                   x        y
npartitions=24
2020-01-01     float64  float64
2020-01-02        ...       ...
...               ...       ...
2020-01-10        ...       ...
Dask Name: make-timeseries, 24 tasks
Number of partitions: 24

Notice: it didn’t load any data. It just recorded the structure and the number of partitions. Now let’s trigger the computation.

# Trigger computation with .compute()
result = df.head()  # head() is eager, returns a pandas DataFrame
print("First 5 rows:")
print(result)

.head() is one of the few eager operations in Dask. It loads just enough partitions to return the first few rows. For the full dataset, you’d use .compute().

3. How Dask Handles Out-of-Core Data: Partitions and Lazy Evaluation

Dask splits your dataset into partitions — small pandas DataFrames that each fit comfortably in memory. By default, each partition is about 100MB (you can change this with the blocksize parameter in read_csv). Partitions live on disk until needed.

When you write operations, Dask builds a task graph — a directed acyclic graph (DAG) of all the tasks needed to produce the final result. The graph is a blueprint. Dask can optimize it (e.g., fuse tasks) before execution.

Here’s the magic: because the graph is lazy, you can chain many operations without any memory pressure. Only when you call .compute() does Dask load partitions one by one, apply the operations, and combine results.

Let’s see the task graph in action.

import dask.dataframe as dd
import dask.datasets

# Create the same timeseries dataset
df = dask.datasets.timeseries(
    start='2020-01-01',
    end='2020-01-10',
    freq='1s',
    dtypes={'x': float, 'y': float}
)

# Filter, groupby, and compute mean — all lazy
lazy_result = df[df['x'] > 0].groupby('name')['y'].mean()

print("Lazy result object:")
print(lazy_result)
print()

# Now compute it
computed_result = lazy_result.compute()
print("Computed result (first 5 rows):")
print(computed_result.head())

You’ll see that lazy_result is a Dask Series with a task graph. After .compute(), you get a pandas Series with the actual values.

If you want to visualize the task graph (requires graphviz installed), you can do:

# Uncomment if you have graphviz installed
# lazy_result.visualize(filename='task_graph.png')

This produces an image showing the DAG of tasks — each node is a pandas operation on a partition, and edges show dependencies.

4. Dask DataFrame in Practice: A Walkthrough

Let’s build a realistic pipeline. We’ll use dask.datasets.timeseries() again (so you can follow along without downloading a huge file). We’ll filter, groupby, aggregate, and then join with a small lookup table.

import dask.dataframe as dd
import dask.datasets
import pandas as pd

# Create a timeseries dataset (about 1 million rows)
df = dask.datasets.timeseries(
    start='2020-01-01',
    end='2020-01-31',
    freq='1s',
    dtypes={'x': float, 'y': float}
)

print(f"Number of partitions: {df.npartitions}")
print(f"Total rows (approx): {df.shape[0].compute()}")

# Filter: keep only rows where x > 0
df_filtered = df[df['x'] > 0]

# Group by 'name' and compute mean of 'y'
result = df_filtered.groupby('name')['y'].mean()

# Compute the result
computed = result.compute()
print("Mean of y by name (first 5):")
print(computed.head())

Now let’s join this result with a small lookup table. The lookup table is small enough to be a pandas DataFrame, which we convert to Dask for the join.

# Create a small lookup table (pandas DataFrame)
lookup = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'department': ['Engineering', 'Sales', 'Marketing', 'Engineering', 'Sales']
})

# Convert to Dask DataFrame (single partition)
dask_lookup = dd.from_pandas(lookup, npartitions=1)

# Join with the aggregated result
# First, reset the index of the result so 'name' becomes a column
result_with_index = result.reset_index()

# Perform the join
joined = result_with_index.merge(dask_lookup, on='name', how='left')

# Compute the final result
final = joined.compute()
print("Joined result:")
print(final)

Interpretation: The mean value of y for each name is around 0.5 (since the timeseries generates random data uniformly between -1 and 1, and we filtered for x > 0, the mean of y is still roughly 0). The join adds the department information.

5. Under the Hood: Task Graphs and Dask Delayed

Not every pipeline fits neatly into the DataFrame API. For custom workflows, Dask provides dask.delayed. This lets you wrap any function call into a task graph node.

Think of delayed as a way to say: “Don’t run this now; just remember that you need to run it later, with these arguments.” You can combine delayed objects to build a graph, then call .compute() on the final object to execute everything.

Let’s compare a sequential for-loop with a delayed version.

Sequential version:

import time

# Simulate processing a list of files
def process_file(filename):
    time.sleep(0.1)  # Simulate I/O
    return len(filename)  # Some computation

filenames = [f"file_{i}.csv" for i in range(10)]

start = time.time()
results = []
for f in filenames:
    results.append(process_file(f))
total = sum(results)
end = time.time()
print(f"Sequential total: {total}, time: {end-start:.2f}s")

Delayed version:

import dask
import time

# Same function, but we'll wrap calls with delayed
def process_file(filename):
    time.sleep(0.1)
    return len(filename)

filenames = [f"file_{i}.csv" for i in range(10)]

# Create delayed objects for each file
delayed_results = [dask.delayed(process_file)(f) for f in filenames]

# Combine them with sum (also delayed)
delayed_total = dask.delayed(sum)(delayed_results)

# Compute everything
start = time.time()
total = delayed_total.compute()
end = time.time()
print(f"Delayed total: {total}, time: {end-start:.2f}s")

You’ll see the delayed version runs faster because Dask parallelizes the calls (by default using threads). The task graph is built automatically.

6. Partition Sizing and Best Practices

Dask isn’t always the answer. If your data fits in RAM and you don’t need parallelism, pandas is faster and simpler. Dask adds overhead for task scheduling and data serialization.

Rule of thumb:

• Data fits in RAM → use pandas.
• Data is 2–3x RAM → Dask on a single machine works well.
• Data is much larger → consider a Dask cluster.

Partition sizing:

• Aim for ~100MB per partition.
• Too small → too many tasks, overhead dominates.
• Too large → memory pressure, risk of spilling to disk.

Let’s see the overhead on a small dataset.

import pandas as pd
import dask.dataframe as dd
import numpy as np
import time

# Create a small dataset (1 million rows)
n = 1_000_000
df_pd = pd.DataFrame({
    'x': np.random.randn(n),
    'y': np.random.randn(n),
    'group': np.random.choice(['A', 'B', 'C'], size=n)
})

# Pandas operation
start = time.time()
result_pd = df_pd.groupby('group')['y'].mean()
end = time.time()
print(f"Pandas: {end-start:.4f}s")

# Dask operation (convert to Dask)
df_dask = dd.from_pandas(df_pd, npartitions=4)
start = time.time()
result_dask = df_dask.groupby('group')['y'].mean().compute()
end = time.time()
print(f"Dask: {end-start:.4f}s")

You’ll see pandas is faster because there’s no overhead. Dask shines when the dataset is larger than memory.

For larger-than-memory datasets, use dask.datasets.timeseries with a longer duration to simulate.

7. Scaling to a Cluster: Dask Distributed

When a single machine isn’t enough, Dask can distribute work across multiple machines. The distributed scheduler is a centrally managed system that coordinates workers across a cluster.

The best part: your DataFrame code stays exactly the same. You just create a Client object, and Dask automatically uses the cluster.

from dask.distributed import Client
import dask.dataframe as dd
import dask.datasets

# Start a local cluster (uses all cores)
client = Client()  # This creates a local cluster in the background
print(f"Dashboard link: {client.dashboard_link}")

# Same pipeline as before
df = dask.datasets.timeseries(
    start='2020-01-01',
    end='2020-01-10',
    freq='1s',
    dtypes={'x': float, 'y': float}
)

result = df[df['x'] > 0].groupby('name')['y'].mean().compute()
print(result.head())

# Close the client
client.close()

With the Client active, Dask distributes the partitions across workers. You can view the task dashboard (a web UI) at the link printed above to see real-time task execution.

8. When NOT to Use Dask (And What to Try Instead)

Dask is a powerful tool, but it’s not a silver bullet. Here are situations where you should think twice:

• Your data fits in RAM: pandas is faster and simpler. Dask adds overhead.
• You need speed on in-memory data: Try Polars or Vaex. They are often faster than pandas for in-memory operations.
• You have many small files: Consider consolidating them into a single Parquet file first. Dask handles many small files poorly.
• Your computation is embarrassingly parallel but simple: Use Python’s multiprocessing or Ray. They have less overhead.

As the Dask best practices page says: “Parallelism brings extra complexity and overhead. Sometimes it’s necessary for larger problems, but often it’s not.” Try better algorithms, file formats (Parquet), or compiled code (Numba) before reaching for Dask.

9. Recap and Next Steps

You’ve learned:

• Partitions: Dask splits data into chunks that fit in memory.
• Lazy evaluation: Operations are recorded in a task graph, not executed immediately.
• Task graph: A DAG of tasks that Dask optimizes and executes.
• .compute(): Triggers execution and returns a pandas object.
• Distributed scheduler: Scale to a cluster with the same code.

What’s next? In the next article, we’ll use Dask to train a machine learning model on a dataset that’s too large for scikit-learn. You’ll see how Dask-ML extends the same lazy, parallel approach to model training and hyperparameter tuning.

Check Your Understanding

Remember: What does .compute() do in Dask?

Understand: Explain in your own words why Dask is called “lazy.”

Apply: Modify the timeseries example to compute the standard deviation of x for each name, after filtering for y > 0.

Analyze: Compare the performance of pandas vs. Dask on a dataset of 10 million rows that fits in memory. Which is faster? Why?

Evaluate: You have a 50GB dataset on a 16GB laptop. Would you use Dask on a single machine or set up a cluster? Justify your answer.

Create: Write a Dask pipeline that reads 100 CSV files from a directory, filters each for rows where value > 0, and computes the mean of value across all files. Use dask.delayed for custom processing.

Related articles

• Part 6: Speeding Up Python With Numba: JIT Compilation Without Rewriting in C — For in-memory numerical loops, Numba can give you 100x speedups without leaving Python.
• Part 5: JAX for NumPy Users: What’s Actually Different (and Why It’s Fast) — JAX offers automatic differentiation and GPU acceleration for array operations.