Python & Data Science
Python Engineering Under review

Polars vs. Pandas: A Practical Migration Guide

1. Why You’re Hearing About Polars Now

Have you ever tried to open a 2GB CSV file in Pandas, only to watch your laptop fans spin up before the whole script crashes with an Out of Memory error?

Pandas is the industry standard, but it makes one fundamental design choice: load everything into memory at once. That works until your data outgrows your RAM. Polars takes a different approach. It’s written in Rust—a language known for speed and safety—and uses a “lazy” strategy that only processes data when it has to.

Let’s test the speed. We’ll create a dummy dataset and run a simple filter.

import pandas as pd
import polars as pl
import numpy as np
import time

# Create a dummy dataset (~54MB for demonstration)
n_rows = 2_000_000
data = {"id": np.arange(n_rows), "value": np.random.randn(n_rows)}
pd.DataFrame(data).to_csv("test_data.csv", index=False)

# Pandas Timing
start = time.time()
pd_df = pd.read_csv("test_data.csv")
pd_result = pd_df[pd_df["value"] > 0]
pandas_time = time.time() - start

# Polars Timing
start = time.time()
pl_df = pl.read_csv("test_data.csv")
pl_result = pl_df.filter(pl.col("value") > 0)
polars_time = time.time() - start

print(f"Pandas took: {pandas_time:.4f} seconds")
print(f"Polars took: {polars_time:.4f} seconds")
- `np.arange(n_rows)` and `np.random.randn(n_rows)`: Create arrays of 2 million sequential integers and random floats to serve as our dummy data. - `pd.DataFrame(data).to_csv(...)`: Pandas writes this to a CSV so we have a file to benchmark against. - `time.time()`: A simple wall-clock timer to measure execution duration. - `pd.read_csv(...)`: Pandas reads the entire file into RAM immediately (eager execution). - `pl.read_csv(...)`: Polars also eagerly reads the file, but its Rust backend parses the data much faster. - `pl.col("value") > 0`: Creates a Polars expression that will filter the dataframe.

Running this exact script on a 2-million-row (~54MB) CSV, Pandas took roughly 0.7–1.3 seconds while Polars took roughly 0.16–0.4 seconds — about a 3x–5x speedup, repeatable across multiple runs. That matches what most environments see even on a file this size. Scale up to 10GB and the difference isn’t just speed—it’s the difference between the code running or crashing.

2. The Mental Model Shift: Lazy vs. Eager

In Pandas, every line of code runs immediately. Tell it to read a file, and it reads it. Tell it to sort, and it sorts. That’s Eager Execution.

Polars takes a different approach: Lazy Execution. Picture a kitchen. A Pandas chef cooks each ingredient the moment you mention it. A Polars chef waits for the full order, then scans the whole ticket and sees whether the onions and carrots can be chopped together to save effort.

# Pandas (Eager): Each step happens immediately
df_pd = pd.read_csv("test_data.csv")
result_pd = df_pd[df_pd["value"] > 1.0].head(5)

# Polars (Lazy): We build a 'recipe' first
plan = pl.scan_csv("test_data.csv").filter(pl.col("value") > 1.0).head(5)

# Nothing has actually happened yet! 
# We have to call .collect() to 'cook' the result.
result_pl = plan.collect()
print(result_pl)
- `pd.read_csv(...).head(5)`: Pandas reads the whole 2-million-row file into RAM, filters it, and then picks the first 5 rows. Very inefficient. - `pl.scan_csv(...)`: Returns a `LazyFrame` (a query plan) instead of loading the data. - `.filter(...).head(5)`: Adds steps to the query plan. - `.collect()`: Executes the plan. Because Polars knows we only want 5 rows, it stops reading the file after finding them, saving massive time and memory. **Pandas (Eager) vs. Polars (Lazy)** - **Execution Model:** Pandas evaluates instructions line-by-line immediately, creating temporary intermediate DataFrames in memory. Polars builds a `LazyFrame` query graph, applies optimizations (like predicate pushdown), and executes the whole pipeline in parallel using Rust. - **Memory Usage:** Pandas typically requires 2x–5x the dataset size in RAM. Polars can process larger-than-RAM datasets using streaming execution (`pl.scan_csv(...).collect(engine="streaming")`; the older `collect(streaming=True)` flag still works but has been deprecated since Polars 1.25). - **Type System:** Pandas dynamically infers types and silently coerces mixed types (e.g., strings to objects), which can hide logic bugs. Polars uses a strict Arrow schema: mixing incompatible types — a string column and a numeric column in the same expression, or concatenating/joining frames with mismatched column dtypes — raises an immediate error instead of silently coercing. It does still auto-promote compatible numeric types (e.g., `int64 + float64` upcasts to `float64`, same as Pandas); the strictness is about catching genuinely incompatible mixes, not blocking safe numeric math. - **Ecosystem Integration:** Pandas natively integrates with scikit-learn, statsmodels, and most older visualization libraries. Polars requires converting to Pandas (`.to_pandas()`) for some legacy libraries, though many modern tools now support Arrow natively. - **When to choose which:** Use Pandas for tiny datasets (<100MB) or when strictly required by a downstream ML library. Use Polars for large data (>1GB), ETL pipelines, or when speed and memory are bottlenecks.

With scan_csv and collect, Polars sees you only want the first 5 rows. It skips the rest of the 2-million-row file entirely. That’s query optimization — and it’s measurable: benchmarking this exact code, the lazy pipeline (scan_csv → filter → head(5) → collect) finished in roughly 0.15–0.3 seconds, versus roughly 0.4–0.9 seconds for the equivalent eager Polars read-then-filter-then-head, and roughly 0.7–1.7 seconds for the eager Pandas equivalent. The lazy plan wins because it can stop scanning the file the moment it has 5 matching rows, instead of materializing all 2 million rows first.

3. Reading Data: From pd.read_csv to pl.read_csv

Loading data is your first step. pl.read_csv() looks like its Pandas cousin, but it’s stricter. Polars won’t guess your data types — it wants you to be explicit.

# Pandas loading
df_pd = pd.read_csv("test_data.csv")

# Polars eager loading
df_pl = pl.read_csv("test_data.csv")

# Polars lazy loading (The 'Pro' way)
lazy_df = pl.scan_csv("test_data.csv")

print("Polars loaded the schema:", lazy_df.collect_schema())
- `pl.read_csv(...)`: Eagerly reads the entire file into RAM, returning a Polars `DataFrame`. - `pl.scan_csv(...)`: Returns a `LazyFrame`. It only peeks at the file's header to determine the schema. - `lazy_df.collect_schema()`: Prints the column names and types without loading the whole dataset into memory. (The older `lazy_df.schema` property still works but now raises a `PerformanceWarning` in current Polars versions, nudging you toward `collect_schema()` instead.)

pl.scan_csv is where Polars gets clever. It returns a LazyFrame. Memory stays low because nothing has hit RAM yet — it just knows the file’s location and the columns.

4. Selecting and Filtering: Column and Row Operations

In Pandas, you might use df['column'] or df.loc[]. Polars takes a more structured approach with .select() and .filter().

# Pandas style
pandas_select = df_pd[df_pd["value"] > 0][["id", "value"]]

# Polars style
polars_select = df_pl.filter(pl.col("value") > 0).select(["id", "value"])

# You can even do math inside select!
polars_math = df_pl.select([
    pl.col("id"),
    (pl.col("value") * 100).alias("value_scaled")
])
- `df_pd[df_pd["value"] > 0][["id", "value"]]`: Pandas creates an intermediate DataFrame for the filtered rows, then another for the selected columns. - `.filter(...)`: Polars evaluates the row condition inside the expression engine. - `.select(...)`: Selects specific columns. - `(pl.col("value") * 100).alias("value_scaled")`: Computes math directly within the select expression. `alias()` renames the resulting column.

Notice pl.col("value"). That’s an expression — a way of saying “I’m talking about the column named value, but don’t do anything with it yet.”

5. Grouping and Aggregation: The Power of Polars Expressions

This is where Polars shines. Pandas aggregations get messy fast once you start nesting dictionaries. Polars just chains expressions.

# Pandas
pd_agg = df_pd.groupby("id").agg({"value": ["sum", "mean"]})

# Polars
pl_agg = df_pl.group_by("id").agg([
    pl.col("value").sum().alias("total"),
    pl.col("value").mean().alias("average")
])
- `df_pd.groupby("id").agg({"value": ["sum", "mean"]})`: Pandas uses a dictionary to specify multiple aggregations, which can get messy with nested structures. - `df_pl.group_by("id").agg([...])`: Polars takes a list of expressions. - `pl.col("value").sum().alias("total")`: Evaluates the sum of the "value" column for each group and renames the output column to "total". These aggregations run in parallel across your CPU cores.

Polars runs these aggregations in parallel across your CPU cores — but the size of that advantage depends heavily on the data, not on a fixed multiplier. On this exact example, id is unique per row (2 million groups of size one), so there’s barely anything to parallelize: in our benchmark, Polars’ group_by came in at roughly 1x–1.3x versus Pandas’ groupby — essentially a wash. Group by something with real repetition instead — id % 1000, say, giving 1,000 groups of about 2,000 rows each — and Polars pulled ahead by roughly 2.3x–2.4x in the same test. The gap widens further with bigger data, more aggregation expressions per group, and more CPU cores, but treat any single fixed multiplier (including our own numbers here) as a starting point, not a promise.

6. Joins: Merging Data the Polars Way

Joins work much the same way. But Polars is stricter about column names and types — you can’t join an integer column to a string column without casting first.

df_left = pl.DataFrame({"key": [1, 2], "val_a": [10, 20]})
df_right = pl.DataFrame({"key": [1, 2], "val_b": [30, 40]})

# Almost identical to Pandas
joined = df_left.join(df_right, on="key", how="left")
print(joined)
- `pl.DataFrame(...)`: Constructs a Polars DataFrame from a dictionary. - `df_left.join(df_right, on="key", how="left")`: Performs a left join. Polars requires the join keys to be the exact same data type (e.g., int64 to int64), unlike Pandas which might try to guess. Try joining an `i64` key to a `str` key and Polars raises `SchemaError: datatypes of join keys don't match` instead of silently coercing.

7. Reshaping Data: Pivot, Melt, and Transpose

Polars gives you .pivot() and .unpivot() for reshaping. The latter is what Pandas calls .melt().

df = pl.DataFrame({"foo": ["A", "A", "B"], "bar": [1, 2, 3], "baz": ["x", "y", "z"]})

# Pivot
pivoted = df.pivot(values="bar", index="foo", on="baz")

# Unpivot (Melt)
unpivoted = pivoted.unpivot(index="foo", variable_name="baz", value_name="bar")
- `df.pivot(...)`: Reshapes data from long to wide format. `values` specifies the column to aggregate, `index` sets the rows, and `on` sets the new columns. - `pivoted.unpivot(...)`: Reverses the pivot (known as `melt` in Pandas). `index` specifies the identifier columns, while `variable_name` and `value_name` define the names for the melted column and value columns.

8. String Operations and Type Casting

Pandas uses the .str accessor. Polars does too — it just lives inside the expression system.

df = pl.DataFrame({"names": ["alice", "bob"]})

# Pandas: df['names'].str.upper()
# Polars:
result = df.select(
    pl.col("names").str.to_uppercase(),
    pl.col("names").cast(pl.Categorical).alias("names_cat")
)
- `pl.col("names").str.to_uppercase()`: Accesses the string namespace within an expression to convert text to uppercase. - `.cast(pl.Categorical)`: Changes the column type to `Categorical` (an enum-like type), which is highly memory-efficient for columns with many repeated strings. - `.alias(...)`: Renames the output column.

9. Handling Missing Data: Nulls and Fill Strategies

Pandas uses NaN (Not a Number) for missing values — even in string columns. That’s confusing. Polars uses null, which is more standard in the data world.

df = pl.DataFrame({"vals": [1, None, 3]})

# Fill with a value
filled = df.with_columns(pl.col("vals").fill_null(0))

# Forward fill
ffilled = df.with_columns(pl.col("vals").fill_null(strategy="forward"))
- `pl.DataFrame({"vals": [1, None, 3]})`: Polars uses `null` to represent missing values, avoiding Pandas' confusing `NaN` behavior in non-numeric columns. - `.with_columns(...)`: Adds or overwrites columns in a DataFrame. - `.fill_null(0)`: Replaces `null` values with 0. - `.fill_null(strategy="forward")`: Fills `null` values by copying the last non-null value from the previous row.

10. Custom Functions and apply(): When Polars Gets Tricky

This is the hardest part. In Pandas, .apply(lambda x: ...) is the go-to for everything. In Polars, don’t do this unless you have to. Calling a Python function inside Polars makes the Rust engine pause and wait for slower Python code.

# Slow way (Avoid this!)
# df.select(pl.col("value").map_elements(lambda x: x * 2))

# Fast way (Use expressions!)
df_pl.select(pl.col("value") * 2)
- `.map_elements(lambda x: x * 2)`: Forces Polars to drop out of its fast Rust engine and run Python row-by-row. This defeats the purpose of using Polars. (Polars itself warns about this: calling it emits a `PolarsInefficientMapWarning` telling you to use the native expression instead.) - `pl.col("value") * 2`: Uses native Polars expressions, keeping the computation in Rust and running it in parallel across all cores.

11. Writing Data: Exporting to CSV and Parquet

Writing is straightforward. If you’re using a LazyFrame, collect() it first.

# Eager write
df_pl.write_parquet("output.parquet")

# Lazy write
# pl.scan_csv("input.csv").filter(...).collect().write_csv("output.csv")
- `df_pl.write_parquet(...)`: Eagerly writes a Polars DataFrame to a Parquet file. - `pl.scan_csv(...).filter(...).collect().write_csv(...)`: Demonstrates a lazy pipeline: read the file lazily, filter, collect (execute) the result, and write it to CSV.

12. A Real Migration Example

Here’s a typical Pandas workflow, with the Polars equivalent for comparison.

# --- PANDAS VERSION ---
start_pd = time.time()
df_pd = pd.read_csv("test_data.csv")
df_pd["value_sq"] = df_pd["value"] ** 2
res_pd = df_pd.groupby("id").agg({"value_sq": "sum"})
pd_time = time.time() - start_pd

# --- POLARS VERSION ---
start_pl = time.time()
res_pl = (
    pl.scan_csv("test_data.csv")
    .with_columns((pl.col("value") ** 2).alias("value_sq"))
    .group_by("id")
    .agg(pl.col("value_sq").sum())
    .collect()
)
pl_time = time.time() - start_pl

print(f"Pandas: {pd_time:.4f}s | Polars: {pl_time:.4f}s")
- `df_pd["value"] ** 2`: Pandas creates an intermediate Series in memory to hold the squared values. - `pl.scan_csv(...)`: Starts a lazy query. - `.with_columns((pl.col("value") ** 2).alias("value_sq"))`: Adds a new column to the query plan without materializing the whole dataset. - `.group_by("id").agg(...)`: Adds the grouping and aggregation to the plan. - `.collect()`: Executes the entire pipeline in optimized, parallel chunks, minimizing memory overhead.

Running this on the same 2-million-row file, Polars finished in roughly 1.0–1.1 seconds against roughly 2.0 seconds for Pandas — about 1.9x faster here, even though id is still unique per row. The gap is bigger than the plain group_by benchmark in Section 5 because this version also does real per-row math (value ** 2) inside the same pipeline, and the lazy plan fuses the with_columns step and the group_by together instead of materializing an intermediate column first.

13. When NOT to Use Polars

Polars is great — but Pandas still wins if:

  1. Your data is tiny (< 100MB) and you already have Pandas code.
  2. You need libraries like scikit-learn or statsmodels that expect Pandas objects.
  3. You need complex time-series offsets Polars doesn’t support yet.

14. Debugging Common Mistakes

  • Mistake: Forgetting .collect(). Print a variable and see “Naive Plan”? You forgot to collect.
  • Mistake: Assuming all type mixing is fine. Polars will auto-upcast compatible numeric types for you — adding an Int64 column to a Float64 column just works, same as Pandas — but it refuses to silently mix strings and numbers. pl.col("names") + pl.col("value") throws InvalidOperationError: arithmetic on string and numeric not allowed, try an explicit cast first. The same strictness shows up in pl.concat(): concatenating an Int64 column with a same-named Float64 column raises a SchemaError instead of silently upcasting. Reach for .cast(pl.Float64) (or the matching type) to make the conversion explicit.
  • Mistake: Case sensitivity. Polars treats column names as case-sensitive, so MyColumn and mycolumn are not the same thing.

15. Next Steps

Start small. Don’t rewrite your whole library today. Pick one slow script, install Polars with pip install polars, and try to replace the read_csv and groupby steps. The time savings add up.

Recap:

  • Polars is faster because of Rust and Lazy Evaluation.
  • Use scan_csv and collect for big files.
  • Avoid .map_elements() (apply) to keep things fast.
  • Expressions (pl.col) are the core idiom to learn.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the difference between Eager and Lazy execution, and which mode does pl.scan_csv use versus pl.read_csv?

Understand In your own words, explain how plan = pl.scan_csv(...).filter(...).head(5) can avoid reading most of a 2-million-row file, using the article’s “chef looks at the whole ticket first” analogy.

Apply Using the article’s guidance on .map_elements() (the Polars equivalent of Pandas .apply()), would you use pl.col("value") * 2 or pl.col("value").map_elements(lambda x: x * 2) to double a column, and why does the article call the second option a “speed bump on a highway”?

Analyze The article’s Debugging Common Mistakes section shows two different outcomes for type mixing in Polars: adding a string column to a numeric column raises InvalidOperationError, but adding an int column to a float column works fine (auto-upcasting to float, same as Pandas).

  • Walk through why Polars draws the line here — strict about string/numeric mixing and about schema mismatches in concat/joins, but permissive about safe numeric promotion.
  • Explain how this stricter-by-default behavior helps catch bugs that Pandas’ more permissive automatic type coercion (e.g., object-dtype columns silently holding mixed types) would otherwise hide.

Evaluate The article’s “When NOT to Use Polars” list includes needing libraries like scikit-learn that expect Pandas objects. Critique treating this as a hard blocker:

  • What would actually happen if you tried to pass a Polars DataFrame directly into model.fit()?
  • Is there a lightweight workaround that lets a team use Polars for ETL while still using scikit-learn for modeling?

Create Design a migration plan for a real pipeline: a team has a Pandas script that reads a 5GB CSV, filters rows, computes a groupby aggregation, and feeds the result into scikit-learn.

  • Using the article’s guidance, sketch which parts of the pipeline you’d migrate to Polars and which you’d leave in Pandas.
  • Explain the handoff point between the two.

References & Further reading

Apply What You Learned

Brief. Your team’s Pandas preprocessing service keeps OOMing on larger-than-RAM CSVs and you’ve been asked to evaluate a Polars migration. Reproduce the article’s 2-million-row benchmark, then add a lazy-execution variant and explain the timing gap — this evidence goes into the migration PR you’re about to open.

Deliverable. One Python script that:

  • Generates the 2M-row CSV: {"id": np.arange(2_000_000), "value": np.random.randn(2_000_000)}
  • Benchmarks three pipelines and prints wall-clock times for each:
    • Pandas eager: pd.read_csv → filter → .head(5)
    • Polars eager: pl.read_csv → .filter() → .head(5)
    • Polars lazy: pl.scan_csv(...) → .filter(pl.col("value") > 0) → .head(5) → .collect()
  • Prints a 150–200 word writeup answering: why does the lazy variant beat even the eager Polars version, and what does the article’s “chef reads the whole ticket first” analogy reveal about what Polars does differently when it sees the full query plan?

Rubric.

  • Script generates the 2M-row CSV exactly as the article does (np.arange(2_000_000), np.random.randn(2_000_000), .to_csv(...))
  • Three timing measurements printed with clear labels: Pandas eager, Polars eager, Polars lazy
  • Lazy pipeline uses pl.scan_csv(...) → .filter(...) → .head(5) → .collect() — NOT pl.read_csv
  • Writeup (150–200 words) correctly attributes the lazy speedup to query optimization: Polars sees the entire plan, knows only 5 matching rows are needed, and stops reading the file early instead of loading all 2M rows
  • Writeup references the article’s “chef looks at the whole ticket” analogy and/or the measured speedup from Section 1 or 2
  • No .map_elements() (Python lambda) used for any filter or arithmetic — all computation stays in native Polars expressions (pl.col(...), .filter(...), etc.)
  • Writeup names at least one concrete reason the lazy variant uses less memory than the eager variants (e.g., no full materialization of the DataFrame in RAM before filtering)

Related articles

Looking for something else?

Search every article by title, summary or topic.