mrkeyoor.com_
Thu 03 Sept 12:16 UTC
Open Source6 min read

Polars 2.0's Streaming Default Can Silently Reorder Rows

Polars 2.0 RC1 sends lazy queries through its streaming engine by default. The speed and memory gains come with an upgrade trap: some operations no longer preserve row order.

Polars 2.0 can make an unchanged LazyFrame pipeline return rows in a different order. The first release candidate changes engine="auto" so lazy queries run on the streaming engine by default, and the project's migration guide warns that joins, group_by, and unpivot do not guarantee row order unless the query asks for it. A pipeline can therefore produce the same values in a new sequence without raising an error. For teams that write ordered exports or feed position-sensitive code, that is the part of this speed upgrade that deserves attention first.

Polars founder Ritchie Vink announced RC1 on September 2 and said the final 2.0 release should follow within weeks. The major-version boundary gives the project room to remove old design choices and switch defaults that could alter existing programs. The streaming engine is the largest of those changes.

Why streaming gets the default

In Polars 1.x, LazyFrame.collect() with the default auto setting resolved to the in-memory engine. In 2.0 RC1, auto resolves to streaming for lazy queries. Eager DataFrame operations still use the in-memory engine, while sink_* methods were already sent through streaming. Only lazy operations change executors here. That API is common in larger data pipelines. SQL calls that collect eagerly follow the lazy path too.

The speed case comes from Polars' own PDS-H tests, a derivative of TPC-H that the project says cannot be compared with official TPC-H results. On an AWS c7a.24xlarge with 96 virtual CPUs and 192 GB of memory, its May 2025 test recorded a total of 3.89 seconds for the streaming engine at scale factor 10, against 9.68 seconds for the in-memory engine. At scale factor 100, the totals were 23.94 and 152.27 seconds. Polars summarized the gap as three to seven times, depending on scale and query, and its 2.0 announcement says it expects roughly a fivefold aggregate gain. These are project-run results on a fixed query set, not a promise for every workload.

Memory behavior is the other reason for the switch. The in-memory engine works across the full dataset in memory, while streaming processes batches and can avoid the same cache pressure at larger scales. In the project's scale-factor-100 test, query 21 still fell back to the in-memory engine for a range join because streaming support was missing. The benchmark report names that fallback, which is a useful reminder that the default can select a mixed execution plan rather than a pure streaming path for every operator.

Row order becomes an explicit choice

The migration risk is easy to miss because relational operations usually promise a set of rows, not their incidental sequence. Existing applications often depend on that sequence anyway. A left join may be followed by a CSV write with no final sort. A grouped result may be compared with a checked-in fixture. A notebook can present an unpivot result as though its current order were part of the output. Polars 2.0 makes those assumptions visible because its streaming executor is free to produce another valid order.

Callers can ask an operation to preserve a supported order or sort the collected result explicitly. The old engine also remains available while a team investigates. These migration controls are short enough to put beside a version-pin change:

# Preserve the left input's order for this join.
result = left.join(
    right, on="k", how="left", maintain_order="left"
).collect()

# Restore the old executor for one query.
result = lf.collect(engine="in-memory")

# Or restore it for the process.
pl.Config.set_engine_affinity("in-memory")

Those controls come directly from the 2.0 upgrade guide. Explicit sorting is often the clearest contract when downstream code needs a total order, while maintain_order can avoid a separate sort for operations that support it. The in-memory setting is best treated as a compatibility switch: it gives a team time to locate order dependencies, but it also opts out of the new default's intended performance and memory behavior.

Tests need to distinguish content from sequence. If order has no meaning, compare sorted rows or use an order-insensitive assertion. If order does matter, add a final sort or an operation-level maintain_order argument and assert the chosen columns. The migration guide says the exact order shown in its examples is itself not guaranteed, so copying the RC's observed streaming order into a golden file would only replace one accidental contract with another.

Stricter failures replace silent coercion

The release candidate also rejects several operations that Polars 1.x tried to repair automatically. One example involves is_in across integers and floating-point values. The old behavior could coerce both sides to Float64; above 2^53, two different integer identifiers can map to the same representable float and produce a false match. In 2.0, Polars raises InvalidOperationError for that mismatch and requires the caller to choose an explicit conversion.

Horizontal concatenation changes in the same spirit. When frames had different heights, the previous default padded the shorter frame with nulls. RC1's strict mode raises ShapeError instead. Code that wants padding must request how="horizontal_extend". The official example uses transaction and fraud-count frames to show why an automatic null can conceal a failed upstream job rather than repair it.

Parsing and categorical conversions become more direct as well. Casting a string series to Date is removed in favor of .str.to_date() or .str.to_datetime(), where the parser can receive a format. Integer-to-categorical and categorical-to-integer casts move to categorical methods. The release post describes these removals as a way to make the intended operation explicit, especially where a generic cast had more than one plausible meaning.

This stricter behavior is also aimed at generated code. Polars points agent-written queries toward collect_schema(), which resolves types and catches schema-level mismatches without materializing the data. Failures that depend on actual values still require real data. For coding agents and CI jobs, the schema call provides a cheap validation step before a long lazy pipeline runs. Polars makes that AI-development connection itself; the practical test is whether generated migrations learn the new methods instead of repeatedly hitting the new errors.

Readers and removed APIs need a second pass

pl.read_csv now dispatches through pl.scan_csv(...).collect(). That brings parameters such as with_column_names, credential_provider, and include_file_paths to the eager reader, but removes n_threads, batch_size, sample_size, and rechunk. It also changes observable details: a list passed to columns is returned in the requested order, and a list of schema overrides must cover every field. The migration guide lists both the added and removed arguments, so wrapper libraries should inspect their forwarded keyword arguments before testing only the happy path.

IPC reading follows a similar route. For inputs that do not use PyArrow, pl.read_ipc now collects a scan and no longer accepts memory_map or rechunk; callers can invoke .rechunk() on the result. The guide notes that memory_map still had an effect in read_ipc before 2.0, even though the same option had already become a no-op in scan_ipc. Removing it may expose configuration code that appeared harmless during an earlier migration.

Removed names should at least fail with directions. RC1 adds AttributeRemovedError and ArgumentRemovedError, and the message can point to a replacement. The release post's examples map LazyFrame.melt to LazyFrame.unpivot, and the former join_nulls argument to nulls_equal. Polars says most affected functions had been deprecated for some time, so applications that already treat deprecation warnings as test failures should have less work than projects jumping from an older 1.x pin.

RC1 is available with pip install polars==2.0rc1, but the stable release has no exact date beyond the project's estimate of the following weeks. Before changing a production pin, watch issue reports involving row order and unsupported streaming operators. Reader wrappers that pass removed arguments need their own checks. Polars also says proper out-of-core execution, a new I/O plugin design, wider SQL coverage, join reordering, and a cost-based planner are still in flight for the 2.x series. They are plans beyond this release candidate, so the immediate decision is simpler: test the new default against real pipelines, and make every required ordering guarantee explicit.

We reviewed this

  1. learn — our honest review
  2. pipeline — our honest review
  3. polars — our honest review

Sources

  1. Pre-release of Polars 2.0
  2. Version 2.0-rc - Polars user guide
  3. Updated PDS-H benchmark results (May 2025)