TL;DR
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Polars 2.0 switches LazyFrame collection to its streaming engine by default and enables initial spill-to-disk support, alongside expanded SQL support and a Map dtype. Polars reports strong results in its TPC-H and TPC-DS tests, but those comparisons used specified hardware and settings and have not been independently verified here.
Polars has released version 2.0, making its streaming engine the default when users collect a LazyFrame and enabling initial spill-to-disk support for selected operations. The release also expands SQL support and adds a Map data type, changes aimed at handling more analytical workloads while reducing memory pressure.
Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says this can improve memory use and performance across many queries. The trade-off is that some operations, including joins, group-bys and unpivots, may not preserve observable row order by default. Users who need that behavior can set maintain_order=True where supported.
Version 2.0 also enables out-of-core processing, which lets supported operations spill data to disk rather than requiring all working data to remain in memory. According to Polars, spilling begins at about 80% of available RAM, with a default disk budget of 64 GB. The initial supported operations include sorts, window functions and many expressions; joins and group-bys are not yet included in out-of-core support.
Polars says its optimizer and engine changes include join reordering, improved common-subplan elimination, dynamic predicates and bloom filters. The release also introduces a native Map dtype for Arrow MapType data. Previously, Polars read that structure as a list of key-value structs; the new type provides dictionary-like operations such as key lookup, checking whether a key exists, and retrieving keys or values.
Streaming Changes Affect Query Behavior
The default shift matters to teams that use Polars for datasets that put pressure on available memory. Streaming execution and disk spilling may allow some workloads to complete without keeping the full working set in RAM, potentially reducing the need to manually choose an execution mode. Those benefits depend on the query: disk spilling currently covers only a defined set of operations, and Polars says its threshold may need tuning.
The change also has a behavioral implication, not just a performance one. Where a workflow relies on row order after joins, group-bys or unpivots, users may need to request order explicitly and check results during migration. SQL improvements and the Map dtype broaden the ways developers can use Polars, but the release post’s benchmark results should be read as vendor-reported measurements, not a guarantee that every workload will run faster.
Products Worth Considering
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
Amazon Kindle Paperwhite (12th Generation, 2024), 16 GB, 7-inch Display, Jade
Apple iPad Air (4th Generation, 10.9-inch, Wi-Fi, 256GB) – Space Gray
As an affiliate, we earn on qualifying purchases.
How Polars Tested SQL Performance
Polars describes version 2.0 as a major version because the streaming engine’s default behavior can change row ordering for certain operations. The release post presents the release as a combination of that execution change, performance and optimizer work, SQL support, stricter handling of data types, and new functionality. It does not provide an independent evaluation of those changes.
For its SQL comparison, Polars ran queries derived from TPC-H and TPC-DS against DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0. Tests used two machines: one with 16 vCPUs and 32 GB of RAM, and another with 192 vCPUs and 384 GB. Each query ran five times in a hot setting; Polars compared the best run by total runtime and geometric mean. The company says Polars and both DuckDB versions completed all queries, while DataFusion timed out on TPC-DS query 72, timed out once on query 67 and ran out of memory on TPC-H query 18 on the smaller machine. Those affected queries were excluded from the results for all engines.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars, in its version 2.0 release post
Limits of the Initial Release
The release post does not establish how the reported performance results will translate to other datasets, hardware, storage configurations or query patterns. Polars invites readers to reproduce the SQL tests and links to a benchmark repository, but the source material does not include an independent replication.
Out-of-core support is also incomplete: joins and group-bys are not yet listed among the operations that can spill to disk. Polars says it plans to add them, without giving a delivery date. The company also says the roughly 80% RAM spill threshold may need tuning. The release post identifies a performance cost when scaling to 192 threads for small-data queries and says Polars has diagnosed the cause, but does not provide a fix date.
More Disk-Backed Operations Planned
Polars says it plans to add out-of-core support for joins and group-bys, which would extend disk-backed execution to two widely used operations. It has not announced a schedule for that work. The company also says it hopes to address the overhead observed on some small-data queries when using 192 threads in a future release.
For users adopting version 2.0, the immediate next step is to test representative queries, particularly those that depend on row ordering or approach memory limits. Polars has published a repository for reproducing its SQL benchmarks; broader independent testing would help clarify how the release performs across different workloads.
Key Questions
What is the main change in Polars 2.0?
LazyFrame collection defaults to the streaming engine. Polars also enables initial spill-to-disk support and expands SQL and data-type features.
Does Polars 2.0 preserve row order by default?
Not for certain operations, including joins, group-bys and unpivots, according to the release post. Users who need observable order can set maintain_order=True where applicable.
Which operations can spill data to disk?
The initial supported operations include sorts, window functions and many expressions. Polars says joins and group-bys are planned for later, but does not give a schedule.
Did Polars prove it is faster than DuckDB and DataFusion?
No independent proof is provided in the source material. Polars reports that its default configuration was fastest in all but one of its tested benchmarks, under specified TPC-H and TPC-DS conditions. Results may differ on other workloads and hardware.
Source: hn
