Thirty tiny Parquet files barely benefited from DuckDB 2.0's new download machinery. In MotherDuck's alpha tests, that workload fell from 3.7 seconds to 3.3 seconds, while a scan over one large file dropped from 18.8 to 7.7 seconds. For developers querying cloud storage, the gap explains why an engine upgrade can save seconds on one pipeline and leave another almost unchanged.
The open-source database's next major release is approaching. DuckDB's release calendar lists October 21 for version 2.0.0, with an explicit warning that dates can move. That leaves a useful window to test existing workloads against the preview. The question is which part of your query spends time waiting, and whether the new engine can remove that wait.
Downloads can run ahead of the query
DuckDB's engineering account of asynchronous I/O describes a limitation of its earlier execution model: a worker fetching remote data could spend its time waiting for an HTTP response. More CPU capacity didn't necessarily mean more useful work. The new design separates download threads from query workers, allowing upcoming reads to proceed while workers process data that has already arrived. This matters most when request latency prevents a machine from using its available network bandwidth.
The team's own S3 benchmark ran on an EC2 instance with 64 virtual CPUs and 512 GB of RAM. For a roughly 22 GB Parquet file, mean query time fell from 8.230 seconds with DuckDB 1.5.5 to 2.844 seconds with a 2.0 development build. The published experiment averaged five runs and disabled the external file cache. Those conditions explain the result's scope: a large remote scan on substantial hardware, rather than a promise about every query on your laptop.
The preview I/O documentation currently lists Parquet and uncompressed, seekable UTF-8 CSV files as supported for asynchronous reads. JSON files and DuckDB's native database format still use synchronous I/O, according to that page. Supported scans use the new path automatically. A team reading JSON directly from object storage therefore shouldn't assume that this particular improvement applies to its current pipeline.
Read-ahead holds fetched data in memory, so it competes with the query's other memory needs. The same I/O reference says the default read_ahead_depth = -1 uses a memory-governed budget. Under pressure, its queue can shrink to one job. A positive fixed depth has no memory budget, while zero switches read-ahead off. Leaving the default in place is a reasonable starting point for evaluation, especially when joins already consume much of the available memory.
File layout still sets the limits
MotherDuck author Mehdi Ouazza ran his comparison on an M5 laptop over home internet. He attributes the small-file result to per-file round trips for metadata and data that read-ahead cannot remove. These measurements describe one machine and connection.
DuckDB's file-format guide recommends Parquet row groups of about 100,000 to one million rows and individual files between 100 MB and 10 GB. A row group is a unit the engine can process in parallel. One enormous group limits that work to one thread, while very small groups introduce overhead. The guide also recommends at least as many total row groups as CPU threads. These are starting ranges for testing, rather than a file-size target to apply blindly to every dataset.
The Parquet writing documentation explains the trade-off. Larger groups can compress better, but each writing thread holds more data before flushing. Smaller groups can help selective queries skip unwanted rows, yet each group brings metadata overhead. Developers controlling the export step can set ROW_GROUP_SIZE when writing files. That makes layout an adjustable part of the pipeline, alongside the engine version, with different choices for broad scans and queries that retrieve a small fraction of rows.
DuckDB's format guidance recommends considering native tables for repeated or join-heavy workloads when disk space permits. Native storage has additional statistics that can improve join planning. Faster remote reads may reduce the cost of scanning Parquet, but they don't remove the need to decide where frequently used data should live.
Flexible records need a query plan too
The release also improves VARIANT, which stores values as typed binary data and can hold different types in the same column. Its documentation describes nested field access and Parquet support, including a technique called shredding. Shredding decomposes nested data into typed columns so the engine can access relevant fields more efficiently. When writing Parquet, DuckDB can infer that structure from the first row group or accept an explicit shredding schema.
The 2.0 feature preview says the new execution path can work directly on shredded storage and push field extraction into scans. For a developer collecting event records with optional fields, that offers a way to retain flexible records while avoiding repeated parsing of every JSON string. It still makes sense to examine which fields the actual queries read. A flexible input format doesn't make all access patterns equally cheap.
Ouazza's five-million-event experiment illustrates that limit. A field-filter query took 63 milliseconds on 2.0 alpha VARIANT, against 366 milliseconds on JSON text. A list-membership query casting the list to VARCHAR[] went the other way: 1.96 seconds on VARIANT, against 357 milliseconds on JSON. These are reported alpha measurements. Test the field and list operations your application uses before converting a whole table.
Deep graph walks avoid repeated work
In DuckDB's recursive-engine report, a reachability query over one million edges fell from a median 4.051 seconds on 1.5.5 to 0.095 seconds on the 2.0 preview. Both versions reached the expected 20,000 nodes. The older version recorded roughly 19.7 billion scanned edge rows, while the preview scanned the base table's one million rows once.
DuckDB's implementation explanation describes how the new engine can retain a hash table built from stable input and probe it with newly reached nodes, instead of revisiting the edges on every iteration. It retains work only when repeating it would produce the same observable result. For someone walking deep relationships, this is a reason to retry an expensive SQL traversal. Ordinary flat aggregations need their own measurements.
Test the workload you will actually upgrade
DuckDB's alpha announcement explicitly says the clients aren't production-ready and asks users to report failures with reproducible examples. It also identifies core extensions available for alpha testing, including httpfs, ducklake, iceberg and spatial. An evaluation should include the client and extensions your application uses. Getting a query to run in the CLI establishes less than getting the application's complete data path to work.
For query diagnosis, DuckDB documents two useful commands. EXPLAIN shows the plan and estimated row counts without running the query. EXPLAIN ANALYZE executes it and reports actual row counts and operator timings. Compare the plans as well as elapsed time. That helps distinguish a scan that became cheaper from a join whose order changed, and gives a bug report something more specific than a stopwatch result.
Persistent databases deserve a separate check. The feature preview says 2.0 changes the default storage format and introduces some breaking changes. DuckDB's storage compatibility documentation distinguishes a newer engine reading an older file from an older engine reading a newer file. It describes the latter as best effort. Test on a copy and verify the return path before letting an upgrade write the only database file you have.
Watch the October release schedule and the final release details before making a production decision. Until then, a useful result is a reproducible comparison using your own files, client and queries. If your small-file scan barely moves, that finding can guide the next export change. The tentative release date gives you time to establish that baseline before the upgrade arrives.