DuckDB is adding asynchronous reads for Parquet and CSV files in version 2.0, a change aimed at keeping analytical queries moving while data travels from remote object storage. The open-source database's engineers say the new path will be enabled by default in the next major release, currently planned for fall 2026.
This matters because DuckDB is increasingly used in a setting its original I/O design did not prioritize: compute running separately from the data it queries. A synchronous reader is a reasonable fit when files are on a fast local SSD. Point the same engine at a large dataset in S3, however, and its worker threads can spend much of their time waiting on network requests.
In benchmarks published by the DuckDB team, asynchronous I/O cut one remote Parquet query from 8.230 seconds to 2.844 seconds, while a remote CSV version fell from 877.563 seconds to 45.264 seconds. Those are project-run tests rather than independent results, and the CSV case is an especially I/O-heavy example. Still, they show why the database is changing a basic part of how it schedules work.
A local-first assumption meets remote storage
DuckDB earned much of its popularity as an in-process analytical database: applications can query files directly without deploying and maintaining a separate database server. Its columnar execution engine can avoid unnecessary reads by pushing filters and column selections down into formats such as Parquet. On a local machine, splitting a scan into row groups or fixed-size buffers gave worker threads enough work while storage responded quickly. Joins, aggregations and decoding were often the more important constraints.
That balance shifts when the file lives across a network. A Parquet scan may need multiple byte-range requests for each row group. With synchronous I/O, the worker that asks for a range waits until the bytes arrive before it can decode them or take another job. Low CPU use in that situation is not efficiency; it can mean expensive compute is sitting idle because too few requests are in flight to fill the available network link.
DuckDB's own example is deliberately simple:
FROM read_parquet('s3://bucket/file.parquet');
The SQL does not change under the new design. The difference is below it: fetching the next pieces of a file can overlap with decoding and processing the pieces that have already arrived. For users, that preserves the appeal of querying remote files directly instead of requiring a separate import step or a new service layer.
Two thread pools split waiting from computing
DuckDB 2.0 separates work into two pools. The regular pool contains the CPU workers that decode data, run joins and perform aggregations. It defaults to one worker for each available CPU thread. A new asynchronous pool handles tasks that are expected to block on I/O, such as waiting for an HTTP response. The implementation defines these as distinct REGULAR and ASYNC scheduler types in DuckDB's source tree.
The async pool can be much larger because most of those threads are not consuming CPU while they wait. DuckDB's default is four async threads per system thread, capped at 256 in total. A read-ahead queue creates jobs before a regular worker needs them and immediately schedules their fetch tasks. For Parquet, a job is a row group; for CSV, it is generally a byte-range scan boundary. Several fetches belonging to one job can proceed at the same time.
When a regular worker reaches the oldest queued job, it either begins decoding completed data or parks that scan task. Parking is the key distinction: the worker is free to execute another pipeline task instead of blocking. The final fetch wakes the scan, which can resume on any regular worker. This is asynchronous behavior built on pools of ordinary threads, not a promise that every storage layer exposes a native non-blocking interface.
Read-ahead creates a second problem, though. Keeping many requests in flight means keeping their returned data in memory until CPU workers can consume it. DuckDB therefore connects the queue to the same temporary-memory manager that allocates space among joins, sorts and window functions. Under pressure, the manager reduces the backlog, potentially to one job, making the scan behave more like the synchronous path.
Users can also set read_ahead_depth: -1, the default, lets memory govern the depth; a positive number caps queued jobs; and 0 turns off read-ahead. This makes the performance-memory tradeoff visible without requiring applications to manage the queue themselves.
The largest gains appear across the network
For its main test, DuckDB ran TPC-H Query 6 at scale factor 100 against data in S3. Compute and storage were in the same cloud region, and the machine had 64 virtual CPUs, 512 GB of RAM and a 25 Gbit/s network connection. The team disabled DuckDB's external file cache, ran each query five times and reported the mean. The lineitem table contained 600,037,902 rows.
On a roughly 22 GB Parquet file, stable DuckDB 1.5.5 took 8.230 seconds. The 2.0 development build took 2.844 seconds with its default, memory-governed read-ahead, close to a threefold improvement. A tuned run completed in 2.227 seconds. According to the project's network measurements, the older version stayed around 5 Gbit/s, while the new path approached the link's 25 Gbit/s limit.
The change also worked across a partitioned version of the dataset containing 976 small Parquet files: runtime dropped from 9.344 seconds to 2.945 seconds. That result is useful because object-store datasets are often divided by time, customer or another key, leaving a query to open and scan many files. Concurrent fetches can overlap both the data reads and some of the latency around opening those objects.
The CSV result was more dramatic. Querying an 80.89 GB file fell from 877.563 seconds to 45.264 seconds. CSV is row-oriented, so the scan must transfer far more data than the equivalent column-pruned Parquet query. That makes it a strong demonstration of bandwidth utilization, but not a general promise of a 20-fold speedup for DuckDB queries. Workloads that read less data, spend more time computing, or already hit storage bandwidth will see a smaller difference.
Local tests reinforce that limit. On a MacBook Pro with an M4 Max, a cold Parquet scan improved from 1.321 seconds to 0.883 seconds, about 1.5 times as fast. Once the file was cached, the difference was negligible because there was no storage wait left to hide.
Faster reads have costs and sharp edges
Using more concurrency can increase memory use and move the bottleneck elsewhere. In DuckDB's four-query S3 test, the default v2.0 development build finished in 15.6 seconds, down from 35.8 seconds for v1.5.5, but peak resident memory rose from 14.5 GB to 20.1 GB. Setting 16 GB and 8 GB memory limits reduced the new version's peak resident use and slowed completion to 22.7 and 24.2 seconds respectively. It remained faster in this test, but the numbers illustrate that read-ahead is not free.
File layout still matters as well. Very large Parquet row groups expose too little parallel work. In DuckDB's test, one enormous row group compressed the data well but left only two large fetch streams and took 25.26 seconds. Versions with hundreds or thousands of smaller row groups finished in roughly two to three seconds. Async scheduling can use available parallelism; it cannot manufacture it from a file divided into too few independent units.
Operators will also need to consider object-store request limits, transfer charges and contention with other workloads. Saturating a network link is valuable when the query is the priority, but it may not be the right default for every shared environment. The available settings for async-thread count, retries and read-ahead depth give advanced users room to tune, though DuckDB's published best result used machine-specific adjustments rather than defaults.
Preview builds are for testing, not production
The feature is available now in v2.0 development builds. DuckDB's preview installation page labels v2.0 as early-stage and warns that nightly builds are in flux and less suitable for production than stable releases. Python users who want to test it can install a preview in an isolated environment:
pip install duckdb --pre --upgrade
That command should not be treated as a routine upgrade for a production application. Results are more useful when compared against the stable build using the application's real files, network, memory limit and query mix. The official release calendar lists 2.0 for fall 2026 but explicitly describes planned dates as tentative.
Async reads currently cover Parquet and uncompressed, seekable UTF-8 CSV files. The team says JSON and DuckDB's native file format are next, while formats implemented by out-of-tree extensions are not yet on the roadmap. It is also investigating Linux io_uring, which could reduce system-call overhead and reliance on threads that block during I/O.
What to watch next is whether the large gains survive broader testing outside DuckDB's controlled setup, particularly on smaller machines, rate-limited object stores and mixed concurrent workloads. Stability under tight memory limits will matter as much as peak throughput. If those results hold and the remaining formats arrive, version 2.0 will mark a practical shift for DuckDB: remote files will no longer be an incidental extension of a local engine, but a workload its scheduler is designed to keep busy.