embedded analytics

DuckDB

DuckDB is an embedded analytical database. This profile covers what it does, how it is used, and where it stops being the right choice.

What the product actually does

DuckDB is a columnar SQL engine that runs inside your process, like SQLite but for analytics. There is no server, no connection string and no cluster; a Python script imports it and queries a Parquet file directly.

The performance on a single machine is the surprise. A columnar engine with vectorised execution reads a few gigabytes of Parquet faster than a distributed cluster that has to shuffle the same data across a network.

How it is used and what it costs

It is free and open source. The cost is the machine it runs on, which for most analytical work is a laptop or a single mid-sized instance.

The pattern that has emerged is DuckDB as the transformation engine inside a pipeline: read Parquet from object storage, transform, write Parquet back. No cluster to schedule and no idle cost between runs.

Where it breaks down

Concurrency. One process owns the database file, and multiple writers are not supported. A workload with many simultaneous users needs a server, which DuckDB is not.

Data larger than memory works but stops being fast. The engine spills to disk and the advantage over a distributed system disappears once the working set no longer fits.

What switching costs

Very low. The data is Parquet and the SQL is standard, so moving to a server-based engine is a connection change rather than a migration.

Sources

DuckDB public documentation, read October 2026.