DuckDB 2.0.0 LTS “Cinnamon Teal”

> Not just reading data from S3, but the actual computations too.

Async io generally causes a CPU bump as a side-effect, when the bottleneck moves from the disk to the CPU (more usually memory bandwidth).

My experience is from a different SQL engine, but the improvement in scan rates directly translates all other operators getting more work to do per unit time.

More CPUs pegged at 100% is always a good sign of smooth flows, but optimizing further requires data layout changes from that point. Even if it is a sub-linear speedup out of concurrency scaling - more with more is still a win, assuming you can pay for more (the Dy-skew paper has an example I can quote).

I’m a huge fan of splink - and how you’ve been able to adopt duckdb.

Do you find that with the performance of duckdb, you see large deployments of spark based workflows less often? It seems hard to justify the complexity of spark if you can use duckdb , but I’m wondering if you see end users still reach for spark for any reasons?

Here's a (vibe coded) Metabase driver that speaks Quack: https://github.com/dragonsahead/duckdb-quack-metabase

Being able to put it behind an HTTP service / load balancer and implement horizontal scaling based on resource utilization, or even scale to zero via Cloud Run etc. Drool!

Yes! And the server isn't really a server, it's still "in process" so it stays lightweight and simple. This is absolutely fantastic.

Back to the fundamentals! Triggers it is, indeed.

try this https://duckdbsnippets.com/snippets/223/exportimportshare-du...

> If it just had KQL syntax support, it would be perfect…

Good idea for an extension, especially with the new PEG parser.

+1 on kusto query langauge. It's much nicer than SQL.

> What are you guys using it for?

Transparent queries on server/browser in identical sql.

If you play all the tricks Google's Capacitor does to the data layout, you can shove ~2-3 million rows to a browser based query engine in 16-20Mb with duckdb-wasm. We can scale it up to ~1Gb with HTTP range queries, with cache-control.

Then we use Jev to write SQL queries over it in 1-2s, whenever someone has a followup question. Getting Jev to write SQL takes a bit of prep, but my co-founder wrote a paper about clairvoyant query execution a few years ago [1].

I could have 3000 users logged into the same application, but all of them would be bringing their own CPU to run duckdb. My compute scaling problems for my data cube is a solved problem with this approach.

[1] - https://dl.acm.org/doi/10.14778/2733004.2733064

do you have not-big big data* that individuals or sub-teams/groups use? Lots of companies have centralized data <things> (bases, warehouses, lakes, swamps, etc) and runtime security is a challenging issue (i.e row-based security). If your audience measures their data in GB or MB, DuckDB can be an excellent way to punt on the security challenges and just provide them with ALL their data - maybe even locally - while they still get first-class DB tooling and performance (or even better than classic client-server). Example - old way: build a bunch of dimensions and facts for marketing, manage their access and permissions, produce reports, make sure you don't leak data they shouldn't see. new way: write (explore using AI) some extract/transforms from your SoT & create parquet files, use duckdb to serve that up to an individual or group's reporting solution or adhoc usage. IME that one-time investment is not that big and replaces a lifetime of scaling & access issues, or worrying about someone writing a query that doesn't join with the user table correctly!

* IME most companies don't actually have BIG data, but are talking about (at most) a few TB, which DuckDB is fine with.

I've used it for data migration combined with Jupyter Notebook.

While I'm fully aware of DataFrame, I am more familiar with SQL. So I used Python scripts in Jupyter to load Excel sheets into DuckDB, and then run queries in DuckDB. Eventually I export the data back to Excel or SQL Server.

Not the most efficient way to do this I know, but it works for me.

Embedded into .Net monolith app to replicate MSSQL data into it and do queries around this data, because MSSQL is SLOW on those queries =)

we get these hundreds of thousands of lines cost exports on a daily basis from our cloud provider

we process it 'daily' convert it from csv to parquet with a little post-processing for attribution, and have a small duckdb-powered dashboard/wrapper for reporting and dynamic queries

I’ve got a number of Claude scheduled workflows that pull data from our warehouse to make a dashboard. I have it add an embedded DuckDB to the artifact and load all data into there. Makes it easy to diff and extend the dashboard.

I recently used it for a Rust profiler which writes Parquet files, so DuckDB then gives an easy SQL interface for things like summarization, analysis, and joins across runs.

Use it for viewing csv and json files. Also, converting between formats can be a one liner.

You can use it to connect every single data source that's around your company and then use it as a federation engine (e.g. spreadsheets!)

Write a log processing/aggregation tool!

[flagged]

Helps use all of the available bandwidth by not blocking (otherwise idle) worker threads waiting for data to arrive. Requests and processing are managed independently, so you essentially get better utilization & improved performance.

highest impact is when there is network overhead, like accessing them over S3/HTTP/S, and you're accessing many files as part of a query, like using a glob `-*.parquet` pattern. I imagine the same applies when you're doing local I/O, but not as pronounced of an improvement due to lower latency.

The DuckDB format supports Async I/O.

Disclaimer: I'm the DuckDB developer who implemented Async I/O for parquet, csvs and the DuckDB Format.

For now, the latter:

> Other formats, such as JSON files and DuckDB's native database format, use synchronous I/O.

For the future:

> Most of its benefit comes from accessing remote data, but cold local reads can also benefit, albeit less. Next, we plan to add async reads for JSON and DuckDB-native files, as these are the two other formats most relevant to DuckDB core. Formats that live in out-of-tree extensions are not on the roadmap yet.

You might be able to make transactions work but it's really the opposite of what duckdb is designed for. Do it only for fun.

We're doing this now...

https://github.com/shreeve/duckdb-harbor

As long as its a single writer, you can get pretty far