Surprised that https://github.com/ucbepic/DataAgentBench hasn't been mentioned on this: benchmarking data agents that need to query from multiple sources is a focus of some research at UC Berkeley -- leaderboard is worth a look to see how the various closed and open-source implementations build ontologies ... would like to see how Nvidia's approach stacks up.
It's at the end of this Netflix post: but the end game is to connect whatever your observability ontology is with whatever system the business people and data engineering teams use to be able to answer previously hard questions like "how much money did this operational or engineering change make or lose?"
In theory, at least.
Observability vendors have been beating the drum for 10+ years that there's so much untapped business value in operational data. Seems like we might be getting close to seeing how real that is.
For context, we previously built custom catalogs optimized for specific use cases. But they were hard to maintain, especially as requirements changed, and Apache Iceberg was too heavy for our specific low-latency work.
Since Ducklake is only a spec, we implemented datafusion-ducklake, and it performs as well as any custom or specialized catalog we built. We use Postgres as the catalog store, and it does not get much simpler than that: a transactional database for transactional data.
Plus, it gives us a clear spec for implementing complex parts like time travel, snapshots, etc.
I always thought the catalogue was a duckdb file. E.g, data lives in partitioned parquet files, but which parquet files are current or soft deleted, etc, etc, is managed in a duckdb data file.
However, looking at https://ducklake.select/, it seems the catalogue lives in PostgresSQL - so it is not really a ducklake, but a postgresslake.
The fundamental requirements of DBMS so it can be a DuckLake catalog are the following:
1. It must support primary keys
2. It must support basic data types (INTEGER,VARCHAR,TIMESTAMP)
3. ACID
There are ofc some quirks from the SQL supported on each DBMS. Hence, the DuckLake extension from DuckDB currently supports DuckDB, SQLite, Postgres, DuckDB + Quack and MySQL. With MySQL being in a rather experimental state.
This is a lot to like here as someone who pretty much exclusively uses the Cloudflare API via agents. Hoping (since it's built on top of Forge), some kind of native terraform support is also in the works.
Great when you're a solo dev deploying a worker, but would be incredible for production deployments if this could also just natively output terraform code.
Feel like AWS got a lot of the concepts right with S3 Tables* ("serverless" Iceberg catalog on top of S3) ... would love, love to see a version of S3 Tables that lets you choose between Ducklake or Iceberg as the catalog.
* Cloudflare also seems to have gotten the same idea right with R2 Data Catalog