I’m a huge fan of splink - and how you’ve been able to adopt duckdb.
Do you find that with the performance of duckdb, you see large deployments of spark based workflows less often? It seems hard to justify the complexity of spark if you can use duckdb , but I’m wondering if you see end users still reach for spark for any reasons?
There are a variety of people still use Spark, but this is usually due to constraints on their platform/infrastructure, not because it performs better. For instance people on Databricks can sometimes find it hard to use DuckDB on large machines, or they're just more familiar with Spark.
For a new user who has flexibility over where to run their workloads, we'd always recommend DuckDB. This is particularly the case now that:
- Splink 5 (released in September) allows you to run your workload in n chunks, meaning you can distribute it across multiple machines even on DuckDB
- DuckDB 2.0 parallelises better on large machines, so throwing a single massive machine at problems is now more viable
In general, after using Spark for a long time, I think for most workloads, it's just not really needed any more. Too complex, too difficult to tune. Perhaps if you have genuinely massive data. I've written a bit more about why I advise against Spark in most instances here: https://www.robinlinacre.com/recommend_duckdb/
I'm attempting to build a coffee bean distributor that can exactly measure out beans into a cup for my morning espresso.
It's really an excuse to get started with things like hardware, 3D printing, and embedded development - I've never done anything in that world before, and its been really exciting to get into! I've just started, so hopefully I'll have a better update next month.
I won't be getting extra credit! My plan right now is to go by weight, and my first attempt is going to have beans in a hopper, moved by an auger that slows down as it approaches the target weight ( ideally moving a bean at a time at the end, to get within a bean of the goal (ideally).
I use it to help with learning a language (French) and have been leaning into the CarPlay app. The model has done surprisingly well in conversations where it has to go back and forth across languages in the same response, and I like “talk to it while I’m driving” format a lot
Do you find that with the performance of duckdb, you see large deployments of spark based workflows less often? It seems hard to justify the complexity of spark if you can use duckdb , but I’m wondering if you see end users still reach for spark for any reasons?