What is extreme scale here? Like you've been able to fit over 1 PB of NVMe in 1U for years now (and before the AI inflation, prices were low enough that you could do it for like $100k IIRC). Are we talking 10s+ of PB? Who actually needs this scale?
At mere 100s of TB of data, I think you'll find that a normal database is far more likely to scale better. Most developers are... not as good as database developers at maintaining a database, and will not do a good job of it when they write their own inside of their application. They're often not even that good at using a database.
e.g.
> You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further
You can just do this with postgres. It works great. Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?
Not just data, but concurrent operations per second for writes and reads involving seperate transactions. Eg: Scaling concurrent write transactions on different shards with minimal write lock contention.
You seem to have missed the final sentence beginning with "Note:" in my original comment. Nobody said you can't do this with postgres.
> Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?
What most people get right is not my concern. If you find building a thin wrapper layer for your application as hard, clearly this may not be applicable to you.
Okay, again, what kind of scale are we talking about here? 100ks of writes/second? Millions? For data that can't do reads from a replica (presumably not since you're saying use eventual consistency)? Always absent from these kinds of blogs are concrete numbers for what kind of volume they're discussing, making them completely useless drivel. Also that blog is saying they were running into trouble in early 2014 and started this project; okay, the first NVMe drive was mid 2013. Did they do that first? They totally could've been using racks of 10k RPM disks around that time with maybe SSDs for the indexes. You'd be way better off going all NVMe today as your first step to "scaling" (in fact you are insane if you are using anything else for an OLTP database), and just that is likely to immediately solve whatever problem you have already.
If postgres can already do partitioning, batching, different tablespaces on different disks, etc. except do it correctly with ACID, then what exactly is the benefit of manually implementing what it already does well?
Also how does going "schemaless" help with scaling at all? That's just asking for data integrity issues for no benefit. Peak mid-2010s money-furnace resume-driven "engineering."
> Okay, again, what kind of scale are we talking about here? 100ks of writes/second
You seem to be confusing writes/second with concurrent write transactions/second. Nvme won't rescue you if Postgres locks up only allowing each write txn to proceed sequentially when what you need is SSD write saturation on separate write transactions on multiple shards in a single machine.
> You'd be way better off going all NVMe today as your first step to "scaling" (in fact you are insane if you are using anything else for an OLTP database), and just that is likely to immediately solve whatever problem you have already.
This is hilarious. So we should all start building based on your prescription.
> what exactly is the benefit of manually implementing what it already does well?
Really? Is that a serious argument. Postgres is not a magic database that does everything that everybody wants very well.
In systems engineering, the trade offs are obvious. But if it is not clear at this point, I don't think I can offer much help.
I will offer one last advice.
Just write a snippet of code inserting 100M small keys and values into, say 1k chunks/shards of separate kv engine (LMDB, whatever). Do normal hashing for shard selection. Now store secondary indexes likewise. Now do parallel fanout for secondary indexes for selecting some 100k data. Collect the index data and do another parallel fanout and fetch the final data.
Now go see the latency results (excluding lmdb env setup time). Additionally observe the latency for writes :-)
Rich index metadata is just icing on the cake. Cheers.
The point of batching is that high requests/business "transactions" don't necessitate high concurrent database transactions. The thing that matters here is actual workload. How that turns into transactions is something you can tweak. And once you have a decent workload, you can build batches with dozens of items in a couple milliseconds, so you can still keep single digit millisecond latency with batching.
Yes, if you're architecting around slow disks instead of just using fast ones (which cost a fraction of the development cost of what you're saying), and if you do things like give up ACID in service to that, you're crazy and incompetent.
It's not magic, but for 99.999% of people, magic is not required. So again, like I asked twice, what is "extreme scale"? Weirdly, no one who brags about "scale" is ever willing to say what that means, and everyone saying they've never encountered this "scale" is willing to put concrete numbers behind it like 100s of TB at ~1M QPS or whatever.
At 100M rows in a table pg can do just fine with single digit ms latency with 5 digit request rates. Is that your "scale"?
At mere 100s of TB of data, I think you'll find that a normal database is far more likely to scale better. Most developers are... not as good as database developers at maintaining a database, and will not do a good job of it when they write their own inside of their application. They're often not even that good at using a database.
e.g.
> You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further
You can just do this with postgres. It works great. Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?