> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
Fortunately the list of VCs funding this with millions and clearly zero due diligence is easy to find. I will add them to the address group for my next Nigerian Prince AI startup email distribution list.
They all got played by some ai dunning kreugers. The investors might not know it but anyone who knows anything about dbs knows that failing a jepsen test this badly is not the start of something. it's the end of something.
Looking at some of the findings because I don't even have the time to pretend to read half of the failures leads me to believe that promoting this "database" should be illegal.
Go figure one of the founders is on openais board. Yikes...
I guarantee you that every investor knows they are not knowledgeable about technical details. So, they have a person on call that they’ve designated as their technical advisor (or even entrepreneur-in-residence, depending on how much the investor is paying this person).
The problem is there’s no incentive for this advisor to admit they are out of their depth because the money will stop flowing: some flavor of “if you can’t help me on this, then what am I paying you for?”.
Experienced investors, a.k.a. those who have previously been burned and learned from it will have multiple technical advisors categorized by their past career specializations.
The other investors will continue to rely on the advice of one jack-of-all-trades with questionable results.
I've been that advisor. You'd be surprised how often investors are straight-up conned by founders into becoming hype men for broken-by-design businesses as soon as the advisors are out of the room.
That's not because investors are idiots. A little bit of it is because a certain amount of credulity/faith is a necessary skill in that business. But most of it is just because humans are fallible and trickable in general. Advice from an expert can only help with that so much.
I'm part of the ParticleDB team. We took these findings seriously and have spent the past few months addressing them and strengthening our testing. We now run sustained Jepsen workloads daily alongside our own test suites. We've also narrowed the supported transaction modes to focus development and testing on core correctness.
Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.
> We now run sustained Jepsen workloads daily alongside our own test suites.
Why would anyone believe you when you say this? From the report:
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."
Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.
What kind of introspection did you do on the root cause of those issues? "Data from deleted table reappears in new vaguely-similarly-shaped table" or "simple row write just doesn't happen" are bugs that are quite impressive to accidentally happen!
How did your design process resuls in an architecture which even allows for this kind of failure? How did your review process allow for code to be released which is seemingly completely unread and untested?
"We added some unit tests" is is an extremely damning answer - the lack of tests is a symptom, not the cause. If you ever want to be taken seriously again, you'll have to completely rethink your development process first, and do so publicly.
Fixing clearly identified issues is table stakes. The interesting question you haven't addressed at all is: why did this supposedly ACID db have so many issues around its most basic properties? Smells of a design issue.
I would highly recommend leaving the industry. You are not at the caliber required for this kind of work, and you are not aware that you are not at the caliber required for this kind of work. You bring shame to our profession.
Why did you pay Aphyr to find all these holes in ParticleDB? Didn't you know about them before? I wish you the best, but this doesn't sound reassuring.
I mean I assume that that's where, say, 3.3.2 came from; the database grows _new_ bizarre brokenness as stuff is fixed.
I'm genuinely kind of fascinated whether there was a _design_ here, or if someone just said "claude, postgres but faster pls". Some of the issues are truly incredible.
If you took the findings seriously why didn’t you take the outrageous claims you were making in your own copy seriously? Why couldn’t you provide cogent analysis of these issues during the process? How did you hack up your db so bad in the first place?! This thing looks so bad bro I can’t help but think this kind of assertion only adds fuel to the fire.
> we're working on benchmarking now as well.
But you already said you were 1000x faster than Postgres??
Honestly best thing to do is to delete this and wipe out its existence and start a new project from scratch and rebuild the reputation for the company.
No one in their right mind will use this product ever.
I would guess in a matter of 1 - 2 years, a random amateur vibe coding a brand new DB will get much better reliability and correctness guarantees than whatever this is. This is just pure laziness.
Its not happening, at least not with LLMs, maybe some future technology will be better with that, but not language models. Writing such a complex software as database engine requires strategy, awareness and rigor. It is somewhat similar to winning a sweaty game of chess with an experienced opponent. LLMs are great with broadness of their knowledge, matching patterns and synthesising that knowledge, but they fail with strategy and awareness part. And this is why they are bad in chess.
It is combinatorial explosion of possible states (not as in memory state of program, but the differing visions of the next state of architecture). Plus high impact of mistakes, suboptimal solutions and randomness when switching between states.
How does this make the case for humans forever exceeding LLMs in planning and architecture? It doesn’t explain why a neural net couldn’t learn to do better than humans on this task at all. It certainly doesn’t point out any deficiency of attention, transformers, layers, encoders, decoders, or how the layers or their inputs/outputs are set up.
Yeah, good question, thanks. Llm reasoning is done exclusively in the way of symbolic manipulation. Human reasoning is done not in symbols but in mental models that are hard to explain for today science but we have evidence they are somewhat efficient for producing correct programs, at least after sufficient training. Ron's post makes the case to explain that symbolic reasoning is not sufficient for producing correct programs. Hope this helps.
> Llm reasoning is done exclusively in the way of symbolic manipulation.
No, LLMs are not a classical symbolic system applying explicit rules to symbols.
The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too. While that suggests neither humans nor LLMs can reliably solve every planning or verification problem, it says nothing about which will do better on the software problems people actually encounter.
It certainly does not present a fundamental problem with LLMs as they exist today.
> No, LLMs are not a classical symbolic system applying explicit rules to symbols.
I do not mean that LLM is a classic symbolic system that generates symbols according to rules,
but instead it's a system that generates symbols according to recorded probabilities. The focus here is on symbols: the LLM reasoning starts with token, ends with token, and everything in between is symbols (tokens). Quite different from how human brain works, where symbols are not that much involved in reasoning.
> The article asserts general guarantees of program correctness have fundamental limits, but it explicitly says those limits apply to humans too.
Yep, however I would like to assert that today we have evidence of systems architected by humans and behaving correctly (like Linux kernel), while there are no evidence of systems architected by LLMs that behave correctly. Even the systems which are architected by LLMs with human assistance (such as Claude Code or Codex) routinely have horrendous bugs (you're welcome to search "Claude Code Bug" and "Codex Bug" on this very website).
> It certainly does not present a fundamental problem with LLMs as they exist today.
The article was created 10 years ago that is why it does not address LLMs. However it asserts that mathematics modeling is not that helpful for writing correct programs. As long as LLM reasoning is closer to mathematics modeling than human mental modeling, LLMs will be worse at writing correct programs than humans. Some future technology less focused on symbolic reasoning could change that, but today it's not on sufficient level.
Holy shit so are you confident that “Its not happening, at least not with LLMs, maybe some future technology will be better with that, but not language models” or not?
> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
reply