(I left Meta in September 2022, so I haven't worked there in four years. My current coworkers can attest to the fact that I'm not proud of working at Meta, but I am proud of the world-class nature of what our teams built. In any case, I don't believe my personal feelings about Meta have any bearing on the matter.)
This is very similar to the way SCMQuery existed before Mononoke and eventually became served by it. The source control team at Meta was, and is, quite familiar with hybrid deployments of the kind you describe, though of course at longer timescales.
Thanks for the comment. To the best of my knowledge, Piper is not Merkle DAG based, and instead uses a (completely linear?) sequence number similar to Subversion. I believe they ended up faking those hashes for Fig. I'm not actually sure what they do for Jujutsu compatibility, maybe they've added a Merkle layer since.
We were compelled to model things as a DAG to be able to handle the occasional merge commit (this makes the algorithms a lot harder), and we wanted the Merkleness for its desirable properties such as being easily cacheable and straightforward to do integrity checks on. We wrote a bidirectional conversion layer between the Mercurial and Bonsai data models, with great attention paid to complete reversibility. (That's how I got into property-based testing, by the way! My coworker also used Alloy to formally verify the data model.)
The closest equivalent is probably Monotone, actually, though it only had one tree/manifest format which was baked into the changeset, while Mononoke got rid of that and _only_ relied on the changes, flipped the pointer directions, and generalized that to arbitrary kinds of derived data. (This was at the cost of having to maintain mapping layers in a database.)
I'm not convinced that Merkle DAGs are useful or worth the bother. Piper's system of linear indices has a few nice properties:
1) when a defect is identified in a library, the extent of the defect becomes an interval [CL that introduced it, CL that removed it] and it makes it trivial to test whether a compiled artifact is affected by that defect; more generally the UX is a lot better
and 2) it makes whole-repository rewrites trivial, so that existing commit sequence IDs are preserved while retroactively removing sensitive files that shouldn't have been added to the repo. Git and Mercurial make such rewrite a huge pain in the arse, while this is an essential feature to have in an enterprise VCS.
These were both solved as part of the data model via globalrevs and obliteration respectively. But that is besides the point here? The next time you're designing a source control system you're welcome to pick your favorite points in the design space.
I'm not aware of anything published, but the code is open source, and if you're so inclined you can probably point a clanker at it. The second one is called "redaction" in Mononoke, I believe (obliteration is a different term for it, as is censorship).
You are correct that integer sequence numbers and redaction are both reasonable things to want -- we just believed (for a variety of reasons) that the core should be Merkle and that these things should be built on top.
Just because you might find this interesting: Scion deviates significantly in this regard in order to deal with merges; coming from git, we land code to our main branch via merge so it's a common occurrence.
We do discourage (but not block) some trickier DAG things you could have done in git and are probably moving towards squashing and rebasing for landing since many things become simpler without merges, but we simply had to deal with merges (and handle git compatibility while migrating) to pull this off. Particularly, full DAG merge bases add a lot of complexity and it gave us some concrete ideas on how to improve git's commit graph data structures by an order of magnitude or more.
Thanks for the response, Durham. I hope you're doing well. I've always appreciated your magnanimity, but I want to note that this belief is not unanimous among the current source control team at Meta.
If the original idea was in those projects, Anthropic "forgot" to credit them too. One wrong doesn't give an indulgence for other's to copycat that wrong action later on.
To be clear, I don't believe either of these were wrongs. In my opinion, this is just a symptom of Facebook's PSC process training people to clout chase and demand recognition.
Perfectly aligns with the V-Capitalist/Zuckerberg mindset: First you salvage them (like Facebook did salvage MySpace and others via interoperability), then you build walls (walled garden) to keep others from doing the same as you did, complaining about it via PR, and lobby law-makers to forbid interoperability to keep your money garden safe. Never mention who's shoulders you stand on or how you got there. Propel the myth It's all made by hand from the mud by the single 'genios' CEO-archtype - that sociopathic mindset infects the whole company and, if these companies are successful on the market, infects the whole economy, then society until we are where we are.
Thank you for responding. With the new footnote [1], I now believe the attribution is sufficient. I've added a link to it, as well as to your response, as an update to my blog post.
> Update: Nick G. from Anthropic has responded on Hacker News, and sufficient attribution has been added to Anthropic’s blog post. I stand by my posts as of the time I wrote them.
This is such a weird turn of phrase. You _stand_ by your posts, implying that you currently stand by them, but then add "as of the time" that you wrote them, implying that you no longer stand by them? Either you stand by your posts or you don't; you can't have it both ways. It's just a little confusing.
So you are claiming that if someone says “it’s raining” while there is precipitation, and then it stops, they can say it is “no longer raining” but aren’t allowed to believe it was raining before?
Except the resemblances extend to the entire server, not just the idea of derived data. The idea of derived data becomes apparent only once you have bought into the core Mononoke insight about Bonsai changesets, later copied wholesale and presented as the core Scion insight about Scion changesets.
As I suspected at the time of writing the post (but could not verify) and as was later confirmed in private correspondence to me, Anthropic did not come up with this from first principles. Claude was pointed directly at Mononoke and asked to re-engineer it.
Thanks. Indeed, the Scion design is quite good because Claude was pointed directly to Mononoke and asked to analyze the code base and copy it. I am very proud of what our team built by making contrarian bets at exactly the right spots.
It isn’t something I care very much about, except when it’s used as a certificate of achievement. I almost never do literature searches with an attempt to dedupe. I only ever do it to see if someone has explored something better than I have. That’s often the case but other times it is not.
If they knew it from working at Meta, it wouldn't be O(n), it would be O(1).
I agree that there will be cases where you implement tricks that others have had before you, without knowing. A few years ago, I had a delightful idea, experimented and wrote it up. One commenter directed me to a post that described it as "folk knowledge". There will be other cases where you encountered an idea before, but didn't remember it.
Still, crediting what you can easily know you cited is good. Even aside from credit, comparisons to existing systems are part of how we understand what a system does.
Absolutely not. When I create projects I only credit other existing systems if they are a direct inspiration. If I independently come up with the same idea as someone else I see no reason for me to do the due diligence and credit them.
Okay, well, in this case (even accepting your argument) I've been told there is direct overlap in contributors so there is no feasible way they couldn't have known.
What happened is that it wasn't a FAANG engineer replicating what they know, rather one asked Claude what to do and it told them to replicate what it knew.
That's how the knowledge that it was a facsimile was lost.
Thanks! Yeah, I think overall several more organizations are going to reach the scale that Meta reached a while ago. I'm not sure if they'll go in the monorepo or polyrepo direction, though; I can see arguments both ways.
reply