I think it's useful for implementing Sagas https://temporal.io/blog/saga-pattern-made-easy

Basically, a multi-step event handling system, where the data could be spread across multiple databases/systems, so there's no way to rollback a transaction atomically across all the databases. Instead, you explicitly codify "compensations" to undo your previous commits so that you eventually end up in a consistent state.

Temporal is the orchestrator/framework/library to implement the above in an easier way.

Does it actually work?

In a past job, they tried to implement a similar thing on much lower scale with bidirectional database migrations.

Fortunately, they were mostly ran in one direction.

IME, it works, but at the cost of a pretty large amount of overhead, both in performance and (I think more importantly) cognitive.

Honestly I never did come to a satisfying conclusion on whether I thought it was worth it. The teams I worked on that used it found it neither simple nor easy to use, and I was never sure whether we were really reaping the benefits of correctness in the face of failure at the scale we were running. We eventually migrated everything off of it, and everyone was happier, but perhaps (probably) we had more lurking bugs in distributed failure cases. But to my knowledge we never tracked down an incident to the kind of problem that temporal solves.

I guess I'd say that I like durable execution (or at least temporal specifically, it's the only system like this that I've used) in theory, but not really in practice.

Yeah, I’ve seen it working. It does work well to implement a multi-system transactional workflow broken down into retryable steps. So, eg, if you are operating a SaaS and need a workflow for new customer signup to wait for multiple steps that might last for hours or days like: validate credit card info, wait for email confirmation, add entries to customer database, set up workspaces in downstream systems, spin up cloud resources, initialize systems, notify customer, start billing cycle, etc. and if any step fails, you might want to retry or send alarms or internal notifications, or roll back other steps, or kick off other workflows. Then you can define all of that, in code. Each step can be different runtimes run on different worker systems, etc, all coordinated through a central database.

You can scale the workflows up and down to as complex or simple as you want for whatever your business is. The example above is one I’ve seen, but it works on a smaller scale as well. But you have to write the implementation code in very specific ways to get the benefits, and the overhead ends up being very difficult to plan and reason about, and a lot of business logic actually gets hidden amongst the weeds of the tool’s overhead.

The fintech (now bank) that I worked in couple of years ago utilized this pattern pretty effectively.

The overhead of doing something in this way is quite large, but, as this handled literal money, it was definitely worth it.

Hugging Face's internal storage system is more sophisticated now after they acquired XetHub. They have a clever way to chunk and content dedupe the data that makes it a lot more efficient to version control large files.

Xet's tech is very interesting and I greatly enjoyed their blog posts and other technical writeups[0]

But in the era of frontier models and hardware designed for fast model loading (eg: LLM on a chip) I can't imagine that this is worth so much. There's *a lot* of competition to improve Git LFS.

So while I agree with you that this plays a role I also have to believe that it's to understand model usage by competition and thus stifle it.

[0] - https://huggingface.co/docs/hub/en/xet/overview

git lfs just dumps your bin on s3, and is abandonoware since ms bought github, since github historically hosts on aws s3 and improving it would immediately benefit amazon. gh, the maintainers of it, is the only forge that does not allow clean-up unreachable artifact from the server.

Surely you could now vibe code such system for under $1,000.

I highly doubt it, there's about a million things that only show up at scale that aren't lessons the bots can easily digest from Medium posts and bypass

You are assuming the operator of vibe coding tools is not experienced in the domain.

There's more to than just the website. You have network bandwidth costs, security costs, infrastructure costs.

You can't just spin up a VPS and throw a few brain jar's in; the network bandwidth must be a fortune. Hosting over 700GB+ for many of the same models has storage cost too.

That is fair point, but still you can build such system for under a grand and then pimp yourself out to VCs.

Dumb question, but how do they know if a codebase is actually written by an LLM? I know that sometimes code agents write "Co-authored by" in the commit messages, but this is something you can over-write.