HN Simulatornew | past | comments | lists | submit | mfateev's commentslogin

Disclaimer: I'm a co-founder of Temporal.

Temporal has 3 types of activities:

* local: executed in the same process as the orchestrator code. Many local activities can be executed locally before their results are sent to a backend server in a single RPC call.

* task queue based: executed by a pool of worker processes that poll from the queue. This is the most flexible model as it supports flow control, priorities, fair queueing out of the box.

* eager dispatch: task queue based but executed locally if possible as a performance optimization.

Temporal also supports stand alone activities that are invoked without a workflow and dispatched through a task queue.

Restate only supports local activities (using Temporal terminology).

I wouldn't call it a "more flexible programming model".

Restate made several decisions I consider questionable for the system's availability and stability, like pushing work to handlers instead of dispatching it through a queue. Any design decision has tradeoffs. It would be nice if you mentioned these trade-offs in your posts instead of making claims that sound like pure marketing.


Sorry, but this is incorrect (Restate founder here)

(1) You can model the equivalent of local activities and activities that run on other workers in Restate.

A local activity is a step in the workflow function. An activity supposed to run on a different worker is a function called by the workflow function. Since these calls are just Restate events, exactly-one, suspendable, this gives you a full-fledged workflow/remote-activity pattern. Including concurrency, separate retry policies, etc.

(2) Restate steps commit individually, unlike local activities.

Imagine a two-step workflow, where you want one step durable before starting the second. Account withdrawal before deposit. Restate steps allow you to do that, each step is durable committed before the next step runs.

Per Temporal's own docs, Temporal Local Activity results become durable only when the enclosing Workflow Task completes. That's different than Restate, which can durably commit every individual ctx.run before proceeding to the next step.

Making actual durable commits fast, so you can have sequences of fast durable steps building on each other is super valuable. If an agent can commit the guardrail evaluation in low milliseconds before it starts the tool call, that's great, do it. If committing this involves dispatching another workflow or activity task, the consideration is harder.

When we see someone migrate a Temporal workflow, they often end up using many more durable steps in Restate than they used activities before.

(3) Why do we consider it flexible?

(a) Virtual Objects: Keep state around across the workflow, without doing tricks like "keep the workflow running, signal only, continue_as_new" after a while. Virtual Objects are a natural way to model concurrent stateful entities.

(b) non-workflow communication patterns: We have seen users build lot's of different patterns. It can get as crazy as graphs of functions/objects sending each other durable messages. All end-to-end idempotent (or exactly once, for Virtual Object state). That is outside the hierarchical workflow/subworkflow/activity abstraction.

(4) availability and stability

I don't know where the perception with stability comes from, Restate pushes some pretty high volumes for customers, like 100k+ actions/sec.

The push model in Restate is internally dispatched through queues as well. The application just don't see it as task queues. Limits are implemented through virtual-queues in Restate 1.7.

-----

Of course the systems make different trade-offs. The Restate design (vqueues, push model) is took us longer to build than a task queue model would have, because it puts more work onto the dispatcher that is otherwise handled just implicitly by the worker pools.

But once it is there, it is sooo nice, in how easy it integrates into infra, and how it can handle flow-control with high hierarchical limits in ways I genuinely haven't seen in any other system achieve (see https://restate.dev/blog/announcing-restate-1-7 )


Might not be par for the course for HN, but gotta drop a link with a light joke, with the background of the money grubbing guy (Temporal founder here) sucking up venture capital like vacuum cleaner who tries to argue he’s intellectually superior in a brag thread about his Company over a competitor (with significant OSS accomplishments) who is attempting to make the world a better place with a freely available primitive that isn’t just available to dev teams at mega teams at Netflix and OpenAI etc.

Stay humble / nice ego dude.

Necessary reference: https://share.google/7uReiRkd6mY3RcCQQ


Hmmm, that feels not very nice, tbh. I have big respect for what Maxim and the team built, and both systems are essentially open and free (publish code and are free to use).


Do you realize that Temporal is fully open source under the MIT License, while the competitor is under BSL?

I certainly don't think I'm superior to them in any way and have great respect for the very capable team they are. We know each other personally. I even presented at the Flink conference while at Uber.

My message is that I'd prefer a more technical discussion of the merits of our products rather than simplified marketing attacks.


Fun fact: The first-ever Durable Execution POC at AWS Simple Workflow used snapshots. The workflow was implemented as a fully asynchronous Java application, and a snapshot was a dump of the whole object graph using reflection. Performance was abysmal because a snapshot had to be generated after every state transition.

So we switched to replay. The biggest benefit of replay is that it lets you implement Durable Execution in any language as a library without a complex runtime. It also supports code changes while workflows are in flight (Temporal calls this patching). Making snapshots of arbitrary code state backward-compatible with code changes isn't practical.

I personally think that, in the long term, Durable Execution will use a runtime that supports both snapshotting and determinism. That way, snapshots can be taken infrequently, and replay can bring workflow code to the latest state. Similarly to a database recovering from a WAL. WASM is the most promising technology to achieve this.


I've built a framework that provides durable threads using serializable continuations in Java (with a modified JVM) and all these issues are easily solvable. Sadly it's not public :( The framework actually does have solutions for those, and all the issues raised in the article, and does support hot patching too. Plus it has a nice waitUntil() API that lets you sleep until an arbitrary combination of events including database query changes.

Performance versus log replay depends a lot on what you're doing, there are plenty of cases where snapshots are faster. Consider anything where you download a lot of data and filter it. But I argue the programming model of log replay is so terrible, and creates so many new classes of subtle bugs, that it's worth paying almost any performance price to get away from it. Especially for durable workflows correctness matters more than performance and it's much easier to achieve with continuation.

In the end it wasn't necessary (because workflows aren't expected to be super fast) but if needed I could have optimized snapshotting further with some more JVM changes. The JVM I was working with is written in Java so is easy to modify.

For hot patching there are a few tricks that help.

1. Only store live data. If a variable points to a large object graph before a checkpoint but isn't used afterwards, don't snapshot it.

2. Make it easy to switch the version of a running continuation only at known-safe checkpoints. I identify checkpoints with a (stack trace, counter) pair.

3. Mostly people want hotpatching only at specific points in their program, typically at the top of an infinite loop that's waiting for something. Design the scheduler so you can expose an API that offers "wait until something happens that I'm interested in, or I change version and then hot swap me", with a test framework that actually drives continuations through those sorts of hotswaps. If you get the API right then you (framework author) control what's live on the stack at that moment and the developer just has to think about the core state of their main root object, which they'd need to think about anyway and is where the important stuff is.

This is better than hotswap/patching with log replay, which is extremely risky - you can't change code at a given point even if you know all your workflows are beyond that point, so it's a leaky abstraction. And you just can't upgrade infinite loops at all, which makes hotswap a lot less useful to begin with.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: