I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything.
The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.
Thanks for the clarification! That's our main idea. I think a shared state with many independent questions is the nature of decision tasks, so we push it to the extreme. This is a decision-native design.
So, if your problem is: tag these 10 000 emails (work, spam, personal, English, French...) it's fast? (Feed large context of mails (input) and question context).
If it's answer if this one email is spam - not so fast?
Currently, the speedup mainly depends on the number of questions per input. The more questions for each input, the lower the average time per question, as the input is computed only once by our engine.
For tagging one email is spam or not, our engine can still be up to 3x faster, but the speedup is not as significant as with multiple questions per state.
So max benefits would come from picking an email and asking "is this spam, is this in english, does this contain grammar errors, does it contain a signature, is it a commercial pitch"?
Thanks for the questions! I think tancop's summary has already captured the essence, and I can add some details.
> Under what specific circumstances is it actually 9.5x faster?
The 9.5x is Qwen3.5-0.8B on QuALITY, one long article that has 5.8k tokens and 18 questions on average. The more questions share a context, the bigger the gain.
> Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
Yes, with a state that has only a single question, it ranges from a near tie with 27B on a long input to more than 3x with a small model on a short input. In this case nothing is shared, so the gain only comes from lower per-request overhead, since Flide runs the whole forward pass as one prerecorded CUDA graph.
> How did you achieve the biggest speed up?
The big speedup comes from cases where one state carries many questions, where we only compute the shared context once. We achieve this by using a mask that keeps the questions from seeing each other, and each question carries the correct position index just as if it came right after the state, so we can get the answers in one pass. The second is the prerecorded CUDA graph.
> Are there any tradeoffs? "While keeping exactness" implies not.
There is no accuracy tradeoff, since each question gets the identical computation as if asked alone. What remains is floating-point rounding that shifts slightly with batch size in every engine, even for a question asked alone, and our results are still the closest to the FP32 results among all the engines.
> Are there any specifics about how one must structure their context to take full advantage of it?
To get the most out of it, you can put everything shared in the state and ask all the questions in one request. And for continuous agentic decisions, use --session-cache-gb to get a further speedup.
In general, traditionally there are underappreciated curators that take on the role of keeping a named package or program consistent, and may go though a personal validation process with a OS build organization to provide a chain of accountability for the user-base.
The amount of abuse these people endure from general users is a classic rerun. =3
Are you asking a bad comedian to explain their joke?
My only active FOSS project right now is a unpublished 3D volumetric slicer for a home metal printer that won't be public for another 2 to 3 years for various trade reasons. I code mostly for fun these days, and the dude doesn't sound like fun to work around. =3
I have no FOSS projects, I'm not sure I entirely agree with it all, there's an element of "getting developers to create free stuff for corporations". I notice there are no "FOSS" architecture plans, or any other professional output. But I digress. Good for you :)
We're still figuring this out. We'll need some synthesis equipment (think CVD, PVD etc) and characterization (XRD, Raman spectroscopy) tools in-house to validate that we're making the right materials. We're considering developing these tools in-house - the models sometimes come up with clever modifications to them so that they can deposit new materials. We think equipment is as central to new material discovery as the material itself, and will probably need to be rethought to allow for high-speed AI based experimentation
Why would this be the case when the text that produces binaries (code) is usually both more token efficient and vastly more effectively organized for modification/extension?
Unless by bits you just mean text in general, or any data since it’s all bits, in which case what you’re saying is trivially already true.
It seems like you’re saying that long term LLMs will output pure machine code as the most effective way to use them.
What about the other thing that’s the actually important part of my reply, modifiability/extensibility? You just chose the weaker of the two factors I provided. And didn’t even provide a convincing argument related to it.
You also didn’t clarify your position at all.
Explain why there is any benefit to outputting raw machine instructions compared to writing shorter, more interpretable, modifiable, extensible code and then using a compiler to turn it into machine code.
Why are agents not going to use compilers in the future?
Almost every possible name that is not a portmanteau, made-up word, or combination of N words has a naming "conflict". This is not an interesting thing to say and I wish that people would stop saying this on everything anyone ever posts on HN. It almost feels like it should be against HN rules to point out that something else shares the same name with no additional statements.
For one, it's interesting to me because I've already known Hoplite for years as nothing to do with AI. For two, I'm not sure how sharing that is so egregious it should be against the rules? Is there an interpretation I'm missing of my original comment? Does pointing out another Hoplite get interpreted as disparaging or accusatory in some way? Does it go against intellectual curiosity?
The term "conflict" implies "there is a problem here". I would say it's very lightly disparaging, because it implies the author didn't even do basic research on other things that are also named the thing they decided to call it.
I actually checked HN rules and just saw this:
> Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.
At least for me, I often have a hard time believing people’s stated reasons for whatever they’re doing.
The substance <> narrative relationship is backwards a lot of the time. Someone does something for a nebulous multitude of reasons, and then post-hoc fits their decision-making into a logical explanation that sounds nice.
There is some psychology research supporting this as well.
I suppose if you see the narrative itself as part of the release, that might be interesting. But most of the time I’d rather just hear plainly and straightforwardly what the thing is.
Or at the very least, I am very accepting of releases which do not include rationalization / narrativization and don’t think it’s required to include.
Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
How did you achieve the biggest speed up?
Are there any tradeoffs? “While keeping exactness” implies not.
Are there any specifics about how one must structure their context to take full advantage of it?
reply