But which effort level should I use for the decider model?
(I think there's at least some evidence that it is difficult for decider models to predict the complexity of a task ahead of time with any level of reliability.)
Of course, invoking 'there are two kinds of languages: one that people complain about and one that nobody uses', there are two kinds of language designs: one that is perfectly consistent and one that has had contact with the real world.
What experience have people had with Mistral models for cyber research? are they as constrainted as Anthropic models? I use claude day2day but have the need for a model with fewer guardrails to pentest our own APIs.
They explicitly lean in to cyber work, and they appear to be very permissive from their marketing:
> This is particularly important in cybersecurity, where provider-level refusals can block legitimate vulnerability research and incident response, and where losing access to a capability mid-incident can itself become a critical security risk. ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies.
> That top score reflects a practical advantage. Several leading closed models, including Claude Opus 5.5 and GPT-6 Astra, score near zero on the same test because they refuse to perform the task. Yet defending software often starts with proving that a flaw is real, exactly the kind of work safety filters in closed models can block. This matters even more as threat actors increasingly jailbreak those same models to support offensive cyber activity
I have been using this super simple skill[0] to make Claude ask exhaustive questions about the implementation of larger features. It has been absolutely amazing and felt like a step change in the output quality.
For complex features there can be 10 or more questions but I have a very strong sense of understanding the changes about to be made and Claude is very good at following all the decisions exactly. It's like working with an engineer who is both excellent at soliciting requirements and fast at implementation.
That's AI getting an example of replicating the state that shows the bug which is exactly what I'm talking about. It does that far more than humans do, and it's ace. That's how you should be debugging a system - replicate the issue, understand why it breaks in that given state, and then make a code change to fix it.
Sometimes you can do that mentally and fix the code. Often your fix will be right especially in a relatively simple part of the code. However, equally often you'll fix a different problem (or something that wasn't a problem at all), and the original bug will remain but you'll believe you corrected the issue. This is why you should always replicate a bug to understand it, and why you should always add a test whenever you fix a bug to prove you actually fixed it as well as preventing future regressions.
Your site has a bunch of marketing copy, trademark symbols, and a link to pay you $2k/mo, but zero technical detail and is broken on mobile. It gives vibe coded.
The benchmark of 2m black-scholes evals/sec is meaningless without additional context. Also seems a couple orders of magnitude slower than what I’d expect even for a single core. What exact transformations are being done, and what’s the throughput in GB/sec? Is it multi threaded? Benchmark against the equivalent query in kdb+ and ClickHouse?
What additional technical detail would you be interested in?
> Also seems a couple orders of magnitude slower than what I’d expect even for a single core.
1) it's more than 2m/s on a MacBook AIR
2) Most of the latency is networking. This was a preliminary setup with a client / server over websockets where the book of 1m contracts combination of (strike, exp_date). Each insert only after a corresponding query has been returned (over the wire) from the previous insert. I can agree the eval isn't great; as in, the throughput is much higher.
reply