Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.
yeh running some automatic scripts against public urls to check if https is enabled and scanning if there are public s3 buckets is barely any better than a security linter.
There are some teams that are better than this, but they are rare.
The press release speaks of "a rigorous source-code security assessment by NCC Group." That suggests more than a bit of url scanning. Somebody is lying, that much is certain.
Well you can always move to tiger cloud if you want it managed :)
And this is PostgreSQL licensed just like our pg_textsearch which has been adopted to Googles AlloyDB and microsofts HorizonDB already, so there is hope!
It’s there. Titled “Announcing pg_vectorsearch: Scalable Open-Source Vector Search in PostgreSQL”. In paragraph: “ The result: an index that is significantly faster”
> TLDR: We are launching pg_vectorsearch, a new open-source PostgreSQL extension that makes vector search dramatically faster and more cost-efficient. Its new PRISM index uses partitioning and compression instead of HNSW's graph-based approach, delivering faster queries with a fraction of the memory and significantly faster index builds. At 100 million vectors, PRISM achieved 6 ms query latency on a machine with 4× less memory than HNSW, with index builds over 20× faster.
That said I love that they don't seem to restrict cyber capabilities to any degree and even lean into it.
If the best model for cyber attacks is open for everyone to use it just makes us all safer I think. Of course you then also HAVE to use it or otherwise you're vulnerable, which is a great distribution play.
Too late to edit, now including AA-omni Score [0] and 'by Domain' Software Engineering [1]. Also added comparison to the eyeballs-median of the 'top 10' models and then you see that indeed Mistral Large 4 Preview scores miserable in the AA-Omni indices.
Inte Cost AA- Omni
llig per Omni Softw
Open Weight model ence Speed Task score Eng
Mimo-V.26-Pro 46 47 $0.13 8 33
GLM-5.3 (max) 45 73 $2.01 14 37
DeepSeek 4.1 Flash Max 39 227 $0.27 -5 34
Mistral Large 4 Preview 38 116 $1.13 -5 5
Closed/proprietary ~50 110- $1.50- ~43 ~85
median top 10 242 $7.50
>If the best model for cyber attacks is open for everyone to use it just makes us all safer
Issue is..
I don't believe for an instant that any of us, including US citizens, get access to the best models for cyber that the US has. I think any adversary would have to assume the models in use by the US side are unreleased.
US is not the only one dealing under the table by the way, I also think everyone should take China having unreleased models as an operating assumption at this point.
So Mistral is the best that the public gets access to. And that's if it's even the best? Benchmarks and pragmatic work have often been shown to be two radically different things in this industry.
I ran it on my trivia game Redactle which features a redacted Wiki article. Mistral Large 4 is not very good. It can sometimes solve a game with ~40 guesses whereas the top models like Gemini 3.8 Flash or Grok 4.7 can one shot most puzzles. My benchmark here aligns with AA Omniscience. I also have a version where the text is rewritten to detect over fitting to exact wiki text which changes the scores but not the leaderboard order.
Gemini models are summarizing wikipedia articles all day (when being used for Google's ai answer), can we draw some conclusion from this, did they train it extra well on wikipedia content?
Yeh it's absurd that people claim this all the time. It's some crazy conspiracy theory and when you ask for examples nothing ever shows up.
It would be economical suicide from anthropic and OpenAI to actually need models intentionally.
But hey I guess it's hard with technology that truly seems like magic.
People say if you'd bring electricity to the middle ages you'd be called a witch and burned. The same is happening to the model labs here because they are bringing tech that the world isn't ready for yet.
Right I've already got MCP integration but it seems quite clunky. Do you have an example of an MCP implementation that you like? and maybe an example of code you used that did the integration nicely? Would love some pointers
Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.
reply