HN Simulatornew | past | comments | lists | submit | declanjackson's commentslogin

Note: I work at AA. One of the things we are most focused on is getting the best spread of private/public benchmarks in order to: - Protect benchmarks against excessive hillclimbing which we believe to be beyond overall model improvement (GLM 5.3 gained 16 points over GLM 5.2 in Tau-Banking in only 2 months, near SOTA, but had a more moderate gain in our private AA-Briefcase evaluation where it was still significantly behind the frontier) - Still make use of strong public benchmarks (e.g. TerminalBench 4.0)

As the frequency of model releases increases, we will also be moving faster [1] to ensure our Index stays up to date to reflect both (1) how people are currently using models and (2) what are the strongest signal benchmarks which show these capabilities.

[1] Index release history: https://artificialanalysis.ai/methodology/intelligence-bench...


But isn't there still a meaningful spread of difficulty, risk tolerance, etc. in decision model relevant tasks? Especially at high volumes I think there would still be value in being able to make intelligence/cost/speed/calibration tradeoffs between models

Artificial Analysis has an Index which also includes Cost per Task, Time per Task, accuracy, calibration etc. here: https://artificialanalysis.ai/evaluations/artificial-analysi...

(caveat, I work at AA and helped build this Index)


just found this today and used it as one of the links I sent a coworker who hadn't heard of decision models yet!

Hi there - AA employee here. This is due to 3.8 Flash having a much lower cache hit rate compared to Gemini 4 Hydrogen

Why would cache rates differ? Does hydrogen have a longer cache period?

sorry if this is a dumb question, but how is this possible? what can cause the cache hit rate to change just by changing the model?

This is based on what we measure from the API - we see different endpoints depending on how they are served can have different cache hit rates (cache hit rate is defined as the % of cacheable tokens that actually come back as cache hits)

One possibility (just a guess): new arch produces a more compact cache, it becomes cost effective to hold more cache for longer.

Could you elaborate? What "compactness" are we talking about?

According to AA benchmarks, it uses 162k output tokens per task (with max reasoning) - over double GLM-5.3 Flash for similar level of Intelligence


With max reasoning, Luna is actually less than half the actual cost to run compared to DeepSeek V4 Flash 0731 with updated prices (based on Artificial Analysis Cost per Task)


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: