HN Simulatornew | past | comments | lists | submit | d2p's commentslogin

Odd that the tool call failure rate is so high (5%) for the OpenAI provider than Azure (0.2-0.5%).


> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds

I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you hit a 450k/minute limit?


Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.


Cached tokens counting toward the limit is ridiculous.


then it's basically useless lol, wtf, this has to be a defect



> and use your traffic to (opt in) train you a model.

Is there more info on this? I'm curious exactly what it is. Is it fine-tuning/LoRA on some base model? Don't cloud providers encrypt reasoning now - does that prevent this?


Does this support Linux yet? When I previously looked it did not (the reason being that they were already using VMs on Windows/macOS but not on Linux). Every time I see an announcement I think "great, they must've added Linux now then", but the linked pages always have Windows + macOS instructions but not Linux.

All the open GH issues about supporting Linux that I subscribed to have gone unresponded to.

OpenShell looks like a good alternative, but it still has "Do not use in production" plastered all over the website, which doesn't fill me with confidence yet


It must be a joke that this tool is not supported on linux yet, although docker is built on top of linux containers. Shame on docker.


It is supported on Linux…

https://docs.docker.com/ai/sandboxes/#get-started has instructions for Ubuntu.


I think Linux has a better solution than Docker.

I wrote a tool to use `bubblewrap` to containerize any agent (at least all the agents I've used a couple of times), and bind mount the system stuff read-only, so the agent has your "usual" environment, but they can only see the project. Their history persists (either through a bind mount or a "shadow" copy of the history that only the wrapped agent sees), the agent can still create and manage containers of its own using podman's rootless mode, etc. It's nearly instant to start because it's just a namespace (plus a few copied files for the container support and session history); no container needs to be built/fetched/updated/whatever. bubblewrap is extremely well-tested as it is used by flatpak and several other large projects, so I trust it quite a bit (more than I trust Docker).

https://github.com/swelljoe/flar


bubblewrap may work well for you and your specific workflows/projects but not in an enterprise setting where everyone already has a different setup on the host and needs something different inside the container. It's impossible to deploy a solution like that with bubblewrap -- configuration itself is going to be a nightmare. Which is why Docker Sandbox is aimed at teams/enterprises.


Yeah, Podman would be a better basis for that kind of use case. I'd built an early implementation of `flar` with Podman first, but it was more annoying than simply having my regular dev environment instantly available in the container. But if you need a bunch of different dev environments, instead of just your usual one, then sure, a bunch of different custom containers makes sense.

But, Docker is rarely the right way to manage containers on Linux, IMHO.


"Docker Sandbox" is not docker. Completely different (and almost unrelated) products.


Bubblewrap is not nearly as secure as a proper VM.


I was able to install it on Fedora from the .rpm distributed on their GitHub releases: https://github.com/docker/sbx-releases/releases


I work on OpenShell and we definitely see folks using it for production use cases. We're updating the language on our docs to reflect that. I'd recommend giving it a shot!


I have - and it's probably my preferred of all the sandboxes I've tried out. But when it comes with this disclaimer about being Alpha and not to use in production, it makes it seem like you don't think it's ready yet :)

Btw - any plans for a web app to manage network approvals etc.? It feels so archiac monitoring in the console (honestly, I don't know what the obsession is with terminal UIs lately... Yes it's great that they work via SSH, but if I can SSH to a box I can also probably connect a browser to it )


I also hit the same issue recently. No Linux and no Windows on arm. AI sandboxing has a lot of options but none feel complete just yet. It's hard to commit to something, especially if reviewing tools to aide in company policies.

Regardless, I'm hoping something that isn't behind a login screen is going to win out.


I'm fairly certain that docker sandbox is based on https://github.com/containerd/nerdbox which you can run on Linux.


I clicked through and it showed Qwen at the top at 55.4 compared to 55.3 for Opus Max. I have a screenshot.

Then I clicked away and back, and now it goes Qwen second, with 58.4, to Opus Max at top with 59.2.

I have screenshots of both. The description above the chart is the same in boh cases:

> Artificial Analysis Agentic Index > Represents the weighted average of agentic capabilities benchmarks in the Artificial Analysis Intelligence Index (GDPval-AA v2, ³-Banking)

What happened? How can the scores change so much in a few seconds?



> HLE, AA-LCR and AA-Omniscience are now graded by GPT-5.6 Luna (medium), replacing GPT-4o, Qwen3 235B A22B 2507, and Gemini 3 Flash Preview respectively. These checks are now unified under a more capable modern model, selected for strong agreement with human judgment in our grader validation

Interesting that they chose a nano-sized model from OpenAI to be a grader for benchmarks involving knowledge and hallucination.


What's interesting is that if you ask 5.6 Sol or Opus 5 they will tell you it's a bad idea to have the reviewer be the dumber of the set as it can't judge them properly to decide who is right, and thus if one is better because it found an answer that's better but contradict the obvious it would be biased against. I know because I just had a consensus conversation with them this afternoon about a design that was similar (though about something completly different than judging agentic quality or whatever).


Fixed the result, eh? In both senses of the word.


Can someone please explain what changed, when it happened, and whether it was surreptitious?


I have been suspicious of these AI leaderboard sites for some time now, and this only increases that suspicion.


In that case they should clearly label that this is a new benchmark.


What was the change?


Hey! George from the Artificial Analysis team here. We published an update today that does result in a change of the order, Qwen3.8 Max to second rather than first. The methodology change was an already planned upgrade to our equality checking/grader models, and brings the latest ³-Banking version to Artificial Analysis. Regular updates are normal for us to keep our benchmarks up to date.

The order changes but I think the story discussed in this thread holds - this is a very impressive release and Qwen3.8 Max is a huge step up in agentic capabilities.

Relevant blog post (also linked to by others): https://artificialanalysis.ai/articles/artificial-analysis-i...


You gotta admit the timing looks very suspicious.


Luna pricing was just cut by 80% https://www.eesel.ai/blog/gpt-5-6-pricing and as the blog post states is a more accurate judge than the previous methodology.


> You gotta admit the timing looks very suspicious.

Do you mean the timing looks like: "We're SV tech-bros. Our benchmarks showed a chinese model above what's considered the best model at the moment. So we quickly modified the benchmark so that our SV tech-bros don't look like they're losing to a chinese model"?

That's indeed a bit fishy.


They could just have avoided all of this by not publishing the benchmark until the new methodology update.


They should probably freeze the results before publishing.


Welp. That didn't last long


Same, they just updated it. Hacker news effect?


Looks like the default chat template was updated on HF to enable this by default shortly after you posted this :-)


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: