A place I worked back around 2020 was running a Grafana instance per customer that got embedded on the web dashboard. We had 110 pods per GKE (kubernetes on gcp) 4 CPU node because that was a network imposed pod limit at the time. The nodes were usually idle--could have shoved a lot more on if not for the IP limit.
I think around that time Grafana changed their license tho so you couldn't host OSS Grafana as part of your service.
if it's agentic stuff, they likely aren't hammering a core constantly and they will maybe sit idle quite often between model requests, so it makes sense. I do wonder how much memory they allocate to each one though.
it's just very efficient use of shared cores that is required to make these kinds of workloads cost efficient
In my projects i use unreal and me and my team have tested out multiple things. We have settled on no MCP complications whatsoever other than simply exposing the python scripting from the editor + a export step that can write the blueprints and asset data into plaintext so that the bot can grep them. This has given by far the strongest results, and we now have no issues having the agents edit game code and do operations.
We found this massively outperforms any kind of agent like this and the official unreal MCP systems. Its similar to the Blender MCP which also just exposes scripting + very minimal api to claude code/others.
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
I'm not totally convinced that models are fungible, the claudes/gpts/Gemini all have pretty individual feels when you're working with them. I wouldn't be surprised if the approaches don't scale or even work the same in a poly model setup
that's anecdotal though, right? your subjective feeling of how a model responds to you will greatly influence how you 'feel' about a model and its performance in the same way that a co-worker who you get along with will fuck something up and you'll be more forgiving than when you work with a too-verbose, mansplainer of a co-worker who fucks up
I think until we have actual repeated-use measurements tracked over time (eg consistent prompts used to do the same tasks, count number of hallucinations and errors and bugs over a long period of time) you won't really have any idea of which model is better
I also think of it like a car - some just feel better to drive even if they are materially worse in other measures. until you start measuring the metrics important to you (eg MPG and cost of maintenance over a long period), you have no idea which car is actually better suited for you. and the fact that you can only do so with a limited number of cars (or hours available to work, or money to burn on tokens) means there's no true measure approaching objectivity
Anecdotal or subjective don't mean "wrong." I would 100% agree that claude and chatgpt have different 'styles.' They do have their own patterns, and those patterns are distinguishable.
it doesn't mean wrong, it's just probabilistically full of unregarded bias and prone to hallucinated ineferences, much in the same way that LLMs sometimes are
You can write a config that points your claude code at any anthropic-compatible endpoint, and replace opus and sonnet with whatever you want. Things like deepseek and glm in there do not feel at all like they do in more minimal harnesses, but neither do they convincingly act like claude models. It's very stark and frankly they run like shit this way. Try it.
I'm going to cherry pick one example where newer models are noticeably improving at least in my experience.
What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.
Agree. Harnesses are effective because they interact with the underlying model effectively. If the latest models were fundamentally different, excluding them would be a miss. But I don’t think they are, at least not in ways that would affect these observations.
I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.
I have to say, I am starting to hate this line of reasoning. Yes, LLMs move extremely fast and a lot of improvements are done in a short amount of time.
And there might be a point to these arguments, vaguely. However:
There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)
And on the other hand a random internet comment saying "nope" because...the models aren't the latest.
If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.
There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"
Which brings me to my second point about these kinds of arguments:
LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.
I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"
The models they are using are Nemotron3 series from last year. For reference at that time, the state of the art was Sonnet 4.6 and GPT was essentially unused for coding. Currently, you can run Qwen 27B in your gaming PC and it will absolutely trounce every model they show in their report.
This is a field that changes significantly every few months, so using 1 year old models essentially invalidates the entire report, as they are 2 model generations behind and the newer generation models are heavily RL-d around their harness. For example they put a lot of emphasis on context management but that kind of context management isnt done anymore because many models now have 250k or 1M context and inferencing the models heavily rewards cache hits so you should never touch the context until you decide to compact. They comment on plan mode, which is something that Claude decided to remove from their client because they dont consider it needed at all anymore.
I sympathise completely but a ~3B parameter model and ~3T parameter model are going to exhibit very different behaviour, one can only infer so much large model behaviour from the former.
They have already been doing it for months https://openai.com/index/openai-broadcom-jalapeno-inference-... . OpenAI on their custom chip brought up lightspeed deepseek as experiment by using AI in the exact same way as this zAI blogpost. And the kernel optimization contests/etc have all been havily done through AI based optimization loops for half a year+.
Insane scores for a model of this size. But it does seem to be a rather insane over-thinker with the biggest token use of any model, which combined with 256k context size it means it wont do much before filling it context
We don't actually know how much thinking GPT and Opus do, the labs won't show us anymore. And they certainly take their time before starting to answer.
completely vibes based, but ive been using it to port Mindustry game from Java to C# with agents, and its been working for 50 hours (its 15-20 tks so super slow inference). Its done a fantastic work and its almost finished now. Better results than deepseek flash and gpt luna by a mile on this kind of long term work. Less good than gpt sol or opus. We dont know the param count but my guess is 200-300 range.
FWIW on a 6000 Pro it takes 68 seconds for 10 seconds 480p video, (cold) same demo workflow as you used. Set "megapixels" to 2.0 (1920x1088) and same video seems to take 5+ minutes, not sure if everything is right/correct at the moment.
As far as I can tell, the current ComfyUI nodes don't even do compilation, and I haven't looked into what attention mechanism they're using, but I'm sure with time these durations will come down even more.
People on reddit have definitely pointed out that sageattention will speed up the renders.
And it's literally the first day. Someone will make a distilled 4-8 step LoRA and we're off to the races.
Edit: did a couple of 10 second long 864/480 i2v videos on my RTX Pro 6000: sageattention bumps them up 33%, that is to say, 140.89 seconds without sageattention becomes 105.69 with sageattention on (if using the KJ Sageattention node, "allow_compile" doesn't seem to affect it, just "sage_attention" set to "auto" works fine).
EasyCache also appears to work, but does affect quality, at least with the default threshold or even down to 0.10. Still, at 0.10 threshold the same render above, with sageattention, is down to 71.33 seconds, so depending on your use case the quality hit might be worth it. Also it seems that with EasyCache the video still matches the un-EasyCached video (with the same seed), so you could use it to do seed hunting.
I'm on an RTX 5090. I told it to make a 5 second 864x480 video, it's been running for over 30 minutes and is only 35% done in the SamplerCustomAdvanced step.
EDIT: Oh, I'm an idiot. Forgot I had a llama.cpp webserver running with a model loaded. Killed it and it finished very fast.
I am particularly curious how multimodal models will work with types of knowledge that are inherently non-text. For example, SOTA LLMs really suck at electronics, especially analog electronics.
Is MiniMax H3 capable of logical / technical reasoning, or is it purely art oriented?
Suck at what aspect of (analog) electronics specifically? Not contradicting the claim, just want to understand it.
I have not tested yet, but I suspect that LLMs with a harness that can execute code can do SPICE simulations rather ok these days? I have seen MCPs for measurement equipment also, maybe they can even close the physical loop?
> I have seen MCPs for measurement equipment also, maybe they can even close the physical loop?
I'm working on this but for various reasons can't have my physical lab up at the moment. But yes there are many options for connected test equipment that could rather trivially be interacted with via LLM or pretty easy to write libraries.
spice is a very bad simulation. it's not like a unit test or a VM. it works very differently to the real thing, only approximating it in some highly controlled situations.
If you have accurate device models, and you know how to use simulations, you can predict quite well the behavior of the analog circuits that you design.
Simulations that work like the real thing are an absolute necessity in the industry. It is frequent to design the analog parts of some integrated circuit, when you might have to wait months to get prototype samples that you can measure in the laboratory.
If the prototypes do not work exactly as expected, you will probably have the chance to make once a correction of the schematics and/or layout, which will be incorporated in the final product.
But when not even the second try does not work as designed, that is likely to be a failure, because any other redesign would add months and many millions of $ until the product can reach the market.
So "spice" simulations that can be trusted are a necessity. When designing integrated circuits you cannot use empirical methods that work when you make manually a single-use board, where you can replace components or adjust resistors, capacitors or inductors until you get what you want. For integrated circuits, you must know in advance how the circuit will work, because you cannot tweak it.
It is true however that it is easy to configure simulations that will give non-sense results, due to lack of convergence or other numerical errors, so to avoid this, one typically needs both experience with laboratory measurements and experience in using the simulator.
I doubt that an LLM would successfully run SPICE simulations, because there are few examples of written instructions of how to do this in the right way (which varies depending on the domain of applications, and each SPICE-like simulator may have its own quirks).
> work with types of knowledge that are inherently non-text
What are those things exactly? AFAIK, everything we can "know" can be written down, one way or another, even analog circuits.
Also, what SOTA LLMs are you referring to? GPTs been handling analog circuits fine for quite some time, I want to say for at least one year? I've been "pair programming" a bunch of working circuits with GPT models since probably GPT 5 or so.
> AFAIK, everything we can "know" can be written down, one way or another, even analog circuits.
writing down circuit diagrams is like cooking about music.
> what SOTA LLMs are you referring to?
I have done a survey among analog electronics designers just a couple weeks ago and they all said that their forays into LLMs were great for digital electronics, code, and firmware, but for analog they were pretty terrible, with a variety of LLMs, according to everyone.
I think that's different though, that's "doing" rather than just "knowing". You can ride a bike without knowing how it works, and obviously vice-versa too. I don't see circuit diagrams as "doing" though, but the soldering part of building circuits definitely is that way though, you can't just read about it and excel first time you pick up an iron, you have to practice and understand it with your body, like bicycling.
Im the author of vkguide.dev and i disagree. I wrote that tutorial specifically for people that have already gone through learnopengl and want to learn vulkan.
The core problem with the modern apis is that they are so incredibly complicated that they will kill any newbie on the spot. When the student still doesnt really know what a mesh is, having them setup GPU side memory allocators and graphics compute pipelines is absurd.
Opengl meanwhile is significantly easier, and will give them the important terminology and math required to do graphics. Once the student has learned opengl and wrote a small renderer with a few basic techniques, moving to vulkan or DX12 will happen far more smoothly.
In particular, learnopengl explains a lot of basics like transformation coordinates, what a mesh is, and other similar "basic knowledge", while all vulkan and dx12 tutorials skip through that because they are meant for a much more experienced audience.
The problem with most modern "how to learn graphics programming" guides is that they over index on "low level GPU programming" and under index on important foundational 3D graphics concepts. Transformations and coordinate systems, model space, world space, view space, clip space, normalized device coordinates, screen space, concepts like parallel vs. perspective projection, viewport and viewport transformation, colors, lighting, materials, reflection, blending, antialiasing, texture mapping, various buffer types, tessellation...
You want to start with OpenGL because the API is organized roughly around these high level concepts. Starting with something like Vulkan means you're dropped straight into hundreds of lines of low level boilerplate setting up VkCommandBuffers and VkRenderPasses and VkPipelines and shaders, and all this GPU programming, and it's all just bewildering if you're used to thinking about things at the high level first, or just want to draw a fucking triangle on the screen.
Webgpu is the worst of both worlds. Significantly more complicated than opengl yet not capable of many "essential" modern techniques like bindless resources.
If you must use a modern api to learn for whatever reason, go with vulkan and use dynamic rendering and buffer device addressing.
WebGPU has the issue that feature/capability-wise it is essentially 5 years behind OpenGL 4.6, which came out 8 years ago. And the other downside being that it adopted old Vulkan concepts that even Vulkan started to ditch, like render passes and static pipelines.
For beginners that's irrelevant. WebGPU has more than enough features for beginners to learn about graphics programming. Not just that, once you're used to WebGPU, going to Vulkan will be much easier than the transition from OpenGL.
I wouldnt recommend anyone going to Vulkan, though. It's pretty much the worst graphics API out there, and WebGPU is mimicking outdated Vulkan design decisions that even Vulkan is currently outphasing, like render passes and static pipelines.
Ive been pleasantly surprised with the quality of this model. Its really good at code review and PR review. But right now due to them being so overloaded + this being a big model, its SO slow. It takes forever to do a relatively simple code review.
Oh, okay. Didn't know that, as I haven't tried K3 myself -- I'm quite content with Kimi K2.7-Code for the moment, which OpenRouter serves me to via US-based inference providers. Supposedly.
Obviously, not relevant if your reason for being here is to ferret out the excitement about K3 specifically. But if this is your first time trying Kimi, try K2.6 or K2.7! If my experience is any indication, you'll be blown away by those, too.
