Qwen3.6-27B has been the main LLM powering my little agentic stack. I have adopted the test and verify approach to any models allowed to run on my machine. When the "heretic" version drops, I will fire up the harness and test. Super excited to see how it stacks up against Qwen3.6!!!
Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM.
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is.
I think when tuning 3.6, I saw more like 33%? But not 100% sure.
I wonder if we could take a page out of the Solar power book and do MPPT but for draft count.
Constantly sweep through possible values to find the best result for the current conditions.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m
This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.
Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33
The unweighted average is 4.1
So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.
Also, cool that it carries over from qwen3.6 like that.
I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.
How many tokens/second you getting? I have the same CPU but a 5070 and only 64 GB of ram. I just got llama.cpp built and am now hitting a whopping 5-6 tokens/s.
I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
Problem with that is I think that it quickly devolves into cargo culting, nonsense and noise.
Arguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot.
Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future.
So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise.
And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with.
The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.
I've found that once you factor in multiple GPUs things can get complex quite quickly because the default packing routine in the LLM runners tends to be very coarse resulting in substantial amounts of VRAM wasted. More so if you start running drafters and multiple models at the same time.
Let me return the favor:
Deepseek V4 Flash 0731 Q8
This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04.
I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.
still could! there's the network site https://ai.stackexchange.com/ just need people to ask questions and to downrank accepted answers older than a year (that might not be possible though).
I can get 128k context on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]:
1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.
2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.
3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.
You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).
Congratulations, you're now running Qwen 3.8 27B.
Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB
Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.
But maybe someone else has experience to share there
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
I've worked with a good few types of managers of the years, none would take ownership of my work. The end result? Sure, for shareholders/this managers that is fine.
But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product.
this topic always goes into this level of ridiculousness. Your question can't even be answered. We should focus more on encouraging discussion, not asking unanswerable questions. It wont bring anything other than points of both sides with no objective correct answer.
I generally agree and expect this to be the case from a legal perspective.
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
So if I hire an artist and am a motivated individual, have an idea for a painting, have background knowledge about paintings and have taste in paintings and can offer a critique of the painting as the artist paints it, then somehow I created the painting?
Nothing absurd about that. What do you think an "Executive producer" is? A "Director" ? Does Peter Jackson get credit for creating the Lord of the Rings Trilogy films? Christopher Nolan for his films? But did he make them ? No, it was the collective effort of thousands of individuals all working under their direction.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
What does it matter who receives more credit? The producer produces the film and the director directs it. If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it.
If there were multiple people involved in the creation of a film I helped to create, I cannot factually say I created it. Just like if someone builds something using code generated by AI, they can't factually say they created it.
There's a bit more nuance to it though, I think, because traditionally there has been more of a firm line between humans and tools.
For example, most people would agree with this line you wrote: "If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it." A pedant could say "woah hang on, that's ignoring the fact that actually the iPhone is the one recording the images that make up the film, how can the human get all the credit", yet nobody would actually make that argument when discussing who created the film.
Generative AI is pretty much the first time (maybe there are some niche contradictions to this claim?) we consider a tool to be contributing enough creativity to the process that we don't all agree "only one person was operating this tool, so that person is the sole creator" - but some people DO still hold that line, and do consider the human who wrote the prompt to be the creator.
And I don't think there's any objective technical metric we can use to say who's right, it comes down to our collective judgement deciding where the line is.
That’s an interesting way to look at it, and still generous to the llm.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
I think in that case it's fair to say you created the painting with the artist, even if the artist should get majority credit. I don't like the analogy though, to me it feels more like you're a project manager directing a team of genius but single minded interns.
I've never encountered a project manager who has said: I built this thing, after a project has wrapped up. I have encountered ones who've said I managed this project.
I've heard some variant of "we built this thing", which I think is valid. I'm not sure how I feel about anthropomorphizing a coding agent at this point though, which is what you're implying.
I just read through a couple of your posts that weren't dead or buried, and it seems like you're pretty anti-AI. You should really start to have an open mind towards it. It's going to be the future (if it isn't already), and as you continue to get older, you're going to really wish you spent your time right now learning and embracing the technology instead of being so against it. A lot of the skills and things that you're holding on to right now might not be relevant by then, but you'll be at a disadvantage from not keeping up with the industry and need to play catch-up.
> I'm not sure that typing messages to a chat bot requires much catching up
Lol, yeah I'd wager AI tooling skills are less relevant than knowing your way around a shell. It doesn't help that all of the AI tooling has event more churn than js libraries and package management systems!
I think there's probably some value in understanding how LLMs work, but beyond that it's the same boring skills that matter the most... Critical thinking, design chops, attention to detail, perseverance, etc.
He still didn't create the paintings, he offered instructions / feedback to the artists that created them. Whether or not he is credited with authoring them is an entirely separate issue.
There have been plenty of workshops where artists hire assistant painters while maintaining authorship over the works themselves, from Rembrandt to Warhol to Hirst.
Sure - and a factual statement would be, these artists hired assistant artists to help them create their paintings. Just like it would be factual to say that the person directed a LLM to build an app for them.
The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.
Does anyone know how to get this working with Claude Code via llama-server? I'm getting a jinja template error about the system prompt not being the first message.
You could run the 2-bit quantization at about 10GB for the weights. You'd probably also need to quantize context to get a decent-sized context. The Unsloth model card shows the 1-bit quantization doing seemingly shockingly well on a task (but there is notable loss in model quality for each bit you give up below about 6 bits). An MoE model can partially reside in system RAM without catastrophic performance loss, but a dense model like this one cannot. You have to run it entirely out of VRAM, or it'll slow to a crawl.
Yes. It won't be as fast as fitting the whole model into VRAM but llama.cpp defaults are pretty smart about GPU/CPU splits these days. Just YOLO it with `llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` and it'll definitely at least run.
You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
For the things it does, what tool is better than ffmpeg? Really struggling to see the cargo cult angle. Similar for llama.cpp, as it is literally the only framework I can get to run on my multi-Radeon rig. It is the most portable runtime out there.
It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.
That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?
Unfortunately I checked, SLI doesnt work for this situation. Because the program loading the LLM uses CUDA library, which doesnt account for/takes advantage of SLI for this purpose at least.
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.
I just want to run `` with some default parameters set and for it to run locally.
What makes you say that it would be hard to do that?
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers.
I think at least. I've never tried. My hubris stopped me from trying auto configs.
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Why would they wish to handcraft this ? That is what agents are for ?
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
The alternative is to label it AI. That is distasteful for some of us, Intelligence is much stronger than automation in our minds so we avoid the term and try to describe it differently .
This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX.
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”?
I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived.
There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.
Even this command above, it's not clear where op got the model from. So I'm with yah.
For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..
Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.
But you're assuming I'm using the Olama studio. This model as far as I see doesn't have a gguf download.. Unless I'm missing something on the page.
If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.
What on earth are you talking about? llama.cpp != Ollama. You can (and should) just use llama.cpp directly. Upstream llama.cpp can take the shorthand huggingface path and automagically download it into a cache folder as part of the launch. Have you read any of the docs?
Excuse me, but thats a direct link you've just sent. I asked where I can find the links. I like to believe in the source of truth.
They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Do you see how steep the barrier for entry is to do anything right ?
unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that, and I want AI to be a better tool.
> Excuse me, but thats a direct link you've just sent. I asked where I can find the links. I like to believe in the source of truth.
You asked where to find the GGUF files of this model for direct download and I provided it. Almost all useful model files that can be downloaded are hosted on Huggingface.
> They shared a lot of links, I'm struggling to find yours. Where did yours come from ?
I went to Huggingface, went to the Unsloth org, as they tend to be the best, went to the Model page, and went to the "Files and versions" tab.
> Who is Unsloth AI? Have they modified the model ? Is this really the source of truth ?
Unsloth AI is a very popular, highly reputable organization that takes upstream model files, performs some optimization, and provides models in various formats. Apart from speed tweaks, they do not modify the models. They also provide useful benchmarks, copious documentation for local execution, and a Studio application for easy execution and post-training of models.
> Do you see how steep the barrier for entry is to do anything right ?
No. Searching for this information is not difficult. The llama.cpp documentation and guides that Unsloth provide are all you need. Search engines can take you further if you want.
> unslothai is not a name qwen has ever used. So you're sharing a link to a model that isn't from the owner, while saying it's the owner's. I'm not comfortable with that
Qwen also provides models in GGUF format on Huggingface, but they will not be as performant. Even when first-party GGUFs are available, most people will prefer quants from Unsloth or a few other popular optimizer accounts.
> I want AI to be a better tool.
Best of luck. Your attitude and unwillingness to even try and learn on your own when people have tried helping have burned my good will, and this is as far as I'm willing to carry you.
I gave you a direct link to where you can get a download, and explained why I chose to link you where I did. I can't post screenshots here drawing you a map of how to use a website. I don't work for Huggingface or any AI company and can affect no changes to how intuitive any of it is, and think is easy enough already.
Huggingface, Unsloth, and llama.cpp all have documentation you can follow that will exceed anything I can tell you here. LMstudio, Lemonade, or Ollama might be even easier for you to use. Take my suggestions or don't.
Start by copying the command line from the Unsloth guides.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
on Qwen's official docs: https://qwen.readthedocs.io/en/latest/run_locally/ollama.htm.... It's not like I just made it up. Of course I searched "ollama qwen3" and saw what you linked, but that doesn't mean it "works". I have other things to do besides to try a bunch of poorly documented and executed tools just to see if it works or not.
I guess the TLDR is that I'm stupid or lazy. Also, everyone is responding about how easy it is, and yet, it's apparently so easy that it's hard to document well.
There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.
I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.
While you probably have the audacity to check the commands and code that your favorite LLM generates to complete this task, i'm pretty much certain that at some point in time, due to this pervasive mindset of "just let an AI agent do it," we'll have a front page news story of someone who had their family photos on their NAS uploaded to some public space, their security cameras connected to a "live porn for lolz" webring, and perhaps even their crypto wallets and bank accounts pilfered, all because their favorite LLM got exploited by some prompt injection attack while trying to be helpful doing system administration tasks.
I remember there was a short story in BYTE Magazine about a similar kind of scenario way back when, I think at least 30 years ago, long before LLMs and AI agents became a reality.
Hmm I'm still struggling with the downloading of all the models I want to try locally, so I might be out of the loop, but all of this, including tuning (to your local HW), feels like it could be automated. I'm old though.
> Every big company in the world uses a network proxy.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
I would love to see more data on that because I've seen it constantly. There is more isolation maybe where you can do whatever on 'open' network, but always some kind of proxy/vpn connection for hitting anything sensitive.
The operlap is there.. But I would be worried if it was just flat out taken away from secure managed connections just because of AI.. Again, would love to see the numbers of your assumptions.
Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic).
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
I hope really badly that we'll get a new 35B A3B or similar MoE model!
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
I'm hoping too that they'll put out some MoE variants.
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful.
I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.
Probably helps it score a little bit better in benchmarks :) `medium` seems like a nice balance so far; along with some light steering to vary think effort as needed for task and being pragmatic.
Interestingly 'medium' is the closest thing the _model itself_ has to a default thinking level. The chat template injects directions [1] at the very start of the system message when the reasoning effort is 'xhigh' or 'low' but 'medium' implicitly just means no added reasoning-level instructions.
[1] The specific directions are "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer." and "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."
"inclined to overthinking," holy cats you're not kidding! On a Mac mini M4 Pro 64GB I prompted it with "svg owl" and it thought for 17m12s, outputting 36.3KiB of thinking chatter. It did end up producing a 20.2KiB HTML+JS+SVG file with a very nice owl, including cursor-tracking animation, but it ran for more than a half hour!
I think that's called test-time scaling i.e using more tokens at infer time to squeeze out higher model performance. That's must be part of the explanation for good benchmark results.
Yep.. it's pretty obnoxious for real-world use with the default 'xhigh' thinking. Ridiculous amount of "Wait, actually.." which might help for complex coding tasks but makes it unbearable for general purpose use.
that seems to be how most Chinese models achieve increased benchmark scores. GLM and Kimi models are "thinkslop" models that reason over their own thinking, which increases cost and decreases speed significantly. That's why GPT is in a different tier altogether - faster, smarter, and sometimes cheaper.
No agentic harness; I just prompted literally "svg owl" into LM Studio 0.4.21+2. macOS Tahoe 26.6.1, Mac mini M4 Pro (14 core, 20 graphics cores) 64GB RAM. (I put the machine config so the 17m12s of thinking has some context.) It got about 12.75 tok/s and used 21,769 tokens.
For comparison's sake, Qwen 3.6 35B A3B MLX, same prompt, same machine, thought for 1.59 seconds, used 2,398 tokens, and hit 80.83 tok/s. It actually output just a straight SVG file of an owl (not a bad owl either!), whereas Qwen 3.8 27B spent a zillion times longer gold-plating everything and output HTML+JS+SVG with interactive animation, sound, etc., etc.
Did you try the claude reasoning traces finetune for qwen3.6? I find that it works muuch better. I assume the same 3.8 finetune will be released at some pointas well.
Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.
[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s.
MoE is literally 5x times faster (on CPU) than comparable dense Qwen!
So you happen to know how to demonstrate this higher capacity, what do we mean by that? Can understand bigger codebases? And how do we explain it, what part of the model does it?
Don't mind! It's 64 GiB dual-channel DDR5-6400, i.e. roughly 100 GiB/s of bandwidth. (AMD 7840U (Zen 4))
I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).
Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
Thanks! I don't have any knowledge of running models locally.
I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?
not parent, but 4-bit quantization is generally consider a good trade off for speed/performance, so you might use it even when you aren't on consumer hardware, but definitely when you are on consumer hardware.
AgentWorld is a pretty recent MoE release from the Qwen folks with quite a bit better performance than 3.6, released around the same time as 3.7 Plus/Max, I guess. Not sure why it didn't get more attention, as it is definitely better than 3.6 35B A3B on all dimensions, but especially for agentic use. Still nowhere near the dense models (even 3.6 27B), but clearly an upgrade in the small Qwen MoE line.
KAT Coder is another Qwen 3.6 MoE fine-tune that also improves on 3.6 by a measurable amount.
But, I do hope for a bigger MoE, in the 70B to 120B range, something in the Coder Next lineage. I've got a Strix Halo that isn't getting used to its best ability because the best models all run fine on my faster desktop dual 32GB GPU setup, and they run too slow for comfort on the Strix Halo.
And, yet it outperforms Qwen 3.6 35B A3B on Terminal Bench and SWE, etc. I dunno.
Edit: I guess you're right; apparently it's for simulation. I didn't look into it beyond the benchmarks. But, it does work in an agentic context, regardless. It'll write code, and drive an agent.
Me too. 35B A3B runs really fast on my MacBook Pro (M4 Max) and is suitable for real-time tasks like dictation post-processing. The dense model is not.
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)
Also radically better on an M1 Max. I get well up into the 60s t/s with the A3B, stuck at 9.5 or so with this new 27B, though perhaps an MLX build will help.
35Ba3b is usable in plain cpu inference on a fast server, the dense model is MUCH slower. On GPU the 35ba3b is still around 2x the tok/s single threaded, which can be a good tradeoff for some applications.
MoE models have less active parameters in play at any particular moment, so they perform much faster on lower bandwidth memories (like Strix Halo or DGX Spark) and also use less memory generally.
Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.
anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.
Or even on fairly high end (by general public standards, not gaming community standards) consumer hardware. Fantastic fit for M-series Macs with 32-96GB RAM.
Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the binomial SE.
How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
> ...but no. They do not beat opus on real-world usage.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.
That’s well and good, but how are we supposed to evaluate the accuracy of random HN comments without anything resembling somewhat objective metrics? People say all manner of things, and usually it’s contradictory. What heuristic do you propose?
> but then we just need meaningful benchmarks that clearly show that!
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.
Yep. These small models are actually worse than GPT 3.5 at some tasks (like recalling facts). You can definitely make models smarter at specific tasks (like tool calling, coding) but you can't compress the entire human knowledge into a 30GB file. It's just not enough bits.
Models cant make any decisions if they have 0 idea that the feature exist in this language or in some general fact.
For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.
Ideal local model would not know stuff like who Britney Spears is, best to leave precious weights for something useful.
Of course the line is very blurry but I'd be perfectly happy with local model that doesn't know anything about history, geopolitics, art or even biology etc. just coding, operating systems etc.
At one point there were specific -Coder release of Qwen e.g. 2.5 but they dropped that, still wondering how much better 3.6-Coder or 3.8-Coder would be when they ignore everything else
In my experience, the facts that are compressed away in small models are ones you don’t need them to memorize. They need familiarity with the essential concepts in a field, so that they will have better comprehension of material put into the context (and make better retrieval decisions). They don’t need to know very particular details which are a hallucination risk and should be derived from the context instead.
"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.
But that's the problem with that logic. The author subjectively claims benchmarks are stupid (a debatable opinion) and treats that as inherently true while making the second claim (hence, you shouldn't say model A beat model B).
I can say I find the law stupid, so no one should say person A beat person B in court. But I did not prove the law is stupid; I merely thought it subjectively and demanded others to follow the second part because I believe the first part is true.
Saying that "if the law is useless, court cases are useless" is objectively true and cannot be argued with. But you still need to prove why the law is useless, not why you think it is and even then if people disagree and use the law as a reference, then it's not objectively useless and court cases are not useless as well.
For high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number.
I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
I didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.
Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.
The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.
KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
It's a cheap to evaluate proxy for totally broken or not, which is a good start.
It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.
You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.
That kind of result makes me suspicious of benchmaxxing. Qwen 27B is 100x smaller than Opus 4.7. Is it really 100x more parameter-efficient? Two orders of magnitude is hard to believe. I don't have the hardware to run a 27B, but I'm curious what real world use is like. Maybe I'll have to buy some usage on a cloud provider to run my own tests, but this seems fishy to me.
Qwen small models are heavily coding focused, whereas Opus is everything to everybody (even if code is their bread and butter). The downside is they'll frequently hallucinate world knowledge so they need to be RL'd to double check their knowledge against sources and verify facts/library names/etc.
It's very agentic coding focused; and I'd say a good executor but certainly not Opus in scale; overall knowledge; long-horizon work and recovery; etc.
e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.
Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
Oh, goody goody goody! Can't wait for a Q4M with *MTP* support. Does anyone know if there are plans for it?
I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.
In the local LLM communities there is a lot of respect for the Qwen models, but everyone comes to acknowledge that they do a lot of benchmaxxing after using them. Even at full precision they're never as good as models with similar benchmarks.
People will claim it's not comparable to Opus despite it beating the score. I'm not sure I disagree, but I'm also unsure whether I care. Most new models nowadays are "good enough". I cannot complain because I'd rather spend that time improving my prompts and docs. Opus might be a _slight bit better_ at picking up vague hints, but it's also extremely expensive, and I hit the 5 hour limit way too quick.
I care a lot about speed and efficiency right now. For my setup I would like to have 2-3 different model families. I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting, Deepseek V4 Pro 0813 for developing, and Gemini flash lite (any recent cheap model) for repo scouting. I'll add another one in the mix for reviewing (in this case Gemini 3.7) and that's all I need.
I've tried most models except Grok.
Qwen is too expensive IMO (Alibaba Cloud subscriptions are hard to come by and I'm not spending 50 euros a month for a tool, so 18 euros it is). If it ever becomes efficient enough to run locally I will definitely look back.
Claude is slow and expensive (the cache hit prices are absurd).
OAI is pretty good, I might add it to my arsenal seeing how cheap it is.
These opinions change every day. Last week I would've never picked Deepseek until I read about the pricing. even post aug 16 it's worth it (although it's getting close to gemini pricing).
Right now my costs are 12 euros a month (z.ai) + whatever deepseek consumes. This typically isn't more than 8 euros a week. 44 euros a month and I have a setup that is doing pretty well.
I'm convinced a lot of the anti-open-weight model comments at this point are inorganic traffic - there's trillions in investor money riding on a world where these models aren't cheap commodities. Having actually used things like the recent GLM, Kimi, and Qwen I think any edge the labs have is marginal at most and actually prefer the open weight models in most day to day usage.
Anthropic's recent releases are wordy to the point of exhaustion. Every time I use opus recently I find myself wanting to yell "GET TO THE POINT" at a terminal, which is exacerbated by it being slow.
Why are you convinced of that? Pretty much every time I see statements like that online, I can find plenty of organic traffic supporting it, not everything is a bot.
I just wouldn't bias myself that way, most people haven't really used local models. This stuff is pretty much all subjective evaluation, there's plenty of reasons for people to favor certain models or disfavor others.
I'm generally extremely skeptical about a lot of the model hype that show up in comments. Except when there is an extreme mismatch the performance, quirks and quality of these things are difficult to nail down. You wouldn't know that from the comment section of every single release.
I think some of these are excited, eager users always ready to hype up the new thing. The same crowd that previously would constantly push for a rewrite from angular->react->svelt->god knows what. Instead now it is on a 6 week cycle and about models/harnesses.
I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
Ultimately, I think some of it is legitimate shifts in who's in lead and what is the best. You gotta dig through a lot of crap to get to that, and I don't really know how to to.
Ultimately I'm saying is that I always applied a fair amount of skepticism about what I see in comment sections but these day it is extreme amounts.
> I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
I've assumed the same as well.
I also assume that many of the companies developing these models engage in benchmaxxing.
At my company we've developed our own internal benchmarks for evaluating LLM models as they become available. The benchmarks are tailored to our particular use cases but the utility and knowledge our benchmarks assess is still fairly universally applicable. I see wide differences between what our internal benchmarks report and what the major benchmarks do.
There was a whole lot of fanfare about how amazing GLM 5.2 was when it was released, but it was pure rubbish on our internal benchmark -- far behind OpenAI, Anthropic, Gemini, DeepSeek, etc. I don't know how to reconcile the fact that GLM 5.2 performed very well on some of the major public benchmarks, but consistently performs so poorly on ours. OpenAI models tend to dominate our internal benchmarks.
It actually is showing in public benchmark if you know how to look for it. For example, in Terminal-Bench 2.1, GLM 5.2 received 78%, while GPT 5.6 Sol received 88%.
Then Terminal-Bench 3.0 came out (where the questions are new), and GPT 5.6 Sol received 34.6%, while GLM 5.2 dropped to a whopping 4.6%.
> OpenAI models tend to dominate our internal benchmarks.
That's odd, since Fable seems to be the leader for the industry. Not cost-effective, but if Anthropic models get dominated by OpenAI in your internal benchmarks, this calls their validity into question. Separately, see the jagged frontier effect. [1]
I've been using GLM 5.2 at my day job (mostly Rust backend work ATM). Nothing that blows away the models from OpenAI and Anthropic, but solidly good enough to get it done. A lot of people have experienced this and the fact that an open weights model can do so is where most of the excitement comes from. Optimizing for benchmarks can only get you so far, and people are quick to criticize models that fall into it (like DeepSeek Pro V4 recently).
I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.
Really no different than when there was suddenly online personas everywhere hyping up TSLA out of the blue. You can see the same thing going on with the BoringCompany subreddit. Crazy that the botnet master isn't able to convince us that Grok is also the best model. I don't think buying twitter was an accident it was probably just literally covering up the evidence.
IMO this is getting hyped because the 27b version runs on a decent gaming GPU. This is NOT a thread for their largest model, this model will run on a mid-high end gaming PC, which you probably have in your household. Mine is 6 years old and it runs quite well.
I strongly suspect that many of these accounts you think might be bots from the labs are just people who only have one interest.
One thing I have noted a lot more of is that there is comparative fanboying going on. Like "this model has done badly, my favoured competition has a model out soon that will beat this in every way" — comparing a released product to unverifiable hopey claims about an unreleased product.
It's tempting to assume that is bot stuff, but if you've been around any other "hot" technical hobby online (cameras, phones, 3d printers, whatever) you will know it's not. It's just fans aligning into teams, some of them laconic and amusing, some of them overkeen and toxic.
Which itself released yesterday? You're writing, reading, and evaluating enough software in a ~36 hour period to form, reject, and form another opinion about which model makes better _architectural_ choices?
Honest question, how do you assess models this quickly? What metrics are you using? Would love to get my suite from multiple days and hundreds of prompts down to minutes. Got a few first pass tasks I run upon release for an initial experience, but those only work because even Fable and Sol fail despite objectively correct solutions existing, so it works because most models fail, but then, those are consciously not enough for coding, tool use, adherence or task specific inference and assessment…
Anything from ML pipelines for language specific pruning over a Rust/JS/CSS mix codebase to assistance in motorcycle maintenance and different canvas coatings. Most of my evals build on those requirements and especially past failures, whether in pure information, task execution and coding or tool calling beyond the overfitted mainstream. All stuff derived from actual failures encountered, some still only few models come even close to passing. With such a mix, it just takes a while to get any serious opinion on a model. Doubt anyone can do that in such short time, unless their tasks are so simple that most modern models not only succeed but could themselves accurately rate output. If even Fable still confuses PU or wax coated cotton canvas with a nylon shell, or tells me with a straight face to adjust the valves on a bike that has hydraulic lifters that needs experience for human assessment and the time that comes with it. Anyone with less knowledge either wouldn’t see the mistakes staring them in the face and just go by vibes, any model rating these equally can not tell what is accurate and will just go by the output sounding accurate over being. Gives sometimes very interesting results far different to public benchmarks. Inkling, e.g. is more accurate in not telling you to adjust valves that are simply not adjustable then Fable or Sol, which just tell you to adjust every 5000km. Sometimes even when their reasoning and search includes sections about the fact this is not necessary or possible. The beauty of overfitting and unbalanced training data…
Most of us aren’t deploying these in general purpose use-cases. E.g. I use Qwen mostly for vision in my personal assistant. I have an eval set for that. Pretty much each of my use cases has a pre-computed problem set.
Likewise I have a tool-use eval set and a browser-use eval set. I use frontier models for most interactive tasks that are not home assistant or background agents.
I can imagine someone could build evals for that but I have never done so.
Deepseek v4 Pro or GLM 5.3 for software architecture I feel are only deployed for general, rather doubtful those are for narrow stuff, if we are sticking with this threads mentioned models. For small models, sure, narrowly targeted sets which can be self evaluated are amazingly valuable, but I feel beyond 500B we are in a different dimension. Rating any model the size of GLM or V4 Pro in hours I doubt is done beyond pure vibes.
Grok 4.6 is a game-changer. I have yet to go back to other models after starting to use it. You just can't beat the price + output quality (even K3 is more expensive)
no they aren't. they discontinued their always-sold-out coding plan and launched QwenCloud (basically a friendly frontend with Alibaba Cloud as the hidden backend) and launched typical subscription plans for Qwen & co alongside it.
With max reasoning, Luna is actually less than half the actual cost to run compared to DeepSeek V4 Flash 0731 with updated prices (based on Artificial Analysis Cost per Task)
They were significantly more expensive for coding compared to DeepSeek's offering because of much higher cost of cache reads and lower cache hit rate. Now, after the price jump, the effective cost of deepseek.com API vs 3-rd party providers is about the same.
I think it's very clear that someone who has checked out all the models but the one that called itself mechahitler and is explicitly being fine tuned to support far-right politics is making the choice for reasons other than performance and cost. It's not like all the other models even had plausible claims to those metrics.
As I said to a dead reply, for coding all of that is immaterial, as long as it codes well then that's all that matters to most people, except it seems those who have an idelogical issue in which case the other model companies also have issues.
No, coding does not make ethical issues immaterial. Nor does it justify your behavior of harassing someone to do something that they've clearly decided not to do for ethical reasons on the basis of criteria that they have clearly rejected.
"Harassing" lol by telling them to try a model, alright. Anyway I didn't know they didn't want to use it for ideological reasons, they didn't say anything about that. Now that I know there's nothing more to be said.
I don't care how good Grok is, it will be a cold day in hell when I give a guy who goes to AfD rallies, instrumentally supported the current Ebola outbreak, and gives the Hitler salute on stage (don't try to tell me he did not, I saw the video) any money I'm not absolutely forced to.
And yes, OpenAI and Anthropic are highly questionable, too, and so I've ended my OpenAI subscription and I'm moving from Claude to other providers. But even people who do not do that are absolutely justified in drawing a line at supporting Musk, because there's a difference between mere normal shittiess and Musk.
The funny thing is it doesn’t even code well. It’s still worse than OpenAI, Anthropic, and the leading open source models.
So you get worse outcomes while also financially supporting one of the most prolific pushers of race wars, the destruction of labor rights, environmental protections, and general misinformation.
Truly a win-win?
Contrary to your framing it seems to only reason to use Grok at this point would be ideological.
It's a better deal compared to those is what I said. It codes well enough so that's why people use it. As I said, go to r/cursor and you'll find loads of people using it while OpenAI and Anthropic are more expensive per token and give less usage.
There are a lot of people who are apathetic to what musk is, people that don't care are not people who should inspire you. What the hell is so inspiring about apathy anyway?!
And yeah, people that don't care DO make the world worse through their apathy.
I would say it is sad that there are people who use Grok when there are so many other choices available which don't come with the issues of supporting Musk.
It is not all just 'politics'. Take a stand on some issues. It doesn't cost much not to use Grok.
> No one actually cares about the politics as long as the model codes well.
I do. There are enough models to choose from that I don't need to use one from a guy who did a nazi salute at the presidential podium, and proceeded to rid our government of brown people he felt didn't deserve their job.
i care about not financially supporting a person that is actively trying to disenfranchise me, why is that a difficult concept for some people? that not everyone is motivated exclusively by financial profit? is moral bankruptcy so pervasive that some people assume it is unanimous?
'They' - in this case - are those who are unable to empathize. People who only learn of other's struggles through visceral personal experience. Conservatives, in general, struggle with this class reasoning... Very much a 'if I don't experience it then it doesn't exist' kind of mentality. Frustrating and exhausting. A lack of imagination, a lack of empathy, they just are lacking in these regards.
Your comments are all in absolutes. As if this was the one thing that mattered to you, everything else is irrelevant. Don't most people make their decisions based on a balance of considerations? Dismissing half the world as nazi or fascist does not seem productive, it kinda makes you look like the insane one.
One thing a lot of people don't seem to factor when hyping Qwen is how much models like this tend to 'overthink' with seemingly endless 'second guessing'. 3.8 seems no different from what I've tried thus far.
As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time. I suppose 'YMMV' depending on your use case.
Also, I haven't used it enough yet to see if it's prone to infinite looping, but its predecessors sure were.
IME this is a strong/reliable model smell, you typically see smarter and less benchmaxxed models' thinking traces spending more time exploring the solution space, and benchmaxxed models more time trying to refine/decide on the response contents. It has always been a big problem with qwen.
In general it's kind of a benchmaxxing/distillation (I don't mean that pejoratively here, read on) artifact where test time compute's purpose is to refine/zero-in on a particular input:output pair. Basically they're trying to "remember"/re-derive the answer rather than arrive at it deductively - it gets comical/absurd when you see it expending a huge amount of tokens trying to figure out the answer to some trivial question or response to some input, as if it were a trick question or the choice of wording was of the utmost importance. But it's not a bad thing when the output matters or the task is hard: basically the model has been trained to respond/act like a much smarter model and it's probably better for it to overgeneralize that behavior.
Also, IME Gemma and most other local models that support thinking tend to have the same problem, and it's partly only a "problem" because they give you access to the thinking tokens themselves (which you don't in many cases see when working with frontier american models) and you can actually see how they're getting spent. For the local or bulk (runnin on owned or rented hardware) use case where you are paying in time/watts rather than purely by the token, IMO it's a good idea to just look at the task success rate / time per task and not worry about the raw thinking token count.
Given that it apparently defaults to 'xhigh', this is probably the answer.
Granted, it's still much lower tokens/s than you'll get out of many MoE models.
Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b. I think gemma4 is just a better 'general purpose' model.
I haven't tried lowering thinking, however, I actually asked a solid question earlier regarding a real world scenario I encountered and all that excessive thinking made it give me an amazing answer. The thinking actually all made sense, and honestly I found it thought of similar stuff to what I thought when I drew my own conclusion.
I don't usually rely on AI for much (I'm actually kind of anti-AI, although I follow stuff like this enthusiastically because of the rapid advancements, "average joe" access, and openness), however for what I asked? It was spot on.
The subject was a bit personal, so I won't share. I was just curious what AI would say about the situation and it definitely surprised me, especially since Qwen, while usually great on development/coding stuff, has shown weaknesses in other areas.
Definitely a solid release, and this one runs on my 4090 with minimal loss of quality!
In my experience with Qwen3.6 35B-A3B, disabling thinking made the model generate inaccurate replies. Ask it for the recipe of egg salad and it gives you the recipe of an omelette.
Did I miss something, is it possible to have that model be reliable without thinking?
Turning off thinking mode you need a different tokenizer and a whole host of settings. It's more like switching your vehicle from gas to diesel, than turning your headlights from lo to hi.
I used the settings recommended on huggingface/unsloth's model page + whatever suggestion from various LLMs - didn't research too much myself which setting did what.
Frontier model labs don't know anything about good sampling settings, and why should they? Their belief is that "sampling doesn't matter, we can simply scale our model and improve logprobs quality that way"
Ignore whatever nonsense qwen says about what settings to use. Turn off top-p and top-k. Turn on top-n-sigma and keep it at its defaults.
See repetitions that you don't like? Ignore the "repetition, frequency, presence" penalties and turn on DRY (don't repeat yourself).
Want more creativity? Temperature is okay but XTC (eXclude Top Choices) is better.
Llama.cpp quietly has had far advanced samplers compared to the shit stuff Qwen et al recommends for years.
Use 3.6 27b as a daily driver for months with charmbracelet crush. Gemma 26b-A3b is not even remotely comparable in terms of coding for me. YMMV depending on how you work, what harness you use, etc I suppose.
Yeah. I use it to do extensive work on full source bootstrapping, deterministic operating systems, compiler debugging, kernel debugging... all with one tiny go binary without endless NPM deps like pi (which /I/ regard as trash)
Works better than opencode (pi based) or anything else I have tried for my needs, and by far the prettiest and easiest to reason about what is going on.
But I will bite. What does pi do today better than crush for your use cases?
> As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time.
Not my experience at all. On my M1 Max MacBook Pro, Qwen 3.6 35B-A3B provides better responses in similar time (~60 tokens/second, vs. 50 for Gemma 4 26B-A3B). 3.8 27B is slower, being a dense model, but so are dense Gemmas.
“Thinking” is just a guiding methodology to help iterations (between the initial prompt, results, and a mixture of harness back and forth to the LLM) converge on something sane in a massive parameter space.
I like to think of it much like (as a common example most people can relate to) the Newton-Rhapson method for finding roots of a (mathematic) function. Your initial prompt runs, then the ‘harness’ kicks in using whatever methodologies are behind them to iterate on that prompt (back and forth with the model, occasionally with the user to get better guidance) and refine the outputs to hopefully converge back to some sensible output or actions the user was initially looking for.
So you’re hoping for an LLM that sort of ‘zero shots’ or needs minimal iterations from a prompt to give usable results. I find from my anecdata it varies across models and what I’m trying to get it to converge on. I tend to prefer models to not zero shot attempt because they tend to not do great, I want them to get feedback often to let me push them down the route of convergence in spaces I already understand well, meanwhile I like them to explore and give me new paths in spaces I’m not too familiar with.
That’s really what all that “second guessing” is, it’s making sure you’re following a sane path in a massive parameter space of an ambiguously defined problem. Imagine if in Newton’s method you checked the slope and it didn’t decrease from the last iteration and you just say “screw it let’s keep trying that direction.” LLMs and their harnesses tend not to have that base assumption like iteration on decreasing slopes to guide them closer to convergence, it’s a lot messier.
Any tips on the best approach at running this at an M4 Max 128GB? Token throughput was a bit slow with the last 27B one (MLX), ended up using the A3B variant but if I could get this one to reasonable speed I'd much prefer it.
I have an M4 Max (unfortunately 64GB). I have been running the Qwen 35B A3B one for a while now, after testing and benchmarking a number of models. That one was consistently the best in class for tasks like despamming, E-mail classification, OCR and dictation post-processing. It was also really fast (90 tokens/s).
I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filtering benchmark, best of any model I tested, ever). But it is slow.
One thing I would recommend is keeping an eye on MTP parameters. I tested and benchmarked extensively, and I use `--spec-draft-n-max 2` with llama.cpp. Longer sequences actually decrease overall performance.
As for running, I ended up using llama.cpp and its llama-server, with a bunch of scripts written by AI, because I got tired of LM Studio not implementing the image-related parameters which made gemma4 useless for OCR.
FWIW, Qwen 3.8 27B is actually slower with MTP enabled, unless you generate a lot of code with it (and even then it's not obvious, because you'll likely spend most time thinking).
Unfortunately, that chip just doesn't really have the memory bandwidth to run this (or nearly any) model at acceptable speeds. I have the exact same chip (M4 Max 128GB) and I've been trying to optimize a completely purpose-built implementation with Fable and this is just not possible. Even if you could reach the full 576GB/s, it's just physically impossible to exceed these numbers with the model's architecture:
2 bpw - ~85.7t/s
3 bpw - ~58.0t/s
4 bpw - ~43.9t/s
6 bpw - ~29.5t/s
8 bpw - ~22.2t/s
16 bpw - ~11.2t/s
without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.
MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.
I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit a similar ~85t/s wall in practice (again with highly specialized inference).
Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.
I consider 'acceptable speed' to be around 150t/s. Why? Well, this is generally what it takes to keep me engaged with the output, rather than immediately switching to other tasks and checking back later. When I check back later, I have more catch-up to do at once, and I haven't been following the process. So I have to recall it, familiarize myself with the new progress, and sort of get back into focus with it, which is a lot of mental work (even if it happens quickly in real-time). I prefer not to have to do this because of how much work it is, so I prefer to watch the agent in real-time and try to follow its reasoning. That also lets me interrupt it quickly when I see it about to make a mistake, or see an important detail I left out.
Interesting, for anything more than side chats/projects I usually am watching the output generate and thinking about the problem. I have the same issue with switching back, takes a while to recall and page everything back into my context, so I try not to alt-tab away.
Go for a slightly more quantised version, and experiment with different MTP settings. I find that MLX versions are marginally faster on my 64GB M1 Max, but I usually use Unsloth's GGUFs via llama.cpp as there's a much greater range of quants available and I prefer llama.cpp. MTP sometimes also helps a little, but I suspect it's less helpful on my system than others.
This is the way if you need speed. It costs a little bit in smarts, but compare the MTPLX option listed above with the oQ4e-mtp quant using oMLX. The good cacheing layer in oMLX will help things feel faster for some classes of tasks in my experience.
Wait for the MTP variants that will likely be out within days. I'm on a 128GB Strix Halo box and for 3.6-27B 8bits I was getting about 9tok/sec (not great). With MTP that gets closer to 18 tok/sec (kind'a usable).
The MTP is available, but I'm definitely not seeing 18 t/s on the Strix Halo from the 8-bit quantization, even with MTP (more like ~10 with full context on long tasks). This is a slow model (but so was 3.6). What's your exact llama-server command that gets 18 t/s?
27B is a dense model so it will be slower with an MoE (A3B), but should have better quality? I still haven’t found very good uses cases on my M3 Max for dense models. Even if you can find a MTP version, it doesn’t help much, especially if you compare against an MoE with MTP as well.
I hope the bonsai team makes another 1bit quant of this model (or releases code/instructions on how to do it), using the Qwen3.6 27B on my 16GB mac mini has been wild . The 1bit quant feels like opus level… for the first couple turns. Then it has trouble eg switching from plan mode to act mode. This is mostly mitigated by starting a new session. (tbf this limitation is called out on the hf page)
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
nope just my normal m2 mac mini. I bought it just as a normal computer to do my taxes and whatever, so it’s mind blowing that I can run this kind of AI workload on it. Well, “run” might be generous, it gets like 3-5tok/s
I’m working on a setup that’s more geared towards running tasks overnight so the slow tok/s doesn’t matter as much
ahh gotcha. that speed sounds about right. i am running qwen 3.6 35bA3B(group-32 int4) on my m3 air (16gb) and get about 8 tokens per sec. i have not yet tried the 3.6 27B.
100k is all the context I have ram for, this is with any auto-compact turned off. This is using Cline in vs code. I’m sure I could tune the system prompt and mode switching more to work better with this specific model, but I haven’t gone down the custom harness rabbit hole yet.
And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much
If the benchmarks are a real indication, we now have a local model that is runnable on a high-end personal PC that trades blows with the leading model Claude Opus 4.6 Max from half a year ago.
can you tell me ideas of usecases of 9 or 10B language models ? I cant find any usecases other than training a lora on them to give good bash commands for example
I use Gemma 4 12B in the 4-bit quantization for all sorts of vision tasks (image sorting, classification, description). It's also good for the same sorts of things for text (but there are probably better/faster models for text, 12B just happens to excel at vision tasks). The Qwen 9B is also very good for those tasks. If you need to do any kind of "search the web, grab some data, do some kind of action" tasks, these small models are perfect for that. Scraping data in a fuzzy format into a database or report or spreadsheet, producing a dashboard of news, etc.
Small local models can also be used for sub-agent tasks in most agent harnesses. But I'd probably run a larger MoE for that; they're faster and have broader knowledge. The dense models, even very small ones, are not blazing fast.
I don't code with any models small enough to run locally, at least not so far. Qwen 3.8 27B might be the tipping point, though. It's looking really promising, though it's probably slow enough that I won't ever actually use it. I'd rather pay $100/month for a faster model, even if Qwen 3.8 turns out to be smart enough for most of my work. Running it locally with the 8-bit quantization is going at 12-30 t/s, depending on how much context it's chewing on. So, if all you do with AI is coding, then you're better off doing it in the cloud.
But, there's lots of things a small model can do that aren't coding.
I found this kind of amusing while running it (using Pi as the harness). Don't know if this is evidence of intense fine tuning from Claude but it smells like it...
" The user wants me to explore the repository at XXXX and report back. Let me start by understanding the project structure, reading the
CLAUDE.md file, and getting a general overview of what this repository is.
Let me start by reading the main project documentation and exploring the directory structure.
I'll take a look around this repo. Let me start by getting a lay of the land.
read resource CLAUDE.md (ctrl+o to expand)
ENOENT: no such file or directory, access 'XXXX/CLAUDE.md'"
Architecture thread! Afaict they continue to use gated attention + delta net, which was also adopted+adapted by K3, but im surprised theres no improvements to the residual stream (deepseek are using manifold hyper-connections, kimi have attention residuals) ?
I use the Qwens as a vision model for my DeepSeek V4 Flashes to handle. But the Qwens run on old RTX A6000 Ampere. Does anyone know if there's any news about INT4/AWQ quants for the RTX A6000?
Same, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it.
Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.
Yeah, I might just replicate what you're doing. Main issue right now is just finding spare vram to actually run another model in parallel... And yeah, if I train up a vision adapter somehow, I'll try to put it up/post about it, seems like we're getting the killer apps for local LLMs right now, where it's just feasible enough if you're enthusiastic enough to be a bit economically irrational, and just useful enough to sort of rationalize.
Model benchmarks are useful, to a point, but it is the long tail of things you do with the model that determines if it's good at a wide range of activities. Ant/OAI, to their credit, build their models -- even the small ones -- so they follow instructions and do tool calling well, without the system prompts confusing them. This is especially important for long-horizon tool calling.
So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.
Why don't qwen/alibaba host the model themselves? I was looking forward to trying it on their coding plan. Google are the same way with their Gemma models.
Can anyone who has that specific personal test he tries on different models , and tries this model , to tell us here if possible , how good or bad is this new model ? compared to others ?
There is a down to earth guy on YT that performs a series of tests against LLMs running on non-god-tier commodity hardware. He will likely be testing this soon enough.
So much potential for that channel. He's got a nice range of tests and a no nonsense presentation style.
However, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.
running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...
this model--
1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)
2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently
3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)
thanks qwen!! hoping to see the full model range get released...
I've been experimenting with a few settings in my 4090 , and if 3.6 run at 90-110 tps, 3.8 staya below 80 tps and it's most often at 60 tps.
I'm using flash attention, mtp speculative decoding (n=2). I've looked at the club-3090 repo, but haven't found anything meaningful but get back to 3.6 performance
I thought the same. But why claim something so shocking when it can easily be discredited and puts your reputation at risk? If they’re claiming Opus 4.6 level, I expect it to at least match Sonnet 4.6.
strong visual reasoning apparently, which is nice. still lacking native audio however. hoping for more companies to embrace the spirit of something like `gemma-4-12b-qat` for actual multi-modality (text, image, video, audio).
