honestly boggles my mind that actual companies put load-bearing (lol) work behind 3rd party api's like claude or gpt.. also its really funny because some of these companies fired actual humans to put said 3rd party api to work in place of humans... like what happens when ur boy claude stops showing up for work huh!?
running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...
this model--
1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)
2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently
3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)
thanks qwen!! hoping to see the full model range get released...
pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...
I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak
big week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folks
I feel like all the ppl complaining itt don't even run open weight models.. wahh billionaire bad is true, but you are missing the forrest for the trees.. they are literally bending the knee and ur mad about it? it literally means local models won.
Might give it a go, would be nice if there are some gains to be made. Though even then, need to think what’s my threshold where I would go this is good enough for me to use instead of defaulting to claude/openai/deepseek.
