Maybe I'm missing something, but why couldn't it be generated by the model? In older classification tasks with transformers like BERT, you could absolutely obtain a confidence score.
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
I wonder how well it performs in practice, because I can't help seriously doubting those benchmarks. That would put this 6GB model in Opus 4.6+ ballpark. Granted, that's mostly to Qwen 3.8's credit, but it's hard to believe that Qwen's already unbelievable capability density can still be compressed this much more.
I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
That's correct. I think there can certainly be issues even with 4bpw with naive quantization (IE you'll notice far better results from a QAT 4bpw vs a naive 4bpw).
One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
1.76 bpw number is kinda misleading if you compare it directly to IQ2/Q2. The encoding is ternary, but the quantization procedure is way more sophisticated than "round Qwen weights to {-1,0,+1}."
They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.
> something I was doing with my "strict MVP" build of Anubis' WASM wasn't in fact sticking to just the MVP features of WebAssembly
Fun fact, this _used_ to be the case - wasm32-unknown-unknown had extra non-mvp features added later, which in my eyes has been a breaking change on stable. You're not the first person to have been bitten by this, there's [1] and a similar story in Ruffle [2].
Weirdly, the response from the rustc side (IIRC we also talked with one of the devs of wasm component on discord a bit around 2024) has contained a lot of (paraphrasing) "in retrospect we think wasm32-unknown-unknown target was designed badly*, so we'd prefer if you either dealt with it or switched to another wasm target than us fixing it". (*I acknowledge that the way std was implemented on that target was unusual and "not clean", but it works _really well_ for a lot of people.)
> I looked into the process involved for rebuilding the standard library twice: once with only MVP wasm features enabled and once with an "all yes config" like usual. Based on some research I did this seemed like a massive pain.
That's what Ruffle does [3], it's a bit unfortunate but wouldn't call it a massive pain.
> Weirdly, the response from the rustc side (IIRC we also talked with one of the devs of wasm component on discord a bit) has contained a lot of (paraphrasing) "in retrospect we think wasm32-unknown-unknown target was designed badly, so we'd prefer if you either dealt with it or switched to another wasm target than us fixing it". (I acknowledge that the way std was implemented on that target was unusual and "not clean", but it works _really well_ for a lot of people.)
That's about right. Specifically, wasm32-unknown-unknown is serving multiple targets, which aren't very compatible, and it implements std by panicking in all the bits that can't work.
One target is "you truly have absolutely nothing, and don't really have much of `std` at all". Another target is "you're in a browser, and can do anything a typical browser can provide" (ideally targeting the future where wasm has full browser APIs without going through javascript). Another target is "hosted environment that isn't a browser, and which capabilities you have depends on the environment". Those are three very different things.
I would argue that we need at least two new targets, possibly three:
- `wasm32-none` or similar (by analogy with `x86_64-unknown-none`), a target where you don't have anything except for what you bring with you. The equivalent of writing kernel code.
- `wasm32-wasi-hosted` (or pick another name), where you the hoster of the wasm can supply whatever capabilities you have available.
- `wasm32-wasi-browser`.
It's possible that `none` could be a special case of `hosted`, where the services you provide are "none".
It’s trading one property for another, isn’t it? Standard beziers with symmetric handles are C1 continuous but not necessarily G2 continuous; author’s scheme is the other way around, which makes them better for some use cases and worse for others.
Small pushback here: it (and bluesky too) is absolutely atrocious for art, at least as a consumer. Once in a while I stumble upon an artist who only posts their drawings on twitter/bsky, and it’s pretty much impossible to browse their historical works because the „media” tab is just an infinite scroll without any organization; and that’s assuming they don’t also post 10 photos of their cat or other life updates in between their art.
It’s obviously still popular for posting art for many other reasons (I just dislike putting „X”, „art” and „does well” in the same sentence). So IMO the replacements - and lots of these use cases already gave replacements! - struggle simply because of network effects, not their inherent quality.
If I ask a model to do a change involving editing a file, and it starts investigating internals of my build system, then sure it might not be counterproductive or break the task, and might have taken only extra 30 seconds; but for all I know, my quick rejection of a shell invocation (with a simple "irrelevant to the task" comment) might have just saved me half of today's Opus tokens, which already makes it worth it.
reply