I though 4 bit (mixed) quant is pretty standard, at least for local models that basically what everyone runs? Unless I'm mistaken it can really be a negligible difference in perf.
Yeah, and if it wasn't for dynamite, we wouldn't have the Nobel Peace prize, which we give to warmongers.
The Apollo program was taxpayer funded. It's priority was to develop and test ICBMs for delivering nuclear weapons, the most destructive thing man has created to date.
The IC's were for the same program.
We went to the moon on a murder mission, so you're right, this is not entirely unlike the Apollo program.
Destruction of the future only requires your passive participation and rosey glassed outlook.
A pox on anyone who does the math which would break those glasses.
Social roles get created at the frontier of what isn't technologically feasible without a human, as culture develops around some form of human activity. Role A does B, role X does Y; I don't know, maybe its just time for new roles that didn't exist before.
Why use apps that you dislike, just ask Claude to click through them for you. And no I'm not being ironic, this is the way things are headed, you just abstract over all the crap by having an agent do it for you.
The main difficulty with fast (low-latency) inference, is not actually computation but loading parameters from memory. The problem with generating 1 token at a time isn't that that its expensive computationally (it is, but so is training), but that you need to stream your entire model from memory for every single token (and also the KV cache but that's besides the point). So the strategy usually taken is to process big batches of user requests, letting you share the memory loads across users. This means individual answers aren't that fast, but you can do lots at once. This is why local inference isn't cost-effective; its because the model weren't designed for it in the first place.
My hypothesis for Jev is that they simply generate many answers independently in parallel from your prompt, and then discard the duplicates (or train to avoid duplication with attention between the ). In that way the entire batch is 1 user's prompt.
reply