> I'm curious to see what you can do on an 8GB consumer GPU these days
Running smaller 4B-7B models entirely on the GPU VRAM will get you fast inference, but you will need to scope and define the tasks well. eg, using it the model as a classifier and just feeding it from a queue.
The best performing "agent"-like model to plug into a harness that I have found so far has been Qwen3.6-35B-A3B (mixture of experts) model as I can park most of it in system RAM and CPU, while the VRAM holds the attention/shared weights.
It's definitely workable as a local AI homelab. But expect homelab levels of tuning/fiddling with it.
With the improved support for AMD GPUs I'm finally considering getting a modern 16GB card (and maybe a second one in a few years assuming prices come down)
It doesn't have to chase it down, just keep plodding along implementing the new standards as they become accepted.
Google likes to make up its own "standards" and push them out on chrome, but Safari tends to be the late-comer to supporting impactful standards, not Firefox.
Do you have a particular use-case where Firefox's lack of support blocks you?
> Safari has much, much better compatibility, but even then this browser is hated for it's quirks
I think this meme has almost run its course.
Funnily enough I just had to fix a Safari layout quirk yesterday (for iOS 16 & 17 devices only). I was excited to pick it up because it had been _so long_ since I had to deal with a CSS-related browser incompatibility. I was nostalgic :')
I raced Claude to fix it. I knew which CSS had the issue, but I needed to set up an iOS simulator to verify my fix. So I had to download xcode, which download 10GB of data for a current simulator and Apple's embedded code-assist model. Then another 8GB for the iOS 16 simulator.
Claude took about 5min to find the right CSS file and then got stuck at the same point as me: needing to simulate it to verify. It just got the old webkit binaries and hacked together its own minimal browserstack clone.
It finished before me because of those downloads, but it only found half the fix. So... I won!
Except...I made a typo (`no-wrap` instead of `nowrap`) and had to do a fixup commit.
The future set of people who once would have "work(ed) for free to design you a state-of-the-art kernel"? If the tail is long enough passionate hobbyists will do it because they love it...eventually.
> but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I suspect this is the only use-case I would consider...and I don't really have a use-case for "creative writing" that I would delegate to an LLM. I suppose for dialogue generation in games?
But yes, hard agree. Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"? So it can put great backdoors into everything it touches? Pass.
I've noticed most fine-tunes, whether "heretic" models or something else, tend to be over-fitting, or something, at least some of the time, and get kind of chaotic. I want to believe normal folks with normal resources can be involved in this stuff, as I'm working on fine-tuned specialist models as we speak, but it seems like it takes notable investment and time. My first experiment was teaching a little Gemma 4 more to write more like me with a LoRA (like you, I don't want to use a model to write for me, but I did want training data that I could ethically use, and I've written several million words on the internet over the years), and it wasn't what I would call a success. It either wrote like an asshole (which I only do, like, 15% of the time) or it just borrowed a few of my quirks, like too many ellipses, if I applied it less heavily.
I use it to write short sci-fi stories. Life is not only about being an SDE.
> Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"?
I need a model which decent enough at coding to write boilerplate for me, yet I do not need to unload in case I want to write an article or enguage in my writing hobby.Exactly why Gemma 4 are so popular.
> So it can put great backdoors into everything it touches?
> For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
Off-topic, but I stumbled upon the first Newcomen engine imported into Australia in a museum in Sydney and I was unexpectedly charmed (not an Engine Guy). It's large, but nothing like the awe of "mega-engineering", it's crude, but it clearly has such amazing utility (when compared to a reality without it) and it changed the world
reply