Even better at the local LLM stage, where QFN can catch a mistake in prior work and patch it without any subscription costs at all. then I can use a cloud model for polish and finish and move on.
I'm building stuff at home with my work focused on managing infra. If anyone wants to sponsor a SOTA model subscription I will check it out. Until then I'll see how flexible a Strix Halo setup and my homelab can be.
I also started with Qwen-3.8-27b and used for a while, I still use it so Claude can delegate tasks to it, but the SOTA models are still a lot better than local models, so much that I don't see much sense to use local models as long as I have a paid subscription.
Even if the difference is a single extra error caught early, that can end up saving me hours of work later.
Building the middle system that can take any model as a back end, based on cost/performance/client choices/etc, and API style present the same interface and functionality to end users is huge right now.
Local LLM usage is bouncing across major runtimes, custom runtimes and forks plus custom quants and variations of open models. llama-swap and some other tools integrate those and provide metrics but as the field expands can't keep up even with AI support.
I need to look at tailscales dashboard/etc again. I started on that when it launched but couldn't get my runtimes to go through it. I think the easiest tie in for metrics/etc is grafana + Prometheus but you need that stood up, and then a custom parsing job for each runtime based on metrics it reports and processing that data into useful dash.
I would love to see more standards here like OpenAI endpoints with metrics and information to link into. ./metrics is great but not standardized.
Heck I use copilot at work and use M365 copilot chat for a lot of short things and small scripts and tools over credits in Github Copilot. At home I throw things things into copilot chat free, google AI, and more to avoid cost.
Then all the real AI work I'm using my local setup on dedicated hardware for large local models. I have usede my Copilot pr sub a few times this month to check work but it's either:
1. I ask about bugs and security fixes, glaring holes, and other checks before calling the work complete and posting a release to my persona github. 2% credit usage over 20 minutes and useful small changes.
2. I ask about information in a repo I'm using, like GUFO or Halogen-flash-next, and some key info. 35% of my monthly premium credits gone before the one chat completes.
How can 20 minutes on a whole custom project looking for bugs and issues use almost nothing, then a question about a repo eats 30%+? I don't even have budget to learn how I failed as 2 more tests and I'm burned on premium requests for the month (in minutes).
So I abuse the free options for everything I can. Draft scripts/sql/etc, stage prompts or handoff/etc files, ask questions or research something and pass on to vs code/copilot chat/etc like I run at work (with much more credits through the enterprise plan).
I bought a Framework Desktop mainboard only, 128gb AMD 395+ setup, and put it in an existing case with PSU i had on hand. I had a fan for it. It was $2699 +$15 expedited shipping. The case and PSU/etc was probably $200 more (It's an overkill 850w SFX PSU and cheap but great ITX case from Deepcool).
I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.
Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.
I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.
Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.
I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.
IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.
Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.
My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.
Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.
Buttons sized to be hit by your physical thumb and buttons sized to be hit by a tiny high-precision cursor are very different things.
I am sitting on a macbook pro with the vertical tab pane open in chrome, with 12 tabs. The outer joint of my thumb covers the entire list, in both width and height. A touch-friendly version of that would be humongous in comparison.
Following the Helium Privatization Act of 1996 and the Helium Stewardship Act of 2013, the Bureau of Land Management (BLM) completed the final sale of the reserve and its pipeline infrastructure to the industrial gas company Messer for $460 million in December 2024
Decades of work to hand over a critical resource to private hands.
What management thinks is good and what employees, especially engineering type employees, think is good are usually counter.
The execs want to run the most profit to cost they can squeeze out, usually short term. The employees see the value in the senior and knowledge rich staff with higher costs long term.
reply