My take as well. The dual-3090 and dual-spark configs have an outsized voice in this conversation. I'd add pipeline parallelism vs tensor parallelism and in the case of TP setting your lanes and speeds per card.
Would love to have multi-gpu support eventually but it's a balance between whats popular and what actually has good quality benchmarks available. Multi-gpu numbers can also vary between setups
Influencers have latched onto the pitch that everyday people can run frontier models on an 8gb GPU while sticking it to the labs. There's a large audience of people who haven't had the hardware to test larger quants to see the difference.
They would be better served with smaller models that can reliably call tools and generate structured outputs without looping or totally hallucinating. These projects exist, but aren't getting amplified.
Most popular would be Qwen 9b or Qwen 35b with experts offloaded to CPU. Even these aren't great, but they can be run at higher quants on lower hardware. We'll see what Qwen releases version 4 in the coming weeks.
While I know it's not apples to apples, the target comparison right now is 2x DGX Sparks. Similar price, 256gb. The conversation has focused on memory bandwidth vs. compute in agentic loops, so for most people the raw numbers will mean less than the "time per task" in coding benchmarks.
This is a great article and bodes well for the M5, but we should expect more like this comparing to other platforms before we truly understand where it fits.
Speed vs task-completion is a new conversation and a great point. Whereas the cost to compute doesn't exist in a vacuum, making mistakes costs less, is easier to maintain with granularity and a whole host of other factors when you own the lab.
Can you share what tools and processes you're using to do this?
I've been using Pi to build custom extensions and wrapping workflows in shell processes to make it more deterministic and enforce certain validations, all guided by Fable. This isn't production work, though, just playing llm factorio at home.
What you want is a bunch of sessions to replay. Something anonymized if it's not yours, and something that's not depending on state.
You replay all your sessions against your harness, and then store all logs all output, everything to a safe place.
Finally use a blind judge to check everything, and score the output.
Then fix your harness, iterate again until better until you are in a point where it's just the model's weakness. If you get to that, use a bigger model.
It's exciting, but that's $9k for a model that's in the ballpark of last year's frontier. For many people that's like 10 years of up-front cost in a landscape that's getting better and cheaper every week.
but compared to the $500k+ racks running the current frontier, it does give perspective.
There's great VST plugins like "Sketch Cassette 2" that will do hiss, flutter, all of that, along with some great presets. That one's less about creating authentic sound profiles, more like creating lo-fi tracks and instruments for music production.
If llama 3 70b were available for $400 today, people would make it work. They would have 4 of them working side by side or 1 of them working with a GPU on another model.
To me, it's like imagining if Sonnet 3 was burned into an ASIC 8 years ago and then never changed. It would still be revolutionary, and today we would have an entire ecosystem of tools and services built around it, likely surpassing some of our current workflows.
The frontier is a different beast, but it would likely mean competing on price.
