Local LLM's are hit worse. Its about 6k for 5090 or 15k for an RTX 6000 and the Mac Ultra 256 is upwards of 12k.
Sure if you already have hardware, you can frankenbuild a system - but even the "affordable" dev stations of the DGX Sparks went from 3500 to 5k and upwards of 8k depending on vendor.
All the meanwhile, OpenAI pushed Luna 6 which is crazy cheap suggesting they have flash models to compete with Chinese models.
I just hope we see more open weights. Nvidia has NEMO but their license doesn't allow NEMO to be re-used on say, Apple or ROCm - Nvidia hardware only. Big ol MEH
GPT is still 20 a month for most people, 50-100 or 200 for pros.
Meanwhile, for local LLM's - everyone is chasing hardware that is crazy expensiv eand to get around it they're leasing it or putting it on credit card. DGX Sparks are insane, and you need 2 for most people talking in this thread. Mac Ultra 5 is amazing but 6k minimum 12k for the build most want so many people lease it for 240 a month which doesn't even include electricity or time in setup so instead of talking about facts, we get into weird arguments like console wars where people give APple a 5 trillion dollar company 250 a month for 36 months and don't even own their hardware money "because they can run local models" vs just paying for output from their choice of frontier.
I say this fully loving local llms and embracing them, but the reality is, local llms have gotten so expensive and continue to get expensive while we keep talking about this "Threat" of apis - where there are a lot more than openai and anthropic available much cheaper and competitive priced.
Oh, and they don't work better than OpenAI or else we wouldn't even have these discussions.
Everyone keeps saying this, but I don't believe it's true. I think Luna is just an MLA or sparse architecture like all the flash variants and it's just cheap to run.
In any case, over the past few years, the only thing that has gotten more expensive is the hardware to run local models while API and Subs have gotten more affordable or feature rich while remaining same price.
They can compete because they have the compute to run the volume and if it's good on agentic work, people will be less incentivised to use other models for "Tasky-y" work.
It's literally increasing their market opportunity
Local LLM's are hit worse. Its about 6k for 5090 or 15k for an RTX 6000 and the Mac Ultra 256 is upwards of 12k.
Sure if you already have hardware, you can frankenbuild a system - but even the "affordable" dev stations of the DGX Sparks went from 3500 to 5k and upwards of 8k depending on vendor.
All the meanwhile, OpenAI pushed Luna 6 which is crazy cheap suggesting they have flash models to compete with Chinese models.
I just hope we see more open weights. Nvidia has NEMO but their license doesn't allow NEMO to be re-used on say, Apple or ROCm - Nvidia hardware only. Big ol MEH