I'm on the same boat with a 4080, the extra GPU RAM of the 4090 over the 4080 would have been nice, but it still wouldn't be able to run any decent models.
I configured my PC in December 2023 with a kit of 2x32GB DDR5-6000 for €249, thinking I would upgrade that RAM later to 128GB by just adding another kit. Now nearly 3 years later that same kit is over €1500 :-(
I've also lost all interest in upgrading to a 50-series GPU. With exception of the ludicrously expensive 5090 the entire 50-series lineup is capped at 16GB, I don't see the benefit of upgrading.
Yea - (I wish I had gone with 64Gb ram...) there is no reasonable upgrade path for us beyond waiting, unless the 5090 comes down to MSRP, which is unlikely. The 4080 is still a nice GPU, but the 4090 would have been a better option, if we'd known what was coming!
40% overhead is quite typical, so you'd be looking at $120k/year. In the USA I'd consider that a competitive salary for an admin capable or keeping a $6M rack of specialized hardware running 24/7.
Yeah but you don't need two such people, or even one, dedicated to this single rack.
A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.
The companies which have the will and the budget to host their own LLMs typically require a ton of other, much smaller, models as well.
They have internal security requirements, guardrails, audit, critical workflows start depending on your onprem setup, downtime is now something that’s not even allowed.
There’s going to be a zoo of tooling, lots of bespoke work with internal clients who have no clue about docker, but now want their vibe-coded app to access a model they downloaded yesterday, and this model better be served and monitored 24/7 because now C-levels use it.
No one is going to budget several millions to then look at an email admin who hears ‘cuda’ for the first time in their life and ask them to just support the entire thing somehow.
And god forbid it’s an AMD setup.
This obviously isn’t relevant for a 10-person startup and their second-hand xeon with a single H100.
From my experience only the companies which are REALLY interested in privacy and data security bother with hosting their own models
My company is not even the same ballpark, but even we already have people for it. And doesn't include the fact that you can get a colo location and just a MSP or a contractor to do it for you
Not really. Spain's traditional format is dd/mm/yyyy with slashes. This applies for a good chunk of Europe. Germany/Austria uses dot, I think nordic countries embraced the dash. While you might see more adoption in offical/digital contexts, I just double checked a few popular spanish websites, all slashes.
I configured my PC in December 2023 with a kit of 2x32GB DDR5-6000 for €249, thinking I would upgrade that RAM later to 128GB by just adding another kit. Now nearly 3 years later that same kit is over €1500 :-(
I've also lost all interest in upgrading to a 50-series GPU. With exception of the ludicrously expensive 5090 the entire 50-series lineup is capped at 16GB, I don't see the benefit of upgrading.
reply