1. Storing Shakespeare's work costs almost no $ in regards to disk space.
2. If the prompt doesn't include "Shakespeare" or relevant terms then no regression is performed for that topic and therefore there is no effective token cost.
Someone may correct me, but I think it's not a big $ win to exclude relevant topics from the models' overall capabilities. Instead you'd tune weights so that #2 better identifies what is or isn't among the relevant terms on which to run regressions.
I think you're probably missing the forest for the trees here... broadly speaking, these multibillion "parameter" (whatever that actually means) models store a lot more than shakespeare, and have storage costs in the 100s of GB/TB (which translates to $$$$$$ in SSD/RAM costs), nevermind the (kilos/mega/giga)watts involved, all the pollution, etc...
Meanwhile, a template (maybe a couple KB) costs less than a couple cents to store and run. Large Languages Models are not really interesting, (smaller) LLMs that only contain "what you need" are.
To me it feels like the difference between crows and humans. Yes crows are smart, but what we really want are all inclusive models capable of human "thinking" . I guess we probably need a mix of both so we can give the crows the easy jobs freeing up resources.
Right but if the cost difference is negligible, as I've pointed out in another comment, in what case would you prefer to tell the crow twice to do something that a better weighted large model could do in one prompt?
I don't see it. In every case my time is more valuable than the cost of the prompt (so far) so the higher dollar cost, one shot, "getter done" model is the better net value option.
This is especially true when taking into account that the crow doesn't just fail on a single prompt, it does something much worse. It creates new problems that need to be undone afterwards. It confidently duplicates, triplicates, etc. a damaging work output that then needs more and more work to clean up before starting over.
Claude Fable 5.1 currently estimates its own total data set at 45TB.
That's a trivial number for sharing between a small user base of, in my case, 150 employees.
What am I missing? You're talking about maybe $60k retail cost in high read speed SANs that are likely already in place for a business of this size anyway? (Probably purchased a few years ago for under $30k. At least mine are.) In a US data center I'm paying a flat rate for rack space, so power consumption isn't a consideration anyway, but honestly it really isn't that much power even if I was being billed for it.
It's really nothing. An overlooked line item on a budget sheet.
So the real cost if I were to self host is video cards. But as mentioned previously, that's not reduced by smaller data sets, it's reduced by better weighted models, right? Let me know if I'm mistaken, please.
I'm very interested to be persuaded otherwise as a decision maker. Thanks for your insight and ideas.
Considering all of the above, I'm currently of the mind that a lower cost model that makes more mistakes is much more expensive in net, actually. So I prefer the most accurate, better weighted model, not the slimmer data set.
1. Storing Shakespeare's work costs almost no $ in regards to disk space.
2. If the prompt doesn't include "Shakespeare" or relevant terms then no regression is performed for that topic and therefore there is no effective token cost.
Someone may correct me, but I think it's not a big $ win to exclude relevant topics from the models' overall capabilities. Instead you'd tune weights so that #2 better identifies what is or isn't among the relevant terms on which to run regressions.