IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
Azure's going to suffocate github. I'm curious to see what's next. Will self-hosting the code repository come back in vogue or will another social-coding platform take off?
I used to work in a small independent team of 30 people within a large corp, half of which was dev. We used to run our own gitlab on-prem, our CI/CD was also on-prem. It worked perfectly, never had down time, devops guy could configure them on-demand to our needs. Me (and some other guys) also jumped in times to times to help (mostly just ssh into the servers for health check, disk partition, etc.). Then we grew (the biz team, dev was the same) and some new PMs with fancy Ivy League degrees came in and pushed for on-cloud Bitbucket. Things went to shit pretty fast after that ... Our codebase was only a few hundred thousands lines, there was only like hundreds of commits per day, the servers our git + CI/CD lived on never saturated ...
I've mostly worked for "engineering lead" corps and a PM has only ever affected outside software sold to users except for JIRA. There's no chance a PM would be able to affect the internal code storage/ci-cd systems at anywhere I've worked in the past unless they went on some proselytizing war path and convinced some senior lead engs to convince the rest of the org to accept it.
In the example, in my 20 year career, I've seen bitbucket in use once. I'm the one who usually manages that stuff since I'm an infra eng.
To be honest, I'm very much surprised Amazon hasn't eaten Microsoft's lunch here. Offer out-of-the-box AWS instances with Gitlab, SLAs, backups and the works, should be straightforward.
We should worry less about what everyone else uses, and more about what we use. I'm self-hosting Gitea and thinking of upgrading to Forgejo. What are you using?
If you move the company too, what would make you pay someone to run forgejo instead of self-hosting it? Disclosure, I run Fjord, a managed dedicated forgejo provider.
There was never a "sense of community", there was only low friction on issue reporting and especially, commenting on an issue you had nothing else to do with. Every time an issue hit HN it would be flooded with bystander comments.
If you're actively considering self hosting, I recommend giving forgejo a try. The experience is much more similar to what GitHub offers, actions API is nearly identical to name one similarity of which there are many. Moving essentially becomes one prompt and a coffee later for a small team.
I find this recommendation funny, because I completely agree from a sysadmin perspective.
I used to use Gitlab at work, and small teams would run into so many footguns with CI that we had to throw up guardrails to prevent mistakes. Far too many links to gitlab issues that were not fixed even after >8+ years of being open ended up biting us. With GHA I haven't had that experience, and same for all of my self hosted Forgejo instances. I used to hate using GHA from about ~2018 to 2021, but they've fixed a lot of things I disliked since then.
What do you need real anchors for? Sharing of pipelines?
Maybe I’ve been burned too much by pipeline maintenance (because we didn’t have yaml anchors?) but I rather have builds defined in make /bazel/etc than in yaml. So the only thing the pipeline does is optionally restoring caches, kicking off the build system, uploading PR validation results, and saving cache. Pushing artifact etc is all done from inside the build system.
There is no “setup” like installing packages because we make the build image seperately.
probably not. The people that have done it before or willing to do it now is probably a very % of the commit volume, they leaving wouldn't change much, probably not gonna even move the exponential growth needle.
Compression is not prediction, it is recall. Can we make predictions based on compression? Absolutely. Is memory encoded into physical neurons technically compression? I would argue also yes.
However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.
Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.
For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.
I know less about this than every other commenter here, but both weather patterns and market performance do seem predictable based on past behavior when modeled at the right level of abstraction. “Sunshine on Monday” does not imply “rain on Tuesday”, but “cold front moving in Monday night” does. (Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does.)
I think this is relevant to the discourse on LLMs/programming because for months, people said “they’re just regurgitating their training set,” but now I think people are seeing (I am seeing) that they do learn more abstract models of the world than that. I don’t really know how, but it’s why they can generalize from other codebases and tools and so on.
> Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does
Stock market is even more pathological than weather here. Most statistical tools have baked-in assumptions about some degree of independence between factors under analysis. In weather, things are hard, because everything affects everything but in some random sense. With stock market, it's worse, because some things affect others in an adversarial way. A pattern discovered is not merely drowned in noise, but rather actively canceled out as other people discover and act on it. The way to describe this in formal terms is, I believe, that the stock market is "anti-inductive".
Fortunately, most things we deal with life are much friendlier to predictions than weather or stock prices.
> I think people are seeing
Better late than never, given that this has been empirically demonstrated two years ago by Anthropic. I actually don't understand why people don't make a bigger deal out of it - maybe it's because the AI hype/hate fight is sucking all attention from the room. But the point is: we not only have a thing that can learn concepts we recognize and build abstract world model, but also we have a degree of visibility and control of it we never had before. We can't see, much less suppress or stimulate individual concepts or concept groups in a human brain directly, but we can with LLMs.
Good points. I looked up the definition for prediction and I suppose I'm stretching what I view as prediction.
> A prediction is a statement about what you think will happen in the future, often based on experience or knowledge. It can also be referred to as a forecast or an informed guess
Based on my reading of this definition, compression may inform prediction but it is not itself prediction. The examples cited in the blog post are examples of probabilistic recall based on past events or instances. More context means a higher chance that the recall is more likely to be aligned.
But it's hard for me to accept the leap to compression == prediction because in my mind a prediction is an informed guess about something that hasn't yet come to pass. But thinking more about it, time is a human concept and so who's to say the temporal reference means anything at all here. Maybe probabilistic recall is the same as predictive forecasting if time is an invented concept and essentially means nothing?
Is everything fundamentally deterministic if you know everything in the universe or does free will exist?
IDK to be honest, I'm just more frequently surprised by new things that happen every day than I am at things that stay the same, even if mostly things stay the same. Maybe I just don't notice them and nothing actually ever happens.
Side note: the inevitable consequence of this line of reasoning will eventually become that LLMs given enough power are in fact intelligent and sentient, and I'm worried about how that affects humanity as a whole. Are we about to subjugate the most intelligent thing humanity has ever created, or is it about to subjugate us?
The rabbit hole gets deep quick when making the leap between a fancy recall mechanism and novel prediction, but I agree they're not that different in the end. I just believe it's important to be nuanced or else we'll miss when AGI actually happens (maybe it's already here).
Statistical compressors are all about prediction. They can trivially be reversed to make a generator (not as good as an LLM, but only because it would be a much much much smaller model). And an llm can trivialyl be plopped in there to replace the whole thing.
And a counterpoint to your counterpoint: If it was raining 10 seconds ago, it is very likely that it is still raining right now. There... you can make a prediction about weather based on historical data. It's like perlin noise, random but with discernible patterns
