Have you tried increasing the repetition penalty? Needing to do this is already a red flag, but I'm curious if it helps at all.
Running LLMs locally on outdated hardware, I've come to the realisation that sub-10b models are broadly speaking kind of stupid. They can answer simple questions, but can't really solve problems.
The effect is so strong that with an 8GB GPU a 27b model can sometimes finish the task earlier than a 9b model, despite residing largely in RAM. If it's a MoE model, then it isn't even a competition.
> The moat of hardware effects processors has just vanished.
There was never one to begin with. With a sufficiently good quality generator and a digital oscilloscope one could reverse engineer any kind of DSP - including harmonizers.
The real moat is the same as in every business - customer service, durability, warranty - things like that.
If someone vibecoded a clone of Valve's Steam nothing would change, because the system itself is not the thing that makes their business so valuable.
200-250 is the rate I've seen my context window fill up during longer tasks with Claude. The 40-80 figure is what Google's AI mode says, but I don't know where it got that from.
I've been messing around low computing power hardware to run models locally and to me 100 is already plenty fast. You most likely won't notice a huge difference between 184 and 250, unless you're running some serious multi-age agent circus.
The context window fills up also/mostly from prefil which is supposed to be 20+ times faster or so. I think 40-80 tps for decode/output is believable for claude models
> Does any real human, aside from the fanboi faction, really love "agentic programming"? That is to use prompts to autogenerate text that is source code?
I don't love it, but it's my job now and I would prefer keeping it.
I'm not breaking any new ground with the project I'm in - just serving our users and there's a lot of stuff requiring improvement that would normally take a great deal of time to finish if it weren't for LLMs.
For my own things I would rather not use LLMs, but it's tempting, considering how little time I have for such activities nowadays.
> Human development is all about removing minor inconveniences.
That's a uniquely American perspective.
Japanese homes often don't have space heating. The Swiss often keep their homes at 20°C in the winter.
Most of the globe doesn't travel everywhere by car. Arguably it's the intention to remove pain as much as possible which resulted in the opioid epidemic.
There's a point at which trying to get rid of inconveniences becomes harmful.
20C in wintertime is quite cool for North America, in my opinion. I think 23C is commonly considered "comfortable," but obviously that will vary significantly by individual.
Wow. Obviously there are humidity differences during different seasons, but 23°C is probably the limit of "this is uncomfortably warm" for me if I set my thermostat above that.
Me too, but my husband and guests are clearly uncomfortably cold at my preferred 66/67, and my sister with poor circulation just about has her hands go numb.
For me, it depends on outside temp. The main advantage of AC is dehumidification. So if it's 95 degrees and humid outside, 78 degrees inside feels great, and keeps the AC running often enough to keep humidity down.
If it's in the 80s outside, I'll probably drop the inside temp to 76, again to keep humidity down. I rarely set it below that. 74 feels cold in my normal summer attire of shorts and a tee shirt. If we have an 80 degree day with low humidity, I'll open the windows.
I don’t think your thought is as insightful as you think it is. You simply fell for the American Exceptionalism propaganda thinking that Americans are some special breed of humans. America is the richest country in the world. And for most of the 20th century average Americans had the most disposable income and the largest consumer base of all countries. Of course they led the way on adopting luxuries and reducing discomfort. The rest of the world followed as the price became attainable and their purchasing power increased.
50 years ago hot middle eastern countries had no AC. Not because they had some sort of symbiosis with nature and the summer heat. It was because the average AC costed the sum of 8-month of average income. As AC prices dropped, and the average income increased, having an AC became the norm.
Of course the richest people, richest economies get their first and to a non inquisitive or surface only thinking, it’s some how a unique was on discomfort or lust for luxury, when it’s just simple economics. You can see rich Asian countries and cities leading the “War on discomfort” now a days as their economies have developed and stabilized, but they face unique discomforts that previous rich nations didn’t.
In absolute terms, yes. But per capita there are quite a few richer countries, like Switzerland, where people genuinely seem to brave the elements as part of their culture.
Sure, everyone adopts conveniences, but I've never seen anyone do it to the extent Americans do.
The per capita numbers are more recent as opposed to be throughout the 20th century but even still the size of the Swiss market (in absolute terms) is tiny compared to the size of the American market. Unless the per-capita wealth or disposable income is 30x to make for the population size difference, the "reducing discomfort" goods market will still favor the larger (overall) richer and bigger markets then trickle down to smaller ones. I highly doubt the Swiss endure through cold winters by wearing more layers in doors and drinking lots of hot tea.
> I've never seen anyone do it to the extent Americans do.
Sounds more like observational or cognitive bias tbh. For example, the Japanese market is notorious for building gizmos and machines for far less grand inconveniences that the weather or long-distance travel. They eat a lot of rice in Japan, so it makes sense they invented the rice cooker. To many Americans (today) who don't eat a lot of rice having an entire appliance for just cooking rice might seem like a crazy war on discomfort and too much of adopting convivences. What's next? a machine just for Turkish coffee?
Running LLMs locally on outdated hardware, I've come to the realisation that sub-10b models are broadly speaking kind of stupid. They can answer simple questions, but can't really solve problems.
The effect is so strong that with an 8GB GPU a 27b model can sometimes finish the task earlier than a 9b model, despite residing largely in RAM. If it's a MoE model, then it isn't even a competition.
reply