It certainly has compaction (since the public launch I assume) and I HATE it. I have some remedies but nothing perfect yet. It never retains ALL the crucial bits. If a conversation runs into two compactions it is often a sign that I have to abandon it and retain whatever I can, to form a seed prompt for an adjacent conversation.
That is really the biggest beef I have with agy over others, the forced auto compaction at the 250k token threshold (3.8-flash), while the model itself (via API) would be fine with a 1M context window. Even if the model is great, restricting context to 250k tokens (and auto compacting no matter what) limits certain applications and workflows somewhat.
I find this discussion interesting. I've had huge increases in accuracy and huge reductions in token usage by capping my Claude models at 200k instead of 1M. I find 1M unusable and wasteful and feel that 200k should be the default. This also makes sense given that the whole reason people use "Ralph loops" is to keep the context window small for all tasks to get better results. Of course, clearing it yourself and manually managing it is better, but if I have 8 projects going in different terminal tabs I'm not watching any one of them that closely to effectively do that.
- That should nowadays be very obvious. You start any substantial work by doing research. LLMs are [perhaps surprisingly to some] good at that. They are also pretty good at doing the plumbing. That is a godsent in the hands of a good researcher and architect. And I don't have any qualms if tokens are "wasted" on research or on the precursor work. That doesn't replace reading good books on a subject but it DOES replace finding and filling knowledge gaps.
- But that shouldn't be confused with getting the LLMs to make the decisions. I believe that would quickly ruin a good design, unless the decisions are about truly inconsequential aspects, which are very rare in language and API design.
- I don't believe that (sharing) is to the detriment of LLM providers either. Not realistically. We would build faster and the questions / research directed at LLMs would be more sophisticated. Believe it or not, they can't cache questions as easily as websearch queries. If anything, I believe the more people learn to use LLMs effectively (rather than just to generate slop), the more their usage will be ingrained in daily life. Some of that will be redirected towards current LLM providers. But perhaps more of the economic share will increasingly go to hardware providers, as more and more people will be interested to run their own models.
I don't get your point. The planet will not realistically run out of either. The water price goes up, then soler + desal becomes more cost effective, then people get to use the material that covers 75% of the planet's surface a bit more. Similarly, when oil gets more limited the price goes up and the planet gets to keep its oil while people stop being able to afford burning more of it. Or perhaps this was exactly your point.
Like taalas.com (very recently acquired by AMD), or cerebras.ai (whole wafer is a chip)? As you said, I also think that is one of the main direction many companies (and academia) is moving to.
To me, the legal concerns seem blown out of proportion. If you use open weight models (of dubious origin) to generate code, you can still verify them with code review and tests and other methods, used to verify human output, right? That is still a great win, maybe not as much as having AI write all the code, but that would be a reasonable point in control vs quantity spectrum for most solid and well made products.
But if you let LLMs talk to people (customers, for example) directly, then yes, you need an LLM provider that you can hold responsible.
That would only be a minor marker. A major marker would be whatever the Chinese equivalent(s) of Amazon are (Alibaba? etc.) going that route successfully. From academic / industrial conference presentations it appears Chinese services are banking on automation far more than an entity like Amazon does.
That being said, I don't know what exact state the industrial automation technology is there and I can only extrapolate (or do websearch, which didn't lead to enough details; I only found things like https://www.youtube.com/watch?v=JnUGgc8R3ng).
When something like https://www.allegrohand.com is mass produced and used industrially, I bet the last mile (meter?) would change a bit, and full automation would be easier and less finicky.
Not sure about that. Labor costs are lower in China and the CCP is strongly incentivized to keep enough jobs around for humans to maintain their own hold on power.
Labor being cheap is more of a locale dependent condition in China now, their well developed cities and industries have had rising wages for 2+ decades now and the labor discount is small enough to be more like a side-benefit to already manufacturing in China, rather than a driving force to outsource to China. China has itself started outsourcing certain productions to chase cheaper labor prices.
That labor costs are rising and are now significantly above poorer countries like Bangladesh or Myanmar doesn't mean they're anywhere near high-income countries or above the cost of automation for low-volume tasks like last-mile delivery, which drivers do for under a dollar per trip: https://archive.ph/SizjZ
Automation can make sense even with low labor costs if it enables higher volume. Like nobody is paying humans to chisel individual screws by hand, because machines can do it quicker and better, and therefore cheaper.
But you can't substantially increase the volume of last-mile delivery by replacing the human with a robot, because the robot would have to move faster, and moving faster would be dangerous. So robotic automation of low-volume tasks can be expected to happen in high-income countries first.
Labor costs are lower, but rising, and already high enough that automation can be appealing.
The Chinese government is also pretty good at finding things for people to do, so I doubt that's a factor. Go to any big park in a city and see how many people are sweeping up leaves, or guarding a building/area that isn't particularly secure.
This year, the migration projects I have worked on were some of the most fun. Migrations have changed from the death-by-1000-cuts that they were, to the which-AI-method-to-best-ensure-100%-fidelity. So, I don't believe the "very hard" part. Obviously it would involved recreating or reshaping many features that very involved in an integration to have a disruption-free move, but I think the complexity is manageable for most companies if they put in some effort.
This fun was not revenue-generating, I suspect, so it's already a loss. Also it's a long-tail change, which will likely create unexpected situations here and there, and de-value experience and knowledge already available. Cumulatively it can be much more expensive than staying, unless GH will continue to falter, of course.
Compute In Memory has been most of what electronics research (more accurately, the parts I am interacting with) talking about this year. There are a few trends and I am curious to see which ones survive in the market. RAM designers are hot in the market and job openings at the moment.
I found HTML itself works [slightly] better with some LLMs (that I happen to frequent). So, when scraping, I parse the resulting html, simplify it, and produce simpler html contents (closer to semantics of what I guessed the real content was). Given processing tools for html are far more mature, I tend to keep content in semantic, low structure html.
