I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.
It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.
I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?
I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump.
This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.
Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.
So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.
Mine's never said such a thing, that's a good idea though.
I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.
Is this cool? It seems like a typical Elon Musk thing some teenager might like, similar to a monster truck rally mixed with heavy dose of financial fraud.
In the last couple of weeks, I’ve seen headlines about SF AI safety workers intertwined in a sex cult, a crowdsourced Bay Area list of guys who committed sex crimes, and now this. What’s going on over there?
Ha, yeah, pro-AI interests have lost some velocity so they're pissing in the pool a bit. Keep focus on oppositional interests while they figure out how to rally. Just vomit threads, nothing of substance.
Stories about lurid sex deviants and socialists threats always poll well, and 'computer nerd sex party' is the new cherry flavor titillating bored suburbanites.
They also did a hit on Bernie Sanders on X for his "no artificial super-intelligence" bill, if you saw that, spun as "crazy old socialist scared of computers wants to take away your LLMs".
SF had the Haight Ashbury clinic (where Charles Manson went and which may have supplied Manson with his LSD), Nexim and OneTaste. It’s a very cult-heavy place.
My team is in the agentic orchestrator phase. I like this software factory pattern in concept, but our biggest challenges in development are acceptance testing of anything UI-related. Mobile app testing in particular is still a huge bottleneck that requires a human. AI models really suck at identifying poor usability and jank, particularly because they only typically process snapshots of the app from an instance in time.
I know there are traditional testing frameworks that can detect jitter and frame drop to a certain level. We could potentially start having agents build that in.
If we had concrete designs and specs on every project, that would also be helpful, but in a fast-moving startup, that gets delegated to the builders. That puts a human back in the loop every time.
Curious to hear what anyone else does to fully adopt a software factory pattern.
I totally agree with this. Agents seem tremendously bad at UI to me. Maybe it's just because I am a back end guy.
Right now I'm working on a declarative UI framework which can help me along here. My thought is that if I sacrifice a little control for sane primitives, that will make that spec /build loop easier.
I think ClayUI is a really interesting "reduced instruction set" for UI. I don't know that immediate mode UI is the right call for anything web related (that's how you get React lol) but his reduced primitive layer is very interesting to me
> Right now I'm working on a declarative UI framework which can help me along here
After your first para, I was about to suggest exactly that (well, maybe not writing your own). I find that frameworks (both front-end and back-end) constrain the LLM's choices and result in both sensible defaults and improved consistency.
Of course, you will immediately hit the problem all frameworks have: customer requirements that the framework components don't quite meet.
Probably one of those things where it's best to try it yourself, because I think anything that constrains the LLM is an improvement.
Think up a simple CRUD problem (lets say a book database - Book *-* Author, Book 1-* Publisher, Book 1-* Shelf). Write a loose markdown spec, and have the chatbot one-shot it in [language of choice]. Then do the same thing, but tack "build it in Filament/Avo/Django/AdonisJS" on the end.
I'd be interested to see if your results match mine - everything's just more consistent, and smoother, and you start getting freebies like autocomplete. The framework acts as guardrails and the LLM has to make fewer choices, so less of the token budget goes on bikeshedding.
I think Google/Meta surveillance just feels like something painless, benign, and invisible. You couldn’t see a reason a huge corporation would really want to target you or put you in danger. With Flock, your local law enforcement has access, and that feels like a much more real threat (warrantless tracking of your real-world activity with the ability to ruin your day/life), not to mention how it’s already been abused to track exes of officers, had be easily hacked in the past, and that AI face recognition matched the wrong people.
Honestly I think the only significant difference is in branding: Flock isn't consumer-facing and is explicitly marketed for tracking people's movements. The major tech companies that make your phone, browser, maybe a few smart home devices with always-on microphones or cameras, all have more data about you and your habits than Flock could ever obtain, and with us paying for the privilege no less. I get that the average person doesn't care when the product works as advertised, but I don't think that's healthy. People should care as much about privacy when it's Google or Meta as when it's Flock.
The U.S. leadership has a bad habit of starting conflicts that the bulk of the population doesn't want, which leads to large political costs and resulting half-measures. Our current leadership literally ran on "no more wars." Morons.
I wonder if this is a sign of things to come for dirt-cheap model hosting: no servers running old versions, only new versions. Just to keep costs down.
Gemini is still my top choice within production software for typical data extraction from unstructured data. Gemini Flash Lite feels like a cheat code for speed, and it's really cheap.
Some other Chinese models are also fast and cheap, but a harder sell in a U.S. production environment.
Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate.
3.0 flash (not lite) handled it like a champ though, fwiw.
Hi, I’m the author of this blog. It depends on how strong of a model you need, but in general, Qwen is easily the best among the Chinese models right now.
Googles local gemma models which target roughly the same parameter count range, are known for being a lot better at vision tasks than qwen, no idea if 3.8 has changed that though
reply