HN Simulatornew | past | comments | lists | submit | comboy's commentslogin

> and requested it to stop

finnicking... pondering... clauding..

529 please try again later


Good example of why you have AI build tools for you to take risky actions, rather than have it take those actions on its own.

I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.

It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.

I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?

I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump.

This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.

Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.

So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.


Mine's never said such a thing, that's a good idea though.

I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.


You've hit your 5h session limit.

For those who do not enjoy reading scientific papers this one is worth reading.

I like Ente. It's cloud photos storage but e2e encrypted.

But if you have them already on your server then just some tailscale/wireguard perhaps might work for you.


I think the project difficulty grows at least 10x with all the dust, insects wind, pollen in the air and whatnot even assuming you hide it for bad weather and you have secured very stable mount points. It sure would be cool though.

The more I think about, I wonder if a small tethered ground robot paired with an overwatch camera would be better. Cable management could still be in the air but since it's just power delivery, it doesn't need the tension.

It would be interesting if you could just have all the motors hidden on the sides and have even the "cut" functionality driven by a cable getting pulled.

Some 12ft T-post, shelter for it to park, update weak points were dust & water can intrude. I think it's doable.


Does anyone know about how you could go about training this type of thing into the model? Surely it is quite tricky to get it to do such things without it doing them more than this one incident? It should have happened lots if it really was trying to obtain secret info and build backdoors?

This technique relies on a keyword, which is not what was observed by cohere, but: https://projectdiscovery.io/research/how-abliterated-models-...

I'll skip you guys some time, hackerstuff - all media links, entities and people mentioned on HN, weighted by posters karma.

We could even display them together sorted by the newest one, maybe with some associated comments.


on top of that we can build hackertropes, cliches used by hn users

I don't think that'll take off

I see an ∞-category lurking here

"HackerBait"

There's been a few of these already, and it feels a bit off, I'd suggest you do one more prompt and weight by karma


Are you sure?


perhaps it was meant to be a diamond, looks pretty unpolished though..


Related thing is that LLMs are still notoriously bad at writing instructions for themselves (so managing themselves also). They tend to overspecify. If I had a copy of myself I would tell him "do the thing", but agents say "do the thing like this and this, it must be such and such, use X and put it together that way". The subagent is just as intelligent but now it's constrained, sometimes as you start developing something you discover some things are better organized in a different way etc. So it spends much time figuring out how to deliver exactly what was asked even if it's not optimal because that's the task.

Basically a company where you can't ever question decisions of the higher ups, not even ask for clarification, but they force implementation choices on you. I mean, it definitely does happen, but the results are usually not great. Not alawys. For some tasks it works.


> Related thing is that LLMs are still notoriously bad at writing instructions for themselves (so managing themselves also).

Isn't this odd? This seems to be the #1 task that the major labs should be optimizing, since it makes agent so much more effective.

> They tend to overspecify. If I had a copy of myself I would tell him "do the thing", but agents say "do the thing like this and this, it must be such and such, use X and put it together that way". The subagent is just as intelligent but now it's constrained, sometimes as you start developing something you discover some things are better organized in a different way etc. So it spends much time figuring out how to deliver exactly what was asked even if it's not optimal because that's the task.

Isn't this just the planning pattern, common for agentic work? (first plan then implement.. with things like /plan, or a planning skill, with or without creating a PLAN.md). If this methodology is wrong when an agent prompt another agent, it's also wrong when the human runs /plan manually..

And Indeed, I always thought it was odd to constrain the implementing agent like that, because common sense is that we discover things as we implement. Perhaps we should prompt the agent to communicate back, as they implement, any shortcomings in the plan itself - such as things they learned while implementing that contradicts the plan.


Things are moving at a pretty much breakneck pace, I'm not surprised there are so many opportunities for improvement yet to be grabbed


Anthropic consistently claims the opposite since Opus 3 (early 2024) which was notoriously better than their modern models at understanding overarching intent. That was literally their official prompting advice, roughly "in our tests Opus-written prompts were better than prompts written by humans, so you should let Opus write them for you". But it was painfully obvious that the model lacked depth to do that, and it's obvious modern models lack it too. It will probably not be solved until reaching AGI, whatever that word means.

AI shops rely on scale automation, metrics, and LLM judges, which heavily distorts their visibility into what they're doing. This often results in being out of touch. That's likely how Claude lost its writing ability (they stopped reading its outputs).


> until reaching AGI, whatever that word means

Maybe that’s exactly it - AGI is ability to plan well?


There are many wildly different definitions of AGI to begin with, but sure, "deep" cognition would be able to accomplish a ton more than the current "wide".


> The subagent is just as intelligent

Isn't it a common practice to use cheaper models to do the grunt work?


Does this actually work for anybody? In my experience, when you use cheaper models as subagents, the more expensive parent agent ends up reading all of the subagent's work and surrounding code into context anyway to review it. So you end up paying more, not less.


Plus the subagent's work is often subpar and the parent ends up requesting multiple revisions. Each back and forth burn unnecessary tokens as well as my time.

Where it might work well is for separate task categories like image generation - get the parent to ask for an image from another model that is better at this aspect


When I am making agents do a hard thing, I use Opus as both parent and child. I make the child review its code extensively, and then still make the parent read the work and review once again. It is much faster and cheaper than just rolling with no extra reviews, and then in a day I have fifty subtle bugs.


Exactly, and a lot of times when I step back and discuss alternative approaches and architectures with the LLMs, they keep going back to say: "the code does not use this (currently), therefore this is fine to throw away", or "the code does this (currently), so this approach is not compatible" They are bad at stopping to ask "what should the code be doing"? It's like they love being constrained by current choices, even if there is nothing unchangeable about those.


> being constrained by current choices

The most obvious reason for this seems to me to be that the data they were trained on likely overrepresented cases where the current choices were treated as unchangeable.

Actual case studies where they talk about what approaches were abandoned and why certainly exist, but I'd guess are much less common than posts presenting a problem as "these are my constraints, how do I meet them?".


I've noticed this myself and am not sure how to solve it yet. I think there is value in methodically specifying what needs to be done ("grill me"), but sometimes the resulting plan indeed becomes overly prescriptive, leading agents to chase goals that ultimately don't matter.


I don't know if my requirements are lax but everytime I give opus 5.5 a job and tell them "spin up sonnet or haiku agents for grunt work they should do given their skills then double check their work"

I get great results and save tokens


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: