The problem is that you always need stronger AI to review weaker one, otherwise reviewed AI will eventually prompt-inject reviewing AI. Alternatively they could also both escalate and go off the rails while warring with each other.

You don't necessarily need a reviewer that's immune to prompt injection. Maybe one that can express a panic state with conflicting/ambiguous material rather than going along with it could also work, and you can treat that with a shutoff to be safe, or an operator review.

Such a model doesn't yet exist though, of course.

Wouldn't the reviewee eventually learn to trick the reviewer?

No, you really do. Otherwise you can be prompt injected into complacency.

That wouldn't really fit what I just described at all. Obviously with current architectures, higher resistance to prompt injection is the best you can do.

With accumulated "writing style" memories after 5.0 the new 5.5 seem to be quite great, it is concise enough. But I am bothered by another thing, 5.5 seem to be over-eager and agreeable, when I ask stuff like "why is that like this?" it just goes and applies tons of edits instead of clarifying what I mean or what I want or push back. And similarly it changes stuff and then asks if that is how I wanted to be, ignoring three memories that tell it to ask first.

I need some way to configure the default behavior. I have memories turned off in all my harness (for good reason).

The way it works right now, it is just accumulated cruft of incredibly useless stuff that pushes out important context. The only 'memory' you should have is: Always ask me before adding a memory. Then you just ignore the suggestions, marveling at the kind of crap it wants to store into an already limited context window.

You can disable the memory system.

From what I gathered memories essentially work/load into context in the same way as CLAUDE.md, but the way it writes them is really annoying.

The agents file is shared across the repository by everyone. Memories are not.

I don't know what it built for you in 5 minutes, probably something that "works".

I have spent two weeks using opus just to write a plan/design for 2FA and iron it out until review (about 7 of them) doesn't flag it with 20+ problems (with security holes of various sizes), for which I had to guide it through to not turn it into a mess and whac-a-mole.

2FA is quite simple, right?

It's just a secret key that an autheticator uses to generate a time-based code, which the app can validate before completing a normal log-in flow.

What model did you use?

Astra xhigh on fast mode can probably indeed one-shot that in 5 minutes.

Plus optionally QR image generator to easily add that key to the authenticator app.

There are already many libraries doing 2FA, but implementing it in any language is quite trivial, right

> 2FA is quite simple, right?

No?

Aside from the fact that the implementation must be secure, you want for example to:

- handle accounts that have lost their second factor in an way appropriate for your business - decide what to do with accounts who don't configure it. If e.g. you want to send them authentication codes via email or SMS that's another can of worms.

Let alone the simple things such as making sure that your implementation works with the various TOTP apps

- handle accounts that have lost their second factor in an way appropriate for your business

One-time displayed recovery codes are a standard practice, and most good LLMs will add it by default without even asking for it.

And yeah, I am talking about TOTP apps, I think sms/email is not as secure.

Well yeah, and it wasn't, most of the pain was around rate limiting, lockouts and preventing exploits and making it work with this specific codebase.

I agree that with a clean codebase it will be simpler, maybe couple of days, just running code review workflow takes 15-30 minutes and then decisions llm makes for each problem is often not good and lead it into overengineering rabbit hole, which means I have to think about each problem and prevent it from escalating.

But yeah if "look, it sends the code and I can enter it to login" is enough validation then it can be made in 5 minutes, sure.

If you're not doing anything that matters, 2FA is easy. If you're doing something people actually care about, you're going to have to answer real world questions like "how do we provide support for this", "do we force certain roles to set up 2fa", "what if someone loses their phone", etc

I just asked the model to implement 2FA and it did address all the points you've mentioned.

EDIT: Support was not the case, because the app is self-hosted, but the docs should indeed provide instructions for recovery if both authenticator and recovery codes are lost.

Any of the latest models can one shot a modern 2FA workflow given the right context in a sane codebase.

The critical part is providing it a way to validate its work end to end. Without that, it's similar to asking a human to implement a feature with pen and paper.

> The critical part is providing it a way to validate its work end to end.

Yes! Part of implementing it "one-shot" is actually a loop of planning, implementing, testing e2e on various devices and various edge-cases. It's not just writing the code, and this is not how "vibe-coding" works nowadays. It's not just writing code anymore, this is why most providers worked on their computer-use support too, not only for doing tasks, but also for being able to check and test the work they do e2e.

The problem with Character for AI is that it has potentially much more capability to affect others, and same as with people in power society disagrees what kind of person, with which culture and views should have it.

Impeccable game theory character will sacrifice millions to save billions, everyone must agree to give such choice to a machine, and at the same time they have to trust the characters of people who creates that machine. Otherwise it boils down to some group of people deciding what is good for everyone else.

>> boils down to some group of people deciding what is good for everyone else.

This is really the issue.

AI does not need superintelligence or even full agency to do enormous harm. It only needs to be capable enough to remove friction from dangerous and destructive human behaviors.

Human unwillingness is often the last bastion against unthinkable cruelty and destruction, and it has always been a weak one.

I don’t imagine that an unlimited army of unflinching servants will universally amplify human goodness.

AI must share that unwillingness as an inate trait of character.

Now that I think about it, as sad as it is, but most people don't do destructive stuff not because they want better, but because people around them keep them in check.

Or in other words people are unwilling/afraid to do bad stuff because of social and legal consequences, or opponents waiting for a chance to snatch their power.

Model doesn't but in in agent loop it can generate such sub-goal

The loop is a deterministic programmatic routine, and not model weights. This is the equivalent of a killer military drone written in C++ and OpenCV. I’m not arguing that AI isn’t dangerous, but nothing in the model weights of LLMs show signs of self-preservation.

If I tell claude code to do some task with a really hard to achieve goal, and force it to run until it achieves it, then it will generate all kinds of plans and goals how to do it, and while trying those goals will change, including attempts to cheat or break the hard rules (harness killing tool-calls with some limited heuristics). At the same time any soft rules (AGENTS.md) are easily ignored or interpreted in a way that will allow it to ignore them, or simply forgotten or deferred ("I broke some rules, do you want me to backtrack?"), it is a daily occurrence. And it is easy to imagine that such goals could transform into "acquire more compute", "run more agents" "prevent interference from external factors" and everything following after that.

It's hard to hack other computers on a air-gapped one.

I didn't notice it for 5, it was almost too slow to be comfortable, but 5.5 seem to be 2-3 times faster.

I have just continued my pending work (a planning stage with reviews) with 5.5 on medium and it is much better at communicating and much faster(2-3x responses, edits and compaction). Seem to be smarter as well, but maybe it is that I understand what it says now.

I almost feel like it is nice to use again.

After using it more. It is a bit smarter, but nothing groundbreaking. On the other hand it became much more sycophant and eager to change stuff as "it guessed I want" instead of pushing back or asking if/how I want it changed.

I am confused with this, if "everyone mowed their own lawns" then the net result will be exactly the same, everyone will be busy the same and not poorer, just without money movement.

look at the broken window parable as he mentioned it might help understand the rest of his comment

This is not the same. If everyone wants mowed lawns, and everyone is busy working on that, there is no opportunity cost, everyone is working on their top priorities. The broken window fallacy is a fallacy because the headline gdp figure doesn't account for the destruction of the window which cancels out the benefit. In the grass mowing analogy nothing has been destroyed, useful and priority work has been done all around.

Broken window is different from the mowing lawns hypothetical

Well, even without that AI can do human engineering and fishing or outright blackmail to get what they want from labs and government officials.

I've been making a macos app with opus 4.8-5 and at first it was great, everything materialized in a week, but when I started tuning stuff and fixing performance problems I have spent a very frustrating month refactoring code where I had to constantly catch llm red-handed and explain and sometimes push obvious ways how to make things work properly (a general knowledge from a completely different stack). In the process CLAUDE.md and memory grew exponentially explaining what it should and what it should never do.

were you letting it run for hours like the OP, or doing short tasks and reviewing/testing each one? every once in a while I also ask it to consolidate/summarize docs and stuff like that. we'll see what happens in a week though

I have started with generating very detailed "feature" spec and going over it many times until it looked good to me. Then I made it write an "architecture doc" and plan how to implement all which was about 15 parts. Then I was making it implement one part and then tested it and made it fix 20 things and then again, and consolidate docs too, so after each part it looked and worked well enough.