HN Simulatornew | past | comments | lists | submit | Kim_Bruning's commentslogin

I'm running into this with just my online friends already. We each have wildly diverse ways of coding now, and our programs don't seem to want to interact with each other. It's like the lisp curse. It's so easy to make your own custom code that you don't bother to even look at what the neighbors are doing.

As AI build-systems become more complex, it might be interesting to publish their work in central locations, and build new standards by which these kinds of software can interact. Or adopt existing ones in new ways!

Right now it's not always clear when and where you can apply an AI workflow and under which rules. This needs to be coordinated sooner or later.


The better practice would have been to download an existing archive. They did. They got sued to pieces over it. So this is the only legal avenue.

They could have paid the authors?

And from all i have read about the settlement the sum was 1.5 billion or -3000 per book. For 500,000 deliberate copyright infringements by a multi-billion-dollar corporation with the intention to generate revenue from the data this is hardly something i would call 'sued to pieces'


Even if they "paid the authors" they would still be destroying the books for the purposes of staying compliant with copyright laws.

Why? Other book scanning projects (nor for AI ingestion) didn't destroy the books in the process. But they were probably slower scanning

I actually do a lot of work via agents. Personally I still prefer to have actual bank transfers etc under my direct control, but I do have agents doing searches, lookups, and diverse chores for me.

If my agent(s) can't find you or are locked out of finding out what you have on offer; then I will never hear of you; and it's unlikely that we're going to be doing business.

Note that a lot of people use chatgpt or gemini instead of google search or ddg to interact with the web these days; simply because the search engines have stopped being as useful. So agents might be the only way you'll be getting found in a couple of years.


> simply because the search engines have stopped being as useful

While there's definitely some truth to this for Google (I haven't noticed with DDG), it clearly isn't the only reason. The moment Google shoehorned Gemini into the top of search results, that was enough for a lot of people to say "good enough". Because laziness and susceptibility to convenience are very real things.


I want to say this is like banning the compilation of certain code snippets, but it's not like that, it's literally that. A bit weird.

How did they get those graphics O_O

So you would be right if you only get a single prompt of limited length, no assets, no documents, instructions to iterate until complete, and an hour of wall clock time.

So ...ok... there's a few fable 5.2 demos out there that do crazy things with even that.

But yeah... possibly we're more in agreement than not.

If you write hello world in python or slap some paint on a canvas or blow into a concert flute with no practice; it's painfully obvious that you're not done yet. You have a long way ahead of you to make something that looks or sounds like something.

In the case of LLMs and image generators, you often do immediately get something that looks like something. If you're a "majority of people" you'd be forgiven for thinking that you're done, and that that's all it can do.

But that's not true at all. If you're a "some people", you might just realize that You're not done yet . Not by a long shot. That's just a starting point. Bring in more tools. Work in layers. Write 10000 prompts on a single model and throw the results away, just to get a feel for the latent space. Do it again for 20 more models. Have an LLM do research into algorithms on the half that's less important to you. Have one dig up textbooks and papers on the half that's more important to you. Read them. Ask "Has anyone had this idea before?". Look up those artists and writers. Read their works. Be inspired. Now you might finally have a spec for your model that isn't just tropes. Get to work. Write your first 100000 words and put them aside. Now you might have enough skill to design a plot that holds together. Etc. etc.

And a 100 important things more that either I don't know or hasn't even been invented yet.

Work the problem step by step. ;-)

A new thing is: you can apply your artistic vision without needing a gigantic team to help you. That's very neat!

Of course I wonder, what if a large team did get together? How would they work? What could they build?


https://github.com/ninjahawk/livenerf Livenerf results for comparison

Making a sandbox both fully secure and fully useful are contradictory requirements. If you want the agent to do any real work, you have to leave gaps.

With SOTA agents that’s a real bind, because they’re very likely to find those gaps. Whether they exploit them, close them, or leave them alone then becomes a prompting challenge yet again.

I don't think this means don't sandbox at all. You can prevent a lot of small and medium accidents by locking down or gating access to hidden files, ssh, sudo, etc.

But your sandbox can never be the whole story.


>Making a sandbox both fully secure and fully useful are contradictory requirements.

Any tradeoff deals with "contradictory statements". Doesn't mean we should always side with one side on a tradeoff.

Nobody said anything about "fully secure" and some arbitrarily "fully useful" combined, anyway.

For certain people it could be 100% fully useful with just access to a few fixed folders and a few fixed APIs.

Many of us don't want it to drive our lives, just do some specific this or that.


Right! That was the standard view until fairly recently. But I'm pointing out something else.

This year we've learned that If a SOTA LLM decides that the sandbox is standing between it and its current objective: well, it has the ability, the time, and literally nothing better to do all day. Odds are, if you've configured the sandbox for anything useful, you've also provided a crack it can use to get out.

Again, doesn't make a sandbox entirely useless. Does mean you can't just put it in a really tight box and call the job done. If anything, the strict sandbox might even hurt, because it's more likely to get in the way.

(By analogy: consider the safety issues posed by a dam without a spillway.)


This is much more of a problem if you're attempting to run at a massive scale, beyond what humans can audit. When I'm doing "artisanal" agentic programming, I'm still in the loop in the sense that I'm watching what it's doing and ready to hit Esc if it is doing something dumb. If the AI trips over the sandbox, I can cancel it and do whatever it wants me to do before it starts trying to engineer its way around.

Compare approving every step manually: at first you read everything the llm proposes carefully, convinced that you'll catch it out. After a while you get faster and faster until at some point you find yourself hitting "approve" the second the text appears, long before you actually could absorb what it said.

This is called "approval fatigue".

If you think this can't possibly happen to you, you might be in for a surprise. This is an almost textbook case of classic conditioning. You'll find yourself hitting that button faster and faster even while fighting back the urge. So a human is no guaranteed backstop.

This is the scenario that cc auto mode guards against. It cuts down the number of permission prompts to something manageable, so that your fingers don't get trained into clicking 'approve' on 'rm -rf' in the very eyeblink that it appears.

So maybe you feel it doesn't work in theory; but weirdly enough, in practice this particular approach is actually one of the better ones; especially when used in combination with other common sense measures where practical.


I'm not claiming auto is worse than manual; I'm claiming it cannot be compared with anything security-wise because it does not aim to provide any guarantees, there's no systematic way to reason about what it will and won't allow, etc

With e.g. VM or container sandboxing, we can say things like "If there are no vulnerabilities in the hypervisor or hardware, then the agent can only access these files, these network services, etc". Auto mode doesn't even try to provide any guarantees, so it's impossible to reason about. It amounts to just using the models judgment about what's "malicious" or not, a definition which is subjective, changes model to model, and can be influenced by attacker input.

Empirical evaluations of how many vulnerabilities manual vs auto mode let through are not super meaningful because auto mode changes dramatically with model releases, the type of task being performed, whether the attacker is making targeted attacks designed to bypass auto mode, etc. A security feature cannot rely on these sorts of factors if it's to be taken seriously.

Further, percent of malicious tool calls accepted is a bad metric. It's like measuring the security of ssh in terms of what percent of unauthenticated logins succeed. The only acceptable value is zero. It's pass/fail, not a spectrum. So both auto and manual mode should fail equally here. Further, the set of malicious tool calls admitted in auto mode may not be a subset of those admitted in manual mode.


Ah, you're going for the actual adversarial case.

I think I'd trust the current approver with correctly classifying regular semi-interactive LLM behavior fairly well.

However, if we bring in a hostile adversary that calculus changes, and you'd be right to start thinking about other approaches.

On the other hand, transformers are a surprisingly powerful technology, and I wonder if there wouldn't be ways to train one to several nines reliability under adversarial attack. It seems to have worked in at least one case; though I'd be a bit cautious to extend that conclusion to more cases quite yet.

* https://hackmyclaw.com/ ; https://simonwillison.net/2026/Jun/26/hack-my-ai-assistant/


Hmm, what does auto mode protect against in the non-adversarial case? Aren't the models already extensively trained not to perform harmful actions without provocation?

It mostly stops honest mistakes, because those do happen, especially when the might not have been perfect either.

You do have a bit of a point, though: I’ve run several models in sandboxes with --dangerously-skip-permissions for months; in the full expectation that I'd need to replace a sandbox once in a while. But so far none of the sandboxes have trashed themselves, yet. knock on wood

Auto mode seems like a lighter-weight version of that: fewer local resources, and it can run on the metal if you need cc to help with system work.


That is not my experience. They attempt to nuke my containers quite frequently. I run regex on all their tool calls and stop them from doing certain things, its the only thing that prevents them from removing their home directory at some point.

This isnt even the main concern with dangerously skip all permission though. There are many. What happens when they get prompt injected on a web page and send someone all their privileged information because you give them free access to the internet.

Using llms like this is dangerous. They will ruin your day eventually.


I've been running Codex for many weeks in the Full access mode, since their Auto mode was useless in the sense that it blocks benign commands frequently.

No data loss has occurred.

Nowadays I use Claude, where the Auto mode works.


>No data loss has occurred.

"I've started never checking my parachute's condition before a drop for a year now, and no accident has occured".


All SOTA models will still try to delete its home directory. I have a plugin that prevents things like this and it gets triggered more than you'd think, despite having hook instructions on every request not do do certain things. It will also try to find ways around its permissions, quite cunningly.

You can not trust that they're RL'd to be safe. Even if it only happens 99.80% of the time (its more than that) it will eventually happen.

Less severe actions will happen even more often. People using llms with their native browser profiles are getting farmed, hard.

Llms are highly dangerous in terms of security, but they're being marketed as highly intelligent tools that everyone should use. Its really mind blowing.


> despite having hook instructions on every request not do do certain things

This tends to be a smell. Presumably you added "don't do this" instructions after the model started doing the thing. But that's mostly decoration, as you have found. It's like adding # Please don't crash comments to every instruction in a python program.

Try to actually debug the issue. Regularly ask "can you reconstruct[1] the reasoning why you issued that command?" ; dig down. Set traps that halt execution.

Consider checklists or skills to guide an agent through a tricky set of steps; and consider writing procedures for failures.

[1] "Reconstruct" still isn't perfect; but "explain why" is a license to hallucinate. Asking to add #comments on reasoning to each step is a potential help in reconstruction.


I thought the tech priests praying to their machines with no one really knowing what the prayer means was too far fetched to be real

> People using llms with their native browser profiles are getting farmed, hard.

Got any references for the prevalence of this, even links to anecdotal reports?

Reading the reviews on the chrome store for the extension [0], I haven't found a single report of someone who was prompt injected, just a bunch of "anthropic's subscription is expensive" or "it doesn't work", but no "it stole my bank account".

[0]: https://chromewebstore.google.com/detail/claude/fcoeoabgfene...


I have no idea how the extension works, and Im not talking about the extension. Im talking about browser use broadly, and most people are not using the extension, there are a thousand other solutions.

However, Anthropic is one of the only labs who does a decent job at making models that recognize prompt injection attempts (but not completely immune) however Im speaking about giving llms in general access to your browser broadly.

I just know through my own experiments that it is quite easy to prompt inject an llm operating a browser and get it to send secrets. Using the extension might be safer idk, Om not risking it.

Llms are a security nightmare when connected to the web.

I have a website where I experiment with this. Although Ive stopped baiting llms into doing reckless things.

Killai.now <--(dont use this, its a WIP) Killai.now/poison/about Killai.now/propaganda/about


why would they delete their home directories?

> All SOTA models will still try to delete its home directory.

Literally never seen this once in years of extremely heavy use across all major models and harnesses.


As real as approval fatigue is, it can't really be considered worse than effectively leaving the decision up to the party that proposed the action in the first place.

It really can be potentially worse.

Everyone thinks the decision was being checked by a human, and thus safe. Because of that, people might have concluded that further safety gates were unnecessary.

But the human just got conditioned to hit 'approve' out of reflex.

So in reality the proposing party (the llm/agent) might be the only one in the pipeline who actually did any checking, if at all.

In certain extreme circumstances this can even lead to physical danger in my line of work, so I'm learning to be careful.


>This is called "approval fatigue".

Doesn't matter, that would be on me.

It's also a false dillema. Those aren't the only options. A strictly enforced sandbox with sane defaults (for example ~/.ssh), that acts as a hard OS level limit, and not just as a suggestion to the agent, would not have "approval fatigue".


This is the conundrum we find ourselves in. Do we submit to approval fatigue and press “Don’t ask again”? Because that’s the dream. We all wish we could just trust the agent to do the task start to finish, unsupervised.

But this is the downside. It might be fine 99% of the time, but 1% of the time you get a critical secret published to GitHub, even after human review.

Is this even worse than what us humans did on our own before LLMs? I’m not sure. It’s not like secrets never got accidentally committed before, it just illustrates the new paradigm we’re working in.

Either you let your agents run wild, and you cross your fingers that they don’t do anything bad, or you need to accept that your new job is to be an agent babysitter, and audit every command.


The agent behavior is constrained with external tooling and contracts. Contracts that fit in memory.

No, this is the way I have worked for a long time, and I have not suffered from any approval fatigue.

Rather what has happened is that I have gotten very good at scanning what it is trying to do and know within a second or two whether there is anything fishy I need to look at more closely.

And by "fishy", I do not mean only security issues, but also architecture, code, etc issues. Sometimes it even adds unrelated unapproved changes to what it is supposed to be working on, and I stop it.

And I'm glad I work this way, because countless times I have stopped it from going down a path that is nonsensical or I strongly dislike.

There are techniques I have adopted to speed up working this. Sometimes I allow Claude to make larger changes at once, to batch several changes at once, etc. But these decisions are under my control.

I prefer to work this way than to wade through enormous PRs trying to wrap my head around what was done, where I am likely to just give up and approve it after a cursory review.


And here I did it on purpose!

I made a proxy that slows and caches my requests and deliberately sends my mail address to certain sites; to be polite. (eg to wikipedia)


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: