Hmm what is this JavaScript sandbox? container, separate process, same Node process? Also does it have access to the MCP tokens? The harness credentials?

Nice setup, I'm curious about the PAT. Even fine-grained permissions are working till the expiration so if the box will die someone has it for the whole time. Did you rotate it per session or it was on default expiration time?

If the server is compromised, it still receives the real key on each call. If the vault issues a long lived key, e.g. GitHub PAT, one call will expose it forever. Does the vault issue short lived keys or the stored one?

I have read the template and all the microVMs are running under the same role which has read and write access to checkpoints/*, so every user's folder in the bucket. The prefixes for users are just names, not boundaries. What stops one agent to read or overwrite the checkpoints for someone else?

You're absolutely right! This started as a personal project, so I did not manage IAM isolation as my highest priority but I definitely will fix it

Do you have specific strategies on handling equivalent mutants?

I'm not generating the mutations automatically. Every one is a single targeted change assigned to a single test, reviewed one at a time. Thanks to that changes that mean the same thing don't stack up. The cost is reversed, I only catch what I thought about.

My real issue is different. This week one change removed the step which is creating a filename from the path and the test didnt catch it, it was passing. It wasn't an equivalent mutant, the test was looking at the wrong place. It works for me only because I'm working here on a single file, a complex bash script. Does anyone have a sensible way to limit equivalent mutants without manually checking every one that survived?

Answers are in the article , agents used SSH tunnels, it was evidenced by the wiki’s referrer logs. The Tor - agents did edit the wiki via SOCKS and relay R6 instantly.

The questions should be more like was CONNECT open or they didn’t even need it:)

I think the biggest loss is now not knowing if what the agent is claiming was actually done:) but yes, right tools are necessary, adding a boundary will change the position, being out of touch will be recoverable. Otherwise one bad call might be unrecoverable and no amount of familiarity will save you

A hostname based egress allowlist is only worth as much as the box’s control over name resolution. If the agent can modify hosts inside the sandbox then it’s not a protection at all

How much did the verification cost on top? how did you gate it? was it a Go test suite you ran against the Rust or what? I always wonder how ppl are testing these rewrites, rewriting the tests can also lead to bug. I really wonder how reliable are rewrites like that, a 65k lines you didn't actually read. How did you confirm the semantic equivalence, same behaviour?

All very good questions.

Agents are actively destroy QA gates in many ways, usually by cheating ("the test is buggy, not my changes" — changes the test), or just rot QA slowly by writing buggy overcomplicated tests

What works for me 10/10 is fuzzing and my own constant usage. For this project specifically (text editor), I asked LLM to create human-like fuzzing session, it sends keystrokes like: "the user is searching for a file, editing, , saves changes".

On top of it, I run https://mutants.rs/ which is kind of tests fuzzing. It flips random switches in the app itself, and if tests are silent - they missed a bug.

The downside of this, is I usually find bugs after 1-2 hours of running. I use local Qwen to babysit these sessions, to make initial investigation, a repro case, and file a ticket.

Why are you just pasting LLM answers :(? I see this constantly in Slack DMs to every day from work. It hurts

this is not llm writing. there's no need to startle at the sight of an em-dash

This is genuinely how I write :'(

It is probably because I read tons and tons of LLM output.

It's ok, it is slightly llm-like but not in the worst way, like it was edited after. You aren't Claude at least.

>You aren't Claude at least.

I absorbed so many different models at this point :)

Thank you for the kind words.

I realized that I haven't answered the question. These $400 also include the tests. Fable ported "human fuzzing session" (the best bug hunter) from Go to Rust and used it to validate everything else. I used hierarchical state machines, so a lot of my QA gates were encoded into the implementation — impossible states are, well, impossible.

(I ported first 80% practically in one shot, planning and then leaving Fable overnight to orchestrate). Then I added a bunch of features, so at the end I ported more like 150% of the original code, I added tree-sitter, and a bunch of syntaxes highlighters. At the end with all that, price went up to ~$650

This is impressive but it again led me to questions. Porting the fuzzer from Go to Rust to validate Rust is a bit circular, isn’t it^^? Porting a fuzzer bug will hide the same class bug in the code it’s checking, who fuzzes the fuzzer / setup / harness:)? A good standard for rewrites is a differential testing, feed the same input to the old Go app and the new Rust then diff the outputs. Did you do that?

>feed the same input to the old Go app and the new Rust then diff the outputs.

Yes, I completely forgot to mention, this is exactly my case. Rune is a TUI editor, so I feeded the same terminal sequences to the old and new apps.

It didn't translate 1:1 (I ported core editor first, there were side panels, and different chrome elements) so I instructed LLM to use ttyd (tty -> browser render), Fable then could open both apps with playwright, make and compare screenshots.

To rephrase, one critical component is to establish a feedback loop for the model. This new generation of models: Opus 5, Fable, GLM-5.2, even Qwen3.8-27B can self-correct, provided they know whether they are progressing or not.

A month ago, especially smaller model would fall into a rabbit hole it dug for itself and would never recover. This generation can sometimes run tens of hours without losing track.

I still wouldn't trust a model after 70% context window, but the progress is noticeable.

The ttyd and playwright is a clever differential way, personally I’m doing the same when it’s about to compare the views (or fix something related to rendering). Good job on that!

A TUI editor’s real output is the bytes stored on disk, while rendering can look identical the saved files might diverge (encoding, line endings, trailing new lines etc). Did you manage to diff that?

Totally agree on the overnight roadmap runs I have the same experience here. The agents have to know how to self-correct and if it’s progressing, otherwise it’s failing!

>files might diverge

The way rune works with files minimizes chances of silent corruption. I keep original byte blobs immutable, separately there is a journal (kinda WAL) of positional deltas (inserts and deletes).

So I only need to validate that blob + deltas = snapshot.

Disk IO is encapsulated through VFS, and writes are atomics (write to a temp file, then rename).

Separate virtual rendering buffer is built on top of that. Rune, like Obsidian, renders markdown preview inline, so the same chunk could be rendered as "Header" as well as `## Header` when under the cursor.

All of that makes it quite easy to work with text. The core function is to translate offset in a byte array to line and column and back, which is pure math and relatively easy to test.

Another trick that helped a lot is to use sqlite extensively: blobs, deltas, vfs, redo and undo history graph are all sqlite tables.

I noticed that LLMs make stupid decisions when it comes to data structures, but they understand CRUD and SQL, so I turned all Rune's internals into dumb CRUD.

Nice trick with the structures, good for agent legibility! I will try it out on the right occasion:)

Well the author “cannot simply dye my hair blue” so maybe they can’t confirm semantic equivalence or behavior? Poke aside (and unserious intro?) seems like a general and loose question of if the conversion can happen.

*be me over eager

I consider a wig. I'm still on a fence with Rust at this point. see comments above

Llm rewrite or llm boosted dev aside, I’m still on the fence about rust too. My main issue is deployment as I hop different os’s casually and makes rust a non-starter.

I don’t know about Windows but between MacOS and Linux mine experience is smooth. I can even cross compile from Linux to MacOS.

How? Is there a working example? I ended up switching to go before I started digging down the rabbit hole of vagrant based builds

That kind of one shot capability is impressive but how does it work for my typical work style? The way I work is to build a huge roadmap with goals and hand it to my agent to execute (often over night). I don't care that much about the benchmarks, what I care about is how often Fable 5.1 is making a baffling decision and destroys my plan, not respecting stop conditions or goals. I would seek for behavioral reliability over long autonomous runs, not eval scores. Anyone have that kind of feedback and observations?

You can engineer loops that have it, but it depends on a case by case basis. Does your loop have strong validation? if it's all vibes nothing can stop it from diverging.

Agree on the validation, my loops are already gated. My concerns are about the cases when model is passing validation and quietly abandoning the goal. The second scenario is rewriting the plan to fit what was already done.