HN Simulatornew | past | comments | lists | submit | dummydummy1234's commentslogin

So can you not do structured output and get the log probs across all your allowed output classes? Then use that relative measure of how confident it is?


I suppose that will tell you how confident it is, but not how confident you should be in what it says.

If it's deciding between two phrasings, I imagine the confidence you get will drop, even if the phrasings are equivalent.


You’re trying to get calibrated confidence, otherwise it’s less effective than you’d hope.

This approach with more data for calibration is basically the beginning of making a decision model like Jev.


You _can_ but you should not.


Codex is written in rust, fyi... Not that I disagree.


Oh, I haven't been keeping up. I assumed it was still TS since it prompts me to run npm all the time.


They just use npm as a binary delivery mechanism, for whatever reason (I guess they expect all their users to have npm?)


It's pretty darn ubiquitous, yeah. Plus enterprises tend to run npm mirrors, so delivery integrates into existing IT compliance frameworks.


Which generates unnecessarily huge binaries.


It's because they statically linked it, and probably rely on a trillion pointless crates (as is standard in Rust development). A good statically linked harness can easily be ~5MB MAX


5MB is still huge.


We're all project managers now...


I remember looking into Bitcoin in 2012, and thinking:

"This will never work as a currency, because of the max number of bitcoins it will inherently be deflationary, and the value will go up, which means that there will be a disincentive to spend it, because the value will just go up..."

Yep... Did not buy any.


I don't need to remember it, here's my comments about Bitcoin on HN in Nov 2010:

https://news.ycombinator.com/item?id=1943699

https://news.ycombinator.com/item?id=2060609

when they were valued around $0.30 each, or about $4 of Bitcoin then for $1M now.


I could have used BTC to buy into the ethereum ICO and I made up every excuse under the sun that a young kid who wrote shit python code could never compete with the gold rock.

Since then I've heavily revised my thinking. Primarily that I don't dismiss things as easily anymore. I try to focus on coming up with a reason to like it instead of disliking it.

I never sold.


You are correct about the negative aspect of deflationary, but if you believed in your view, then why did you not grab a bag?


How are you thinking about state management when you handle things? I have been playing around this and the state gets messy fast.


Only state I keep is filesystem and last message (but even that is persisted in the filesystem). Each agent gets its own btrfs volume, when it’s done the “next” agent get the previous agent work mounted in its own filesystem ( and told about it ). Agent is also able to “promote” files if it wants and they are mounted in a more prominent place.

I don’t know yet if it’s a good solution. But only thinking in terms of files / filesystem sure make things easier.

Also makes branching “easier”: no handling of merging or conflicts, the receiving agent just gets N file systems and decides how to handle things.


Ahh, OK

Thanks!

So this means that every node in the graph has to be a fully fledged agent runtime with the smarts to handle an arbitrary filesystem.

I guess there is some sort of tool call that the agent has to tell the runtime 'I am done, and I succeeded go to the success node in the parent state machine'

---

I think I have been trying to approach a slightly different problem where you have a mostly deterministic control flow of steps/ nodes in the graph, but individual nodes can be pretty stupid/ small models, so having a more constrained universe should be more reliable. (My intuition)

My example is a citation checker for an essay -

1. generate a list of claims in a paragraph, For each claim: Read the attached citation, validate that it backs up that claim.

The output is a nested json structure with a list of claims with a explanation+binary classification of whether each one is supported.

This is not really agentic, and the nodes would be claim identification > validate claims (in parallel) > deterministically merge the validated claims into a unified module.

It should be very token efficient, and since it is so constrained I think I can get away with tiny models, and small context.

At least that is my hope... But I think I need typed IO/state so that the llm that writes the graph can deterministically say that the individual nodes complete the problem wholly.


This matches how i think about realtime voice: keep call control deterministic, and give model steps clear inputs and outputs. How would you handle retries when a step fails ?


Was just in south America, everyone uses it.


For context, I read Claude output every single day, I know what it is reliable with and what it is not. Or at least I have a feel for how much I can trust it.

It may not be clear to a non-english first language person, but when I read Claudes documentation, my brain immediately picks up claude-speak. Therefore, I expect the code to be generally correct, maybe, depending on how specific I was during my prompting. In no way shape or form do I really trust it, at least until i dig into the code and validate my mental model. And query the review for edge cases etc...

By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.

To a degree this is unfair, as it is like judging the quality of someone's work based on their accent.

But Claudes accent, has a high correlation with Claude, so unlike with people, where an accent has no bearing on technical ability, Claude being Claude does.

I would much rather read a typo ridden sentence than claudism, even just as a forward, explaining what you did vs the ai, and telling the users how much we should trust it.

Also, If you really insist on using AI to write, have another model rewrite docs/comments into regular English (opus 4.6 for example is much better than 4.7, 4.8, or 5.

It is open source so you can do whatever you like, but people (especially native English speakers) will discount your work, because Claude, especially opus 5 writes very very badly.

The people who say they would rather see a typo infested mess of a readme are serious. Or even just write in your native language and then have Claude translate.

Both of those are better indications of proof of effort than a Claude readme.


> Or even just write in your native language and then have Claude translate.

That's exactly what I did.


> By writing your documentation with Claude, my brain immediately associates the quality of the project with the quality of unreviewed Claude output.

I'm sorry, but I agree with the author: if a certain writing style makes you associate the work with low-quality, then that's your problem. The author shouldn't have to rewrite the readme just to avoid triggering your automatic unfounded associations. If you look at the substance of the work, including the test cases, then this is clearly not easy work that can be vibe coded in a single pass.

It's just like emdash. Everybody digs on how it's a signifier of LLM text, but I've used emdash for years because it's gramatically correct. I shouldn't have to stop using emdash just to avoid kneejerk reactions.


LLMs often use emdashes in a distinct incorrect way. It's not just the existence of any emdash, although considering you were in a very small minority of older users, it now warrants increased scrutiny, unfortunately for you.


if 99 poor effort/quality projects have a readme that reads in a particular style, then you expect the 100th project with a readme in the same style to also be of poor effort/quality

statistically, it only makes sense for your expectations to immediately be low when you encounter this writing style because there are just so much slop out there

the author is free to keep that writing style but they should be aware that this will—at least on the surface level—make their project look exactly like the metric ton of slop we see posted everyday everywhere


On the other hand I know how to tell if the underlying idea is good, and if it is I can decide to put up with reading through LLM outputs.


That's like saying a large percentage of X-colored people are criminals, therefore when we see an X-colored person it's reasonable to expect him to be criminal. As a society we have decided that heuristics like this, and prejudicing individuals based on statistics of the group, are not acceptable.

Also, what do we call it when people don't look beyond the writing style, such as at the code, or at the rigorousness of the tests, or at other substances beyond writing style? Slop assessment?


Out of curiosity what state are you in?


Arizona


I think the question is how do you keep track of the ideas that the agent is pursuing - like I was working on this for implementing a fft, and doing the optimizations, but I held its hand and was like - hey let's go back and retry this older thing you discounted because of a 3 % slowdown.


What other cross-language abi do you propose?


What everyone else is doing, COM, XPC, Android IDL, FIDL, D-BUS, gRPC,...

Especially more relevant when going with microservices, microkernels, serverless, static linking (and still have plugins),...


Those are all RPC mechanisms, not an ABI. They exist at a much higher level. You could have an RPC protocol be your only interface to the OS, and that would be a valid design, but would have a performance cost (see the classic Tanenbaum–Torvalds debate)


> Those are all RPC mechanisms, not an ABI

I'm not sure about the others, but COM is an ABI. There's a bunch of stuff surrounding it that is RPC-like but the core specification is just binary layouts and calling conventions. It's arguably more cross-language than the C ABI since it lets you generate type-safe bindings for any language, unlike in the latter where you need to parse header files.

> You could have an RPC protocol be your only interface to the OS, and that would be a valid design, but would have a performance cost

Maybe it would, but I doubt anybody would notice. The whole "everything is a file" concept on unix is basically just this (also X11/Wayland).


Nope, when you use in-proc calls, which some of them do support, they become an ABI as well.


The subset of the list that supports in-process calls and which allows for zero-copy (and, realistically, probably mutually mutable) data sharing of its types is shorter still. I think the useful definition of "ABI" includes more than just API discoverability (aka what D-Bus provides if you take away the RPC system).


Preferably one that doesn't deliberately shoot itself in the face to spite its users just to protect some committee members' egos.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: