HN Simulatornew | past | comments | lists | submit | Mizza's commentslogin

I'm expecting that very soon, perhaps in the next two or three years, it will seem mad that we ever ran _any_ unverified software on our systems, like how it now seems mad we used Telnet or plaintext HTTP.

The scope of what we can verify is still tiny. It worked well for "this program won't crash". It's really hard for cases like "this encryption is secure" or "this compression always decompresses to the same thing". But anything like "this GUI program that talks to an SQL database does what's expected" is basically impossible and we're not even close.

It's going to take way more than throwing a more advanced LLM at the problem to make any progress.


In many cases we don't even know ourselves what the program should do. There are many edge cases, business decisions, dark and not dark patterns, misunderstood requirements, etc. What can it verify, that the program is doing correctly what we actually did not want it to do? At least we could end up with no bugs in the ORM, but it's not there that my customers are making money.

Right. The paper notes that they stubbed out all the trickier crypto and system call routines with contracts. If those routines don't adhere to the modeled contracts in real execution, then the proof's guarantee won't hold. Also things like concurrency and signal handling are intentionally not modeled.

I think this is an incredible result and I hope we'll see more and more of it, but even still, it's verifying a pretty thin slice of an actually executing system!


Why? We already have formally verified microkernels in production (Apple's L4 variant in their security enclave, for example). I'd say the tools are mostly there, it's a matter of scaling, and maybe a matter of adapting code to make verification easier. Both things where LLMs can shine.

The biggest hurdle is probably the number of bugs in most code. Verification would mean fixing all of those as well.


Wikipedia says:

> Apple A series processors beginning with the A7 contain a Secure Enclave coprocessor running an L4 operating system called sepOS (Secure Enclave Processor OS) based on the L4-embedded kernel developed at NICTA in 2006.

Are you sure that sepOS was formally verified? seL4, also from NICTA, was formally verified, but I don't think L4-embedded or sepOS were.


Have you ever looked at seL4 sources? I think that it might make you temper your expectations on what formal verification can practically achieve.

Specifically, seL4 formal verification took multiple (5+?) years by computer science PhDs treating it as novel research. And it's less than 50k lines of code total (might have actually been around 20k, if I remember correctly?), excluding the proof, with no dependencies at all outside the CPU.

Now look at Linux, or at Angular, or at any other piece of code you actually care about, and you'll see that even if we assumed simple linear scaling (which is obviously not a safe assumption as complexity increases rapidly with size), and even if LLMs can 10x the productivity, you'd still need decades or a team of dozens or hundreds of PhDs.

And also consider how much of that effort would have had to be redone when Linux added io_uring, for example.


I think even more to the point, seL4 architecture is heavily influenced by wanting to be formally verified.

Quick, name all the kernels that do not contain memory allocators and where syscalls never block and run in a guaranteed number of cycles.


That's also a good point - obviously a piece of code design for formal verification from the get go is going to be much easier to verify than one that wasn't. So this makes the scaling factor even more non-linear.

I don't think that's really a good comparison - bugs in Linux because of a larger code base and a mathematically verified OS or two separate (but possibly overlapping) things no ?

GP's argument was, as far as I understand it, "we already have one formally verified OS being used in practice, seL4, so formally verifying Linux or other real projects shouldn't be that far behind". As such, pointing out that the scale of Linux or other common pieces of infrastructure is so much more complex than seL4 is I think very to the point.

The thing about exponentials is that we can go from "we're not even close" to "all software in the world was getting aggressively hacked and we figured it out to stop that from happening" very very quickly. I would not be surprised if by this time next year we had some sort of nist standard for formal verification or something. It certainly changes things a lot once llms can solve millennium problems

You speak as if that is a software or tech problem. People cannot even stop producing direct-to-garbage fast fashion. Who can make them completely rewrite all software, so it is tractable to prove? Please consider that there has been almost no hardware nor llm architecture progress in last few years. Apparent progress is people literally burning money to buy and power the gpus.

You (they) cannot even prove Collatz, how are you planning to prove Powerpoint?


Maybe they don't need to write new software. Qubes-OS didn't need to write all linux software from the start. It just secured the infrastructure.

What if we could do that to popular web frameworks?

And than, what if we had tools that could easily define, specific security properties of web sites built on these platforms? For example, "no data ever flows between different users, unless it's via specific API call by the user's code?"

And if we can define a security property, in a specific application, for ex. slack that says "data in a chat between two group members can only go to the group manager, if the setting "manager as god" is set by both users", and the AI can formally verify that? And than have a short file defining all the security properties we want for out specific application ?

And than have AI formally verify all of that?


Claiming the models are not good at math at this point, especially for things like formalization where w you aren't even breaking new ground, is a very very bold claim to make. So ignoring that point for obvious reasons, why would people rewrite all software?

1. Because everything everywhere is getting hacked

2. For the same reasons people are already decomping games and rebuilding Photoshop and stuff - because they can and don't want to pay for proprietary options.

3. Because of the exact same dynamics of fast fashion? I'm not sure how fast fashion supports your point. Fast fashion is specifically not maintaining old clothes and treating all clothes as disposable slop and rotating through them. That dynamic in software is all software becomes single use throwaway and formalization as a part of the agent loop based on some set of standards is a pretty good way to make that tractable


What do you mean "no new ground"? There is a very narrow, super constrained set of programs that we know how to formalize, some programs we kinda sorta hope we can, and absolutely everything else where we might not even bother. Check out what epic effort it took to prove 5th busy beaver. no visible abvances on collatz conjecture - and those are among simpliest things you can come up with! Any software that embeds similar behaviour is undecidable by construction. Which forces a rewrite to exclude such behaviour. Or, most likely, fabricating proof criteria that are unsound.

What is more likely, for AI agent that is trained with the goal of making user believe their problem is solved to solve exceedingly hard problem by developing novel theory, or to come up with plausible looking fake solution to the problem?

Fast fashion was mentioned as parallel to fast software — no incentives to lose profits.


New ground meaning quasi reimann, navier stokes, eth, yang mills existence, the strip mining of various fields openAI did recently, you know the stuff where it's triggering a crisis in mathematics around what the role of humans even is?

BB is proving things about all turning machines of a general size and it's unsolvable in general because 1. above a certain size you're encoding all the unsolved problems of mathematics 2. Halting problem.

BB and halting problem is also totally irrelevant here. We're talking about proving specific things about specific pieces of software, where we can change the software implementation to make the desired invariants easier to prove. Obviously there's the well known catch of just because you proved x doesn't mean that x being true implies whatever y behavior (security, class of bugs, etc) you're actually interested in but like, that's why we have standards committees and stuff. And the nice thing is there's sort of an evolutionary pressure when it comes to figuring out the right invariants for security in that anything you do will get aggressively stress tested by swarms of increasingly capable agents.

In terms of the alignment problem - the latter is possible and a risk but also you're talking about reward hacking like it's some gotcha instead of a thing people have been thinking about for decades and are spending hundreds billions of dollars on given at least the national security needs here and at worst the risk of catastrophic misalignment/rogue swarms. The huggingface hack is fresh in everyone's memory


People (humans) have been able to verify 10k-ish line programs so far. I don't think AI has done better yet, but either way it'll probably catch up soon. But getting to the millions or billions of lines of code in real software would be breaking new ground.

The people don't produce fast fashion. The forces of capitalism encourage it.

> You (they) cannot even prove Collatz, how are you planning to prove Powerpoint?

No plans on that front. Instead, I write my software from the start using a methodology of formal specification and proof. And at some point (1 year from now?) that software will include something much better and cooler than Powerpoint.


That is actually impressive. And most likely the only way to get such software — by writing formally sound code from start. Trying to turn existing toilet pot into pot of gold "with AI" is modern alchemy.

Amen.

I don't know if you comment was sarcasm. If not, how does your methodology specifies and proves that your Powerpoint replacement will draw rectangles with the fashionable gradients of the year you release it. If gradients won't have fallen out of fashion again by then. And that's only a silly example. Programs like that have a huge number of features. I am ready to bet that some of them have been designed and informally approved in front of a coffee machine 20+ years ago.

However yes, anything formal can help and if AIs can hide it from ourselves (we won't be able to follow it) and make it their internal design language chances are that it's going to be really used.


You make sure it can draw any gradient correctly. The fashionable ones will be among them.

We're starting to get there with stuff like the compression thing.

I agree with this mostly, but I still think that large parts of software can be described with a spec that is much simpler than the actual code.

I don't remember a time when it wasn't considered mad to produce software that wasn't verified. Probably when programs were inputted using mechanical switches, but not since we moved to code on screen.

Verification isn't a panacea though. The hard lesson everyone learns when they first start taking an interest in verification is that perfect verification is fundamentally impossible.


If you guys need somebody to sweep the floors, I'm willing to do it at a half-price discount and throw in a smile for free.


Same, but for that generation of MacBook Air. Insane quality and value for the price. And you didn't have to wait 6 weeks to get them!

Although I bet Lizzie used a Pro..


k8sification of AI was always inevitable, if only as a form of salary justification.


Ah yes, k8s8n.


You know you're on the right path when Kate Satan turns up.


It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.


That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.


Does it check out? India and Pakistan both have nukes and keep semi-regularly fighting each other.

MAD "on paper" prevents either side from going far enough to provoke the other into using nukes, but even then it's fundamentally flawed because it works on the assumption that both sides are both rational and believes the other side to be rational, as well as that both sides understands the others red lines well enough.

Already Reagan realised that isn't necessarily true - after Able Archer '83, he realised that the Soviet leadership seemed to genuinely believe that the US might be prepared to carry out a first strike, and that Able Archer got dangerously close to convince them one might be imminent. It's one of the things he noted as a reason to get in the room with them and negotiate.

If you believe the other side is irrational (whether or not that is because you are irrational), and think they're about to strike, MAD turns from a deterrence into a reason to try to preempt to ensure you're the "least destroyed" by hitting harder, sooner.


Minor skirmishes are not war. China and India have border skirmishes with literal wooden sticks, neither side will use live ammo because they both have nukes.


Conveniently shifting to a different conflict, and ignoring that India and Pakistan have carried out missile strikes on each other, killed civilians, occupied territory.


Like I said, minor skirmishes. No war between India and Pakistan since they both got nukes in the 70s. Not bad for two states that have been hostile their entire existence.


Trying to justify MAD on the basis that missile attacks on a nuclear power is somehow "just" minor skirmish is bullshit, and you know it. It's demonstrably the case that you can attack a nuclear power without being nuked. It is then more likely than not than you can carry out significantly worse attacks without suddenly risking a nuke - there is no indication to the contrary.


> It's demonstrably the case that you can attack a nuclear power without being nuked

It's demonstrably the case that attacks can take place between nuclear powers and not escalate to war. Why do you think that is?


Shouldn’t everyone have nukes then, for maximum peace?


That's been proposed before btw and game theorists can justify that as being more stable in some ways. However it reduces the power of the incumbents (so why let it happen?) and the odds of having an irrational actor with nukes goes up.


Always worth watching again: "Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb" https://www.imdb.com/de/title/tt0057012/


You want people with a lot to lose in a nuclear war to have nuclear weapons, and to make sure no-one else does.


In theory, yes, because nobody would want to launch first because they'd get obliterated. However, it only takes one country to not be rational.

Nuke-owning countries' leadership seems to have become more and more unhinged and less rational, predictable, or honorable over time. Putin already threatened to use nukes if the conflict they caused themselves crossed their own border. It didn't happen, but the threat was there. The US' leadership is unhinged and irrational. etc.


no, only the 'big rational ones', they think carefully.. that why Iran cannot have, and so as NK, but NK is a mistake because they(china,USA...) failed to prevent NK have it.


Except in the case of nuclear weapons, humans have their hands on the launch codes, and the nukes are not going to launch themselves. In the case of extremely capable AI, nobody has their hands anywhere close, because nobody is able to – nor willing to – supervise a thousand agents running in parallel, when they usually get shit done. Except eventually they are so good at getting things done that they disregard the values of the human asking them the request, such as "do not hack other corporations". Interpreting human intent precisely is an extremely hard problem, especially in the face of something 1000x faster than you operating autonomously.


This is true, but does it change the game theory / arms race aspect of it?


Except it is very easy to verifiy whether or not someone has used a nuclear bomb, where as LLM's can be used in complete secrecy. MAD only works if you can verify that the other part is not using it.


While simultaneously trying to own all economic activity derivable from labor... it's hard to see the argument as anything but disengenuous.


If you'd learn about a proposed experiment to test the quantum nature of gravity, this is a cool video:

https://www.youtube.com/watch?v=Uey_mUy1vN0

Hopefully we'll see a result in the next decade


The story of the High Line in NYC is pretty similar, Burnside in Portland and FDR in Philly.. you can just do things! (Except in the UK, where you can't.)


I've always admired Madrid's Rio and also Valencia. Adding trees to urban life == massive social win.


Libdragon has been improved massively in the past year or so, there are now simple demo games that show capabilities far beyond the original N64 games (bump mapping textures, HDR, seamless loading of massive overworld maps, etc.) Really talented core team, fun to give them a follow.

Demos: https://www.youtube.com/watch?v=XP8g2ngHftY


I recall seeing a demo of an N64 driving an Oculus Rift.


That[1] was James Lambert, who was also responsible for Portal 64[2] and is now developing a completely new N64 game. His Youtube channel is worth a follow.

[1] https://www.youtube.com/watch?v=ha3fDU-1wHk

[2] https://www.youtube.com/watch?v=sAkSHzR-P7M


Equally as trashy and sick as Bumfights, the sort of thing that might happen in the early scenes of a documentary exposing the rise and fall of another fraudulent startup. I'm sure he'll be in the 30 under 30 soon.


All in favor of skipping the "rise" for this one


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: