> to fly manned missions, starship either needs to be redesigned to accommodate an escape system, or it needs to show a level of reliability and safety never previously achieved by a rocket.
I assume they'll do the latter. If the vehicle is truly rapidly reusable, that's less crazy than it sounds. Just fly it 10,000 times and fix all the problems you find :) Escape systems are only really useful during launch anyway. The only way to protect the crew from a failure during reentry is reliability.
People are relatively small and precious. Why not keep Falcon 9 / Dragon for human-rated space flight, and rendezvous with the rest of the system in orbit?
Because it would take take mission support people away from the starship program, and those people are a limited resource. Also, being the "low volume secondary project" tends to negatively impact reliability too, in many ways. So its not even certain, that there is a crew safety benefit here, compared to a hypothetical Starship program safely flying hundreds of missions a year.
Besides the fact that they're going to do a lot of those Falcon missions with the Starship, I think they actually are pulling the rug up behind them. They don't want to give a lot of launch capability to their future competitors.
that's not feasible for a mars trip or even a trip to the moon and back with real cargo. Even with starship you need many refueling flights for it to be possible. Not to mention the cost.
You could use Dragon to lift humans to LEO and transfer to Starship, and vice versa for the trip back down. They're already adding docking capability for Artemis. It'd cost more but it could be a useful transitional phase.
An escape system could also be useful for a failure during landing. If I were flying on Starship, that’s what would worry me the most. Incredibly slim margins and no room for delays. If the engines look wonky on launch, you can stop and try again later. On landing, better hope the issue isn’t a big deal because you’re making a crater if it is.
This seems to be technically possible for uranium enrichment, but I'm not sure it's technically possible for ML research. You can do it on any CPU just really slowly, right? Or you could hide small numbers of GPUs in any basement. I think AI-2040 has an estimate of how much time we could possibly buy ourselves this way (even ignoring all the political problems), and I don't think it's especially long.
> It seems like the AI can really write safe types and fast, efficient code in rust that beats anything else.
This is also true for humans! Rust/C/C++ are all about equally fast, but Rust is more productive for most people. The downside for humans is that Rust's learning curve is pretty steep, but that doesn't bother the LLMs.
Both C and C++ have a much steeper learning curve than many realize, just because the compilers let you get away with doing things that are fundamentally invalid (Undefined Behavior). Rust isn't actually much stricter, it just catches these problems at compile time.
Anyone capable of writing actually correct, standard-conforming C and C++ is mostly able to first-try the equivalent Rust code in almost all cases, because they are already thinking the same way that Rust requires you to think.
However, a significant portion of people who write both C and C++ don't realize that their code is full of UB - or worse, they don't care.
From a language perspective, the only new restriction in Rust compared to C++ is that mutable borrows (references) cannot overlap. Good C++ programmers are already avoiding this at all cost, because it's a mess to keep track of. When you actually do need it, Rust gives you the tools to build a safe abstraction.
EDIT: And to add, Rust is actually less restrictive in a few areas compared to C++. For example, Rust allows bitcasting between unrelated but compatible types (without going through char pointers).
To be fair, as far as I know even bleeding edge LLMs have problems with the borrow checker and resort to similar workarounds as newbie human Rust coders.
AI needs to be instructed to do a thing. To use the "steam engine designing another steam engine" analogy, why do the big labs give the highest pay of all to their prized researchers? Why do they take steps to prevent poaching? Surely a non-expert could prompt a model "find a superparameter combo that makes an amazing model" and the next gen model would emerge! The AI should be able to design a better AI on its own!
Why do the labs have thousands of employees? Lets think about this...
Who's designing the evals needed to validate the model quality, who's deciding what goes in the system prompt? Who's negotiating to buy training data from hundreds or even thousands of data vendors? Who's responding when the Internet goes apeshit over some prompt tweak that makes models over-verbose? Who's out there listening to users and feeding what they say back into the product pipeline?
Computers????
However automated things get, that quickly becomes the new baseline. Customer expectations are always going up.
And as soon as humans figure out how to automate any of the things listed above - it becomes part of the new baseline too. It's always advancing. To beat the baseline and be competitive, you need humans. No matter what. No matter how automatic the automated things get. Always and forever. Because you need someone to drive, to find the next automatable thing and automate it - and to prioritize where to search for automatable things.
> Always and forever. Because you need someone to drive, to find the next automatable thing and automate it - and to prioritize where to search for automatable things.
If we look at history of automation throughout the times it always decreased the amount of people needed to do the job. If you can't see a future where this new steam engine learns how to do a better steam engine, then you are blind. With the trajectory we're on it will happen sooner or later, and most people are having existential crises because the amount of humans required to be "in the loop" are decreasing increasingly fast.
I agree the people needed goes down - but will add a nuance.
Roles go away and fewer people are needed to do a thing, but I think it's good! For example the number of people on farms decreased orders of magnitude over the past say, century - yet we still feed everyone. Farming got mostly automated and got insanely more productive per human.
What happened was those people who would've been needed on farms went and did other stuff and the economy got way more complex but also that new complexity let a lot of more awesome things got provided for a lot more people by a lot more companies, a trend that continues today.
If you want to be a farmer good luck; it's harder to get started than it was say, 50 years ago - so why bother when there is no net new demand for farmers and there are also many better ways to make a living.
That's what'll happen for cognitively drudgerous work vs. creative new work.
What is the new work? If AI was limited only to some areas I would agree with you, but ultimately it will be more cost effective than humans at everything. Physical jobs like operating doctors might be the last bastion before AGI robots become more cost effective there as well.
https://github.com/JelleZijlstra/unsoundness is also a great collection of Python type system holes. In these cases, it's mostly deliberate design decisions, not bugs. The unfortunate fact is that a fully sound Python type system would be a pain to use.
Perhaps I'm just an old dog that doesn't want to learn new tricks, but I still can't think of Python as typed and I never use type hints. I started with Python during the 2.3 era. I still view Python types as merely docs. So it's nice that someone tracks mypy quirks but I can't imagine relying on the results for anything serious.
I've been writing python for closing in on two decades (I want to say maybe 2.5 was my introduction point?).
I've found type hints to be remarkably useful and catch significantly more bugs than I ever would have thought, and have quickly picked up the practice of using them wherever I code. My general finding is that if I have to struggle to put useful type hints in place, I probably need to rethink my code.
I even type hint my unit tests because I've learned that sometimes type incorrectness means my tests haven't been testing what I thought they were testing.
Frameworks like pydantic[0] can help you turn those type hints into something runtime enforcing too.
Generally speaking, static typing is the great enabler of writing bad Python code.
As soon as static typing is introduced into a Python code based, the developers start writing unscalable coding patterns that don't scale to larger programs.
If you deal mainly with small-to-medium size Python programs, which is probably the norm, that makes sense to me. But if you have a large Python application, or if you own a library API that might have a lot of callers, typecheckers catch sooo many mistakes.
You can't rely on Python's type hints for serious verification, but it can be great for expressing abstractions and giving people (and, today, agents) better affordances for using them. Even static errors that will not always fire can still help a lot for using a library correctly and for noticing mismatches in your codebase as it evolves over time.
To be very pedantic, Python was always strongly typed in the sense that every value has a concrete type. This is very different to, e.g. C, where `void*` is often used to actually erase the type or, conversely, more abstract languages where there are only interfaces.
But of course Python function were always duck typed so it often didn't matter to the user.
> do not consider yourself a contributor to humankind
Good lord. You could apply the same logic to any taxpaying resident of any country whose government is doing something bad. There's some truth to it, it's worth bearing in mind, but it's also the very definition of an Isolated Demand for Rigor.
> You could apply the same logic to any taxpaying resident of any country whose government is doing something bad.
You can't, because those taxes are coerced from you at literal gunpoint if you refuse (or your property seized at gunpoint to the same effect). It's completely different than choosing to do that of your own free will.
When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for, but nobody is morally tainted for being forced to contribute a small slice to something evil. Maybe there's a threshold where you should be actively laying down your life to resist, eg. if being forced to commit actual murder, but I don't think you can reasonably expect that of paying taxes at all.
---
Edit: rate-limited, attaching a reply to a below comment.
> The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good.
Indeed, but I also chose to contribute money to other good causes instead. In that way, the same sum could be used for 100% good instead of, say, 80% good 20% bad.
Now that I live in a country of my choosing, I am proud to pay taxes because of how they are used. 0% of my tax money goes to killing innocent people overseas, for instance.
When multinational trillion-dollar companies do it, it's purely for greed. Tax avoidance for moral reasons is dubious to start with, but at least ostensibly the motivation isn't the same as what drives thieves and muggers.
When Antifa attempts to keep more of its money so that Hindenburg can have less, that's good. When Hitler attempts to keep more of my money so that Hindenburg can have less, that's bad. That's because Hitler is worse than Hindenburg and Hindenburg is worse than Antifa.
Let’s not ignore the fact that multinational trillion-dollar companies will venue shop for the least tax burden to satisfy what we presume is greed, while GP venue shopped for a place to call home where he wouldn’t need to minimize their tax burden because he agreed with the taxes.
This is a massive difference in intent, which is why it’s bad when it’s multinationals.
Wrong. I specifically stated I hold myself to a higher bar than I held anyone else. I took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did.
> Wrong. I specifically stated I hold myself to a higher bar than I held anyone else. I took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did.
This is not what I was trying to say.
First let me state the assumption I made: You think that working at Anthropic/OpenAI is morally bad because that's the parent thread, and you made a comment about how somethings are good enough and other things are too bad to tolerate.
So yes, you hold yourself to the bar of I should not work at Anthropic/OpenAI - but the point is that this bar is convenient to hold yourself to.
Did you "took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did."
Or did you simply never work there, maybe never even had an offer to work there.
I wasn't particularly commentating on specific companies. The scope of my reply was strictly responding to a claim that voluntarily choosing to contribute to an evil organization is the same as being physically coerced to do it.
That said, I do hold myself to professional standards that are 'inconvenient' as well. User surveillance, manipulative dark patterns for maximizing engagement and monetization, and just about everything else big tech does is antithetical to the way I live my life. I strive to develop software that I myself am happy to use and that my users are delighted to use. My career certainly got off to a slower and less lucrative start than it otherwise would have if I had no scruples about how I made a living. I could've made much more money if I harvested my users' data and sold it. And yet, I chose not to. That is a choice you can just make. If you were in a position where you were facing homelessness, I could understand taking any job you can get, but the people chasing $300k+ TC jobs at $evilcorp aren't usually in that position.
This isn't the gotcha you think it is, and you're basically accusing the individual of "sour grapes". It isn't clever, it's plainly transparent, and about the easiest way to trivialize someone else's moral convictions. Shame on you. Be better.
> easiest way to trivialize someone else's moral convictions
Did not trivialize it at all. Regardless of what's "right" or "wrong" we are definitely in the territory of "advanced ethics" and not "trivial/beginner ethics"
> When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for
Social security? Medicare?
The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good. Every country will do a mix of things you judge good, and things you judge bad.
> nobody is morally tainted for being forced to contribute a small slice to something evil
"A Blank on the Map" is an incredible documentary about this sort of contact, though 40 years later. Here's an un-paywalled link: https://www.dailymotion.com/video/xsynjb
I don't know anything about pharma, but having worked in tech and had my share of patent lawyer meetings, it sure doesn't seem like software patents are necessary at all. That said, a patent system has a lot of free parameters that you can dial to reduce the cost. At the silly limit, if patents were only good for 24 hours, the cost to society would be ~zero (though also the benefit would be zero), so it's hard to call that "evil" in the sense of definitely causing a lot of harm. The question becomes whether/when you can set all the free parameters to something worth having.
This seems like a fundamental misunderstanding of how this sort of watermarking works. (Either that, or I have a fundamental misunderstanding of how it works lol.) It doesn't change the probability distribution of the next token at all. If you were getting XYZ 48% of the time before, you're still getting XYZ 48% of the time. What's changed is where the random numbers come from. But as far as you're concerned, there's just as random as they were before, just like an encrypted message is indistinguishable from random bytes if you don't know the key.
It is definitely blurrier whether you can say this approach changes the distribution then. By definition, it _has_ to change the probabilities of output tokens, but it's not totally clear that the pseudorandomly generated scoring functions does affect the learned distribution.
put another way, I think it's safer to do:
compute distribution -> sample -> watermark from sampled options
than it would be to do:
compute distribution -> watermark distribution -> sample
You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.
Based on the SynthID-Text paper https://www.nature.com/articles/s41586-024-08025-4 I agree that the LLM's learned distribution isn't modified, but I don't think it's correct to say that the sampling process is not modified. Also I just read the paper today so I could be misinterpreting things.
As described in the paper, you're right that it doesn't affect the main sampling technique, but what they do is they sample the distribution for 2^m samples, and then use Tournament sampling to choose the tokens among those 2^m samples, and the watermark key changes the scoring of the tournament options, using the watermark key as an input to the random generator that generates the scoring functions.
Then, to calculate the watermark, they take the text, and compute the mean g-values of the text, and a higher score means that it's more likely that it was sampled using the provided selection of tournament watermarking functions.
let's say you had some top P words: mango, banana, pineapple, guava, and you sampled 8 times, and got each one twice in the following order:
without tournament sampling, you'd truly see any of those come through. But in tournament sampling, you take those 8 options, create m scoring functions based on the pseudorandom generator, and score the 'tournament' by sampling the biased distribution you create from the g values. That does change the sampling from based purely on the LLM and entropy, but i mean, if the watermark key is also generated from some entropy, it's probably representative of the original sampling options as expected?
this is a very fascinating topic! I do still stand by my point that anthropic is the only one who can tell if something is watermarked or not and feeling icky, but the paper has mostly quelled my concern on impacting the intelligence part.
I think you're correct. It does alter the distribution for each output token, implicitly giving each candidate token a different probability. Maybe that's fine, but it's not as magical as Anthropic [and a lot of commenters here] are making it out to be.
Finally convinced myself that non-distorting watermarking is real, a la Anthropic / Google SynthID. It's not just spin / marketing. This really is a "Monty Hall" like problem. (h/t @random_walker for that analogy.)
Sharing here in case it helps anyone else.
On one hand, watermarking is "obviously" damaging to the output. The LLM does all this work to compute token probabilities... then you essentially perturb the probabilities? Of course that's bad! Every single token is using perturbed probabilities!
On the other hand, you obviously can freely inject a layer of randomness into sampling. Take any categorical distribution. For each sample, perturb the probabilities, then draw the sample. Done correctly, over many samples the results will match the original distribution.
It still kinda feels like magic, but it's not surprising that you can take that core trick and shape it into a non-distortionary watermarking scheme.
(i.e. it gives you sequences that were just as likely to be generated by the original model as any other sequence)
I assume they'll do the latter. If the vehicle is truly rapidly reusable, that's less crazy than it sounds. Just fly it 10,000 times and fix all the problems you find :) Escape systems are only really useful during launch anyway. The only way to protect the crew from a failure during reentry is reliability.
reply