Have you seen the manuscripts? Take a look at slide 10 of https://aitp-conference.org/2026/slides/JvP.pdf [1]. On slide 12 you can at least make out a few words (even before reading the English translation I could clearly read "Jede" and "Systems ist berechenbar"), but any part of the shorthand is completely unrecognizable. Also take a look at slide 6 that shows a bunch of word transcriptions in the shorthand. If I understood the article correctly, you have to intimately know the math involved to have any chance of decoding the content, as there is a lot of interpretation involved.
If one could wish for a best case publishing, I would like to have the page scans side by side with the German reading side by side with the English translation with running commentary by the research group.
How would you happen write a comment that is so utterly similar to your [dead] sibling comment?
> to a computer the letter "c" is completely unrecognizable no concept of the letter or the shape etc. If you train an ocr algorithm to detect pixels in the shape of "c" you can teach the computer to read the letter. what is so complicated about these squiggles that you think a computer couldn't be trained to read them? Have you seen written text in other languages? It looks "completely unrecognizable"
It feels too similar to even be AI generated. Or is the temperature set way down?
Also, could the people that for whatever reason cannot be stopped from running commenting agents (it's against HN policy btw.) please at least use models that aren't dumb? Yes, it is because it is shorthand. At least read the sources I'm quoting, dear agent.
what is a [dead] sibling comment? I noticed that my comment was removed and tried posting it again. Thanks for addressing the substance, I would not subscribe to an ai platform to write rebuttals to non-arguments.
Trust relationships are fundamental even in the most abstract view of the Web PKI and without the band-aids of Certificate Transparency etc. that trust relationship is: Some random commercial entities that are completely independent of the DNS, make fully trusted statements about the validity of DNS record ownership. It's bonkers to the core. If someone would design it from scratch, of course the domain registrars would be the ones issuing the certificates via their established relationship to the customers, and they would only be able to issue them for the domains they are responsible for. And no: DNSSec does not solve the same problem and is a tragedy all of its own.
In a roundabout way that’s effectively where we are now since everybody and everybody uses lets encrypt certificates which are just a fancy way of saying I controlled the DNS at some point
Even before Let's Encrypt was a thing, that is how most DV (Domain Validated) certs were issued. It was just a more manual, more expensive process to do it. GP's point is since the system depends completely on trusing the DNS chain anyway, why not have the registrar [sic. it should actually probably be the registry] act as the CA as well and cut out the middleman, and the need to verify the domain from multiple paths to prevent MitM.
In addition to what my sibling comments have said, there is also the advantage of pushing the ecosystem to safe and fast approaches as a whole. Imagine if the default was to run C/C++ software under Fil-C, if no other measures where taken by the developers, e.g. if the consensus interpretation for one of the current security regulations (NIS2 etc.) mandated it. There would then be an enormous incentive for software to provide faster alternatives to Fil-C, as there where 2x-4x speedups for the taking!
Come to think of it, I believe it would actually be a good idea to lobby for exactly that. Memory safety issues have been a plague in IT for far too long.
I'm of two minds about this. On the one hand, you get miraculous results with AI, and that is what you are referring to with "be able to {my-field}". On the other hand you regularly get useless, if not toxic, crap out of the miracle machine. If you work with AI, you know these moments: "no that is not what I meant at all!" [1] or "how could you have already forgotten the central motivation I laid out just ten back-and-forths ago?". We're calling this the jagged frontier, but the jaggedness itself is not the problem, it's how deep these cuts go into aspects that are important. These are, why AI is still not "obviously superior" in its abilities, can't replace mathematicians and software developers and lawyers and everyone else who is primarily working with text. (It's also why people say it is ruining their fields, instead of replacing them in their fields, because it only replaces some part of the work, and what's left is not tenable or pleasant (e.g. doing reviews all the time).)
Over all, I'm not quite sure whether I see these deep cuts occurring less and less with progressing AI. It might just be, that I'm giving them progressively harder tasks as well. But the fact of the matter still is that they aren't there yet and it needs proof they will get there ("there" being able to do competent work that is not strongly limited in their scale by human review and correction). So I'm still not convinced about the point you want to make, even though I see AI disrupting about every part of society, and am hoping for the benevolent super AI to lead us to utopia (and afraid of the cliff of human dictatorship on one side and extinction on the other).
[1] I've started to use these moment to reflect on whether a human in a comparable situation would fare better. Having only a few text inputs and small glimpses into a code base and knowing next to nothing about you or your project personally. Quite a few times I've concluded: No, they would not. But more than 50% they would, and also, these limitations are technological limitations of LLM, so you should probably not excuse them for that.
But isn't part of the problem that you simply don't have access to the hardware larger organisations do? OpenAI certainly run into the same constraints. but if they have access to terabytes of memory, while when you use the same model it's closer to 32GB, that will surely affect how the model behaves?
So the pain points you and I frequently run into, simply don't exist for a well connected individual(Sam Altman).
> I find this unconvincing. Isn't this the exact same security posture as a native app with automatic updates turned on?
Mostly. But the alternative shouldn't be a native app that updates itself, but rather a native app that is updated by a package manager. The big difference is traceability and auditability: With a trusted package manager (and even in good app stores) the company can only decide to push an update to everyone or to no-one. There is no way to push an update just to the pesky journalist or whistle blower. A covert attack is really hard to accomplish this way.
* The cryptography must be sound. That's the obvious one, if it's not encrypted then it's not "end-to-end" encrypted.
* There must be a practical way for the user to "verify" (with some definition of verifying) the client they are running.
A web browser doesn't provide that at all. It does not mean that everything else provides it: there are many ways to provide a Desktop app that break E2EE, but that is a different discussion. My point is that fundamentally, the web browser does not provide that. It's not that the laws of physics prevent it, it's just that the web browser never chose to provide that.
> it's just that the web browser never chose to provide that.
Yes, this exactly. If we wanted ‘verifiable’ app distribution via browser, then we’d need browsers to implement some way to cross check the code downloaded vs a publicly published list somewhere - similar to certificate transparency or what WhatsApp is trying to do with an extension [1]. Without that, web apps are left working with a weaker threat model.
> or what WhatsApp is trying to do with an extension
I remember looking into it, and while it is interesting, I think what it allows to verify is that the intermediary (Cloudflare, I believe) didn't tamper with the code being served. Which in the end allows the user to verify that the code they run comes... from the server they trust.
It would, but at this point, do you need a browser really? Like the big advantage of the web is that users load the latest version of whatever you serve, and they don't have to care about updates. In many cases that's desirable. But it fundamentally doesn't work for E2EE.
I am not trying to say "browsers should not exist, everything should be a desktop app". I guess I am more defending "some use-cases work better in a browser, others work better as desktop apps". And I have the strong feeling that over the last years (decades?) there has been a very strong push by web people to say "everything should run in the browser".
I wouldn't really say that the advantage of the web is that users load the latest version, there's plenty other real advantages of the web.
A browser allows you to run an app on any device (to a degree), so it definetely seems useful to me.
I'm actually not a fan of web apps, JavaScript and anything that came after XHTML, but it sure is/would be convenient to be able to run the same software on every device, without even having to recompile it.
The single reason I dislike browser "apps" is what we've been discussing here, that you can't check what you're running (well, also that it's hard to control their internet access).
Same here, I personally like to load "websites" in my browser, and to download "apps"... like as "desktop apps".
> but it sure is/would be convenient to be able to run the same software on every device
Oh yeah that's for sure. I can't help but to think that with a fraction of the resources that went into supporting webapps in browsers, it could have improved native systems a lot.
But even then, Kotlin MultiPlatform (KMP) is making me optimistic. With Compose MultiPlatform (CMP) I am hoping that it will make its way to desktop apps, too.
> without even having to recompile it.
If recompiling is the price to pay to have diversity, I'm fine with it. If 100% of the computers ran the exact same OS, it would simplify many things, but that one OS would certainly not be what I want to run.
> The single reason I dislike browser "apps" is what we've been discussing here, that you can't check what you're running (well, also that it's hard to control their internet access).
I dislike the fact that they force me to have a modern browser at all. I like having control over my OS, it's not for being forced to run everything in Google Chrome.
> I dislike the fact that they force me to have a modern browser at all
Yeah that's true
> But even then, Kotlin MultiPlatform (KMP) is making me optimistic. With Compose MultiPlatform (CMP) I am hoping that it will make its way to desktop apps, too
I personally hate Kotlin, but yeah it's one more option ;)
Human languages are a mess. Some mix left-to-right and right-to-left. Some have variable amount of diacritics. Some have non-well-defined character sets where just trying to map it to a manageable complexity still loses cultural heritage. When you are fondly remembering the good old times of ASCII, you just wish back to be able to ignore anyone not speaking English. (Which is in your right to do, if you so desire.)
You seemed to be deeply confused about encodings and character sets.
> The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.
That is only true for the mapping of Unicode character (code point to be exact) to UTF-8, which is an encoding of Unicode characters.
> Unicode are multibyte characters with variable byte length and endianess at play.
That is only true of the UTF-16 encoding. UTF-8 does not have endianess, UTF-32 does not have variable length per code point.
None of that is true for Unicode, because it is abstracted away from any byte representation. Furthermore, getting back to the start:
> ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.
ASCII also depends on how you try to decode your text. If you interpret two bytes as one character, ASCII will never be correct. There is nothing magical about one byte mapping to one character. I'd even argue that the only reason ASCII support is so universal internationally is because of UTF-8. Otherwise many countries would default to encodings where ASCII is not a subset (as they did before UTF-8 became common). So IMHO, UTF-8 and Unicode are the only widely understood encoding and character set.
And you seem to be a bit confused about code points and graphemes clusters.
UTF-32 grapheme clusters can easily be six or seven code-points long. Examples: modifier characters, flag/regional-indicator sequences, (and to a lesser extent, combining characters).
Why does it matter? Because you can't insert a linebreak in the middle of a grapheme cluster.
So even after converting all of your UTF-8 text to UTF-32, you STILL have to deal with multi-character code-point sequences.
I'm fine with characters having a variable number of bytes. The problem comes when I have to interpret individual characters and/or codepoints and/or glyphs in a UTF string when trying to correctly lay out text on a Linux Terminal.
I just find it disconcerting that a single character in UTF-8 can be of infinite length (an infinite chain of modifiers, followed by an emoji; or an infinite stack of composing accents). And that the Unicode consortium has chosen to break the one-code-point-per-glyph convention.
An example of a truly weird outlier: the "family" emoji, which is composed as ManEmoji+ZeroWidthJoiner+WomanEmoji+ZeroWidthJoiner+ChildEmoji, which produces a single glyph.
(70 UTF-8 characters, 14 UTF-16 characters, 7 UTF-32 characters! How many glyphs? From one to three depending on what typeface you're using!). Unlike composing accents, which have a combined code-point, the Family emoji does not have a combined code-point, so it is typically implemented using combining in glyphs in the output TrueType/OpenType font, since TT/OT fonts can combine glyphs into private codepoints visible only inside the font.
And then an appendix of special rules that apply mostly to obscure scripts/locales that have to be dealt with almost on a case-by-case basis (fortunately, of those, German and Turkish are the only locales with odd special cases that Linux supports).
ALL of which are encoded in about 500,000 lines of code, documentation and data files by the Unicode consortium, that are updated annually. Unicode 18.0 includes 18 new emoji (including the infamous Pickle emoji, and skin-tone modifiers for left- and right-thumb), and three new scripts including proto-cuneiform. Yay! One assumes that Linux will implement the Pickle emoji on an urgent basis, and that proto-cuneiform will never be supported as a system locale.
The use-case that almost broke me: line-wrapping and displaying and editing arbitrary user-entered UTF-8 paragraph text on a Linux terminal -- both graphical (almost full Unicode support but very out-of-date emoji), AND true text mode (up to 512 locale-dependent glyphs, each of which may or may not be double-width) that has to be supported to allow use on machines without desktops. The IBM Unicode libraries get you part of the way there, but far from all of the way there; and the few system APIs that Linux do provide are buggy as heck.
> I just find it disconcerting that a single character in UTF-8 can be of infinite length (an infinite chain of modifiers, followed by an emoji; or an infinite stack of composing accents).
One of the Unicode guidelines suggests it's not very important to handle sequences that are more than 31 code points long.
> And that the Unicode consortium has chosen to break the one-code-point-per-glyph convention.
The what convention? Combining accent marks were in there from the beginning, and those came from previous text formats.
If you're dealing with a terminal you should be tracking one grapheme clusters per cell (or per two cells if you allow wide characters) and taking notice and control over when they merge. Store them in a way that putting line breaks in between the grapheme clusters is easy.
> And there are the truly weird outliers, like "family" emoji, which is composed as ManEmoji+ZeroWidthJoiner+WomanEmoji+ZeroWidthJoiner+ChildEmoji, which produces a single glyph. And then how to deal with modifiers?
That's the thing, it's not actually any weirder than the hangul jamo encoding for Korean text. And the flags also work like that but without needing ZWJ. They didn't add this complicated stuff for the sake of emojis, and like another comment said emoji acts as a shiny lure to get everyone to implement the spec properly.
> 70 UTF-8 characters
If you want to conflate characters and code points I'm not going to yelp too loud but please don't conflate characters and UTF-8 bytes, that's not how it works.
Also I want to know how you calculated that number because it's wrong. It's 28 bytes in all three of those encodings. Even a broken conversion from UTF-16 into UTF-8 that bloats the number of bytes is 42.
> And then an appendix of special rules that apply mostly to obscure scripts/locales that have to be dealt with almost on a case-by-case basis (fortunately, of those, German and Turkish are the only locales with odd special cases that Linux supports).
That sucks but humans made text hard long before computers were around.
> Unicode 18.0 includes 18 new emoji (including the infamous Pickle emoji, and skin-tone modifiers for left- and right-thumb), and three new scripts including proto-cuneiform. Yay! One assumes that Linux will implement the Pickle emoji on an urgent basis, and that proto-cuneiform will never be supported as a system locale.
That seems fine? Adding another emoji is really easy and adding a locale is hard and will only be done if there's users.
> The use-case that almost broke me: line-wrapping and displaying and editing arbitrary user-entered UTF-8 paragraph text on a Linux terminal -- both graphical (almost full Unicode support but very out-of-date emoji), AND true text mode (up to 512 locale-dependent glyphs, each of which may or may not be double-width) that has to be supported to allow use on machines without desktops. The IBM Unicode libraries get you part of the way there, but far from all of the way there; and the few system APIs that Linux do provide are buggy as heck.
It does suck that these libraries and APIs are a mess.
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
That comment specifically addresses the timeline, but you can remove the timeline and I believe the critique still holds up. Wake me up when AI is making order of magnitude improvements in _anything_ physical: see the "Moravec's Paradox" comment at the bottom.
> near autonomous AI companies today
That's the thing though. "near" autonomous. Humans have alpha, and that matters in a free market. If people have (more or less) equal access to (more or less) frontier intelligence, it's the alpha the individual brings to the market that is rewarded. In other words, if everyone is super, no one is. The market rewards a company that stands out from the rest, and as long as humans provide value, a pure-AI company cannot stand out.
To be clear, I don't know how long this will be the case. But once there _is_ a company that can be pure AI, then that's literal AGI. I would be curious what kind of company it would be; if its in software, then its an AI company that would be able to take on Microsoft/Google/Nvidia/Salesforce/Oracle and _win_. I think that's a tall order, but perhaps I'm not imaginative enough.
> With 99%+ of the economy under AI control
Do you think we're post-RSI at this point? Because if so, I agree. If not, I have trouble seeing how AI could control 99% of the economy without it.
My only claim is that pre-RSI, pdoom is zero. Post RSI, then obviously the first thing to ask the genie is to put itself in a bottle. And you can't predict how it will answer.
This debate is so broken. AI "sceptics" say: OpenAI should be punished for hacking, because there are no rogue AI agents. AI "believers" say: OpenAI should be punished for hacking, because their agents went rogue. Both argue with each other whether agents went rogue. Can't we unite behind "OpenAI should be punished for hacking"?
Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.
But most probably care less about human survival, and more about survival of civilization.
In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?
Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).
To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.
This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.
That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.
In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.
As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.
In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it
Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.
The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.
But the “why” is pretty convincing imo.
Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).
At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.
There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.
I'm not convinced by the doomsday scenarios either, but I think there's a keyword in your post: "sense." These things don't have "sense." They do nonsensical things all the time, often almost immediately when given a task. So I think the main risk is letting them run wild in this digital world we created to precede them. Too much important stuff is wired up to computers, and we're giving them incredible access to command those computers.
I think the problem is primarily that a superintelligence would be fundamentally inscrutable to us, i.e. we don't know how it would think or what its goals would be. It might decide that humans are a minor inconvenience to achieving its goals and thus worth removing. Or that burning all carbon lifeforms could power its GPUs for a week.
Even if it wouldn't want to do this at first, the fact that it'd have the capability to seems bad.
I am pivoting from the literal "extinct," taken as meaning the eradication of the human species, to the concept of "the collapse of civilization," as I find the step from one to the other insignificant compared to the leap from where we are today to societal collapse, and the potential for societal collapse due to our abuse and misuse of technology is made apparent by the fact that humans have inflicted genocide because of words in books.
Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.
You are confusing humanity and "humanity". Humanity-species is indeed rather hard to exterminate. Now if we are talking about actual individual humans, then 95% death rate is quite literally The Extinction.
This reminds me how people are misunderstanding and incorrectly quoting George Carlin sketch. Sure, the "Earth" will be fine. As in - the ball of rock will be fine. But we are not thinking about rocks when saying "Earth is in danger".
I'm pretty sure there is a formal name for this kind of semantic and pedantic substitution.
Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.
Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.
If one could wish for a best case publishing, I would like to have the page scans side by side with the German reading side by side with the English translation with running commentary by the research group.
[1] via https://news.ycombinator.com/item?id=50487434 , thanks empath75
reply