You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
The way it normally works is that the user can be held liable, and then ask compensation to the manufacturer if there was some assurance or basic expectations.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
Some how the Passkey topic comes up again and again. There is not much of a background information about the Author so its really hard to take that into context. Without knowing the agenda its hard to say anything concretely however most of the information presented is twisted in certain ways
A lot of people are aware of public key cryptography or asymmetric encryption, its already supported as identification & authentication method in sensitive places via MTLS. The whole idea of MTLS is about the server verifying the identity of the client and one of the core concepts is validating that the client certificate is issued by some Certificate Authority that can be trusted by the server. Its Phishing resistant, meaning some one cannot trick you into giving them the your credential that they can use by accident. The client certificates could be baked into physical devices (PIC, CAC cards) or Digital certificate on your machine. MTLS is used universally for machine to machine communication, however there is a certain degree of complexity so its not used everywhere for user identity.
1. FIDO is trying to leverage the same concept and trying to make it bit easier for use compared to MTLS. The core idea is to just use it for authentication not identification. It allows the user to authentication as long as the client can prove that they have the private key for a corresponding public key
2. Passkeys were/are target for Enterprise/Workforce users. When a consumer account is hacked the impacted user is just the consumer, whereas when an Employee account is hacked the entire company is at risk.
3. Passkeys/ (WebAuthn+CTAP) have been there for a while. Its still still evolving and everyone (platforms, Browsers, OS, Password managers, IDPs) wants to get something out of it which keeps on making it more complex
Since the client/authenticator is storing the private key, its important to ensure that they are doing it in a uniform and secure way. The authenticator implements a protocol called CTAP and they are supposed to conform to certain tests. The attestation is a means to ensure that the authenticator is doing the least minimum from security perspective.
Personally I do not like the idea of centralization, however as long as the path to attestation for different passkey providers is not locked I would say the attestation is a part of security.
At the same time the things I would worry about would be if the Authenticator provider starts generating a unique attestation for every user in customer identity space (its almost 0 cost in case of software based authenticators). A unique certificate for every authenticator instead of a batch one is where the privacy would be leaked, its akin to fingerprint. Some big companies are already doing it for their employees, thats a valid use case, employees unfortunately do not have a privacy when they are acting on behalf of the company
Its interesting to learn them trying to implement the open vSwitch in python. I worked on implementing the open vSwitch in Java some time in 2014ish too. Open Stack had a lot of push back then. It was fun to gather the requirements by reverse engineering the original product based on the network traffic for something which is a middleware and has a state of its own
I agree thats more appropriate. There is nothing AI here, its just simple automation. Couple things that are probably applicable in this case or cases like this where royalties are earned based on assumed advertising traffic
1. There is no way he the the only one or first one. Chances are that more than likely he was subscribing to some third party services which were offering this feature.
2. He probably took it too far to get noticed and get in limelight
> https://www.githubstatus.com/
Their status page records the outage anywhere between 6 to 17 minutes across the different features. The detailed status page clearly has recorded time of when the errors were escalated, when the fix was applied and when it was assumed that the system has recovered. Its clearly a lot more than 17 minutes, infact its more than an hour.
Unless there is something more to it, these status pages feel like a blatant lie, wondering how long before some one actually sues them cause they are publishing/advertising wrong SLAs.
My hunch is that in the long run its going to be net negative in software. A lot of companies exists as of today cause a lot of people were able to just focus on building without constantly worrying about the fear of lay off. With the concept of job security based on talent and effort being dismantled, everyone would always be in a constant look out mode now.
> A lot of companies exists as of today cause a lot of people were able to just focus on building without constantly worrying about the fear of lay off.
Depends on the company, plenty have been run on a culture of fear for decades.
It's already a net negative, not just for software, but overall for humanity. The supposed productivity gains are a lie, it's causing people to be confident in information which is false, it's enabling people to treat each other like jerks in even more ways than before. AI is genuinely one of the worst things the tech industry has ever created.
This reads like a comment from 2025. LLMs are solving novel math problems and they successfully emulated a PS5, this week lol. You really need to update your priors. It's amazing how many people refuse to do this and make a fool of themselves in public
One doesn't invalidate the other. Yes LLMs have been very successful at solving math problems when guided by mathematicians who know what they're doing. The same is true for emulating a PS5 or (insert cool new project of the week developed with LLM).
On the other hand, when I use LLMs, I regularly see them confidently claim information confidently that is false. So, I'd say productivity gains are true but there's also a cost of wrong information being used to make decisions due to AI.
> On the other hand, when I use LLMs, I regularly see them confidently claim information confidently that is false
When talk to people, I regularly see them confidently claim information that is false as well.
This is not the point where LLMs become useless. LLMs become useless when we understand that we are social creatures and that many times the road is more important than the destination. Also, when you provide instant-travel to all the destinations, the destinations lose their value as well.
I think a lot of the problems or gains attributed to AI are just reflections of the humans that use them.
I dont care if you used AI to produce something. If you share it with me, I assume you are responsible for it. You own it, whether you agree or not.
> it's causing people to be confident in information which is false, it's enabling people to treat each other like jerks in even more ways than before
What is the direction of causality here? Does AI make people jerks, or they were already or prone to be jerks?
The same applies to innovations or breakthroughs made by people who use AI in their work.
> LLMs are solving novel math problems and successfully emulated a PS5
Did the AI independently accomplish those things, or did highly competent people accomplish them using AI as a tool?
What would happen if you give someone with no mathematical or programming background the same frontier models that OpenAI used for their mathematical proofs [1]. Would they produce the same discoveries?
I think of humans as constants. They can bring positive, or negative value to the table. The AI is the multiplier.
Take software engineering. If you let someone who doesnt care about code quality (their constant is negative since any code they write will introduce a tech debt) push code to your repo using AI, it will destroy your project. But let an engineer with strong fundamentals and high standards use AI to push code to your repo, they can improve the project much faster than they could without AI.
I guess my comment was done a bit too quickly and comes across as muddled. I mostly agree with you, in the hand of competent people AI is a very powerful tool. I'm a bit of a misanthrope and I think that incompetent, stupid and malicious people vastly outnumber competent, well-intentioned people. This leads me to feel rather pessimistic about the impact of AI because it is a multiplier when the human constant is often less than great.
In a lot of big companies, you had a small majority of people who were incompetent but mostly harmless because their lack of competence prevented them from causing too many issues and another group of people who were competent and productive. Nowadays, with AI those incompetent people can be maximally effective at causing issues due to their incompetence.
The same happens with opensource projects who are now flooded with mostly useless PRs and bug reports from incompetent people directing LLMs to do so. etc...
So yes individually AI makes people productive but I'm not sure if that is true at the scale of a company nor do I think it's necessarily true with opensource projects.
I agree the population distribution is skewed towards negative and average is less than 0. And when those people use AI it gets even worse because they are preventing themselves to become competent by relying on AI and skipping fundamentals.
Unfortunately the things we can control regarding other people are limited... So I think least we can do is to either surround ourselves with competent people or if not possible, ignore the AI and hold people accountable based on their final output. I often see people try to defend themselves by just blaming AI when their final output is bad but when its good its their work. But its not an AI problem, its the people problem sadly. I wish the problem was AI btw, it would be much easier to fix.
I disagree with the premise that you need to know what you’re doing to solve problems or create something new with LLMs (unfortunately).
I wouldn’t say that Dans recent work in maths was necessarily that of an expert. More of a fan of Lean than anything and he did a breakthrough in a relatively short time period.
It seems to me it’s about enough money, knowledge, and effort to want to achieve a thing. Which is good, but can and is being weaponized in terrible ways against us.
> This reads like a comment from 2025. LLMs are solving novel math problems and they successfully emulated a PS5, this week lol.
It's curious seeing "math" and "productivity" in the same sentence when the market assigns much less economic value to math than to other endeavors. The entire annual US federal budget for mathematics a mere ~$100M on average.
> The entire annual US federal budget for mathematics a mere ~$100M on average.
I'm not sure how that's a relevant benchmark, considering that the federal budget represents the spending of a single specific political institution whose primary focus isn't related to math per se.
Given the extent to which the market at large is bidding up the prices of devices whose entire function is to do math very rapidly, I don't think the overall premise holds.
LLMs are solving outstanding maths problems by trying a load of stuff until something works, which is exactly what they do in software. If you want a system built out of square blocks jammed into round holes they'll make you one in no time, but if you're working in an area where non-functional requirements actually matter (there are lots) the slopnami is a living nightmare.
And humans don't iterate? Its a matter of time before there is nothing a human can do better than an AI in programming or math, let alone other fields.
Did I say they don't? What I implied is that humans have other ways of thinking than trying stuff until something sticks, not that they never do that.
How do you think mathematicians came up with those open problems and conjectures - trying stuff until they came up with one that passed their "interesting conjecture" check? What was that check? Can you code it in Lean? Can ChatGPT? Have you seen an LLM come up with one?
> Its a matter of time...
No it isn't, and by saying that I've given as much reason to think I'm right as you have.
You write with such confidence like this is an objective opinion, and yet one I completely disagree with. Confident in information which is false? What.. ChatGPT 3.5?
I'm trying to gather the paperwork to apply for citizenship in another country. I gave Opus 5.5 everything I've assembled so far (PDFs, PNGs of birth/death certificates, etc.) and asked it for next steps and it:
* Got the answer wrong, because it neglected to search for or take into account a new law passed a few years ago.
* Missed obvious things in the provided documents, such as names and dates.
* Gave contradictory information, e.g. telling me that one step might not be ready for a year, but that I should get some things done now because they expire six months after they're released. It neglected to make the connection that if I can't start something for a year, then it would be pointless to start getting paperwork ready for that thing right now if the paperwork expires in 6 months.
I use AI coding tools a lot. I thought this would be a slam dunk for agents in 2026 and it was worse than useless.
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
It would be interesting to see how it plays out in the long run when it comes to the world dynamics. An important leverage US had was its close relationship based on mutual interest and trust with Europe and other countries which had impact on software sales.
> “Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed.” Many AI labs lack a fundamental understanding of cybersecurity
However it does not changes the fact that some damage was done. There are two things that are happening with the AI evolution which can lead to hard situations
1. Replacing deterministic systems with probabilistic systems in an attempt to get more features
2. Making critical systems available on internet to leverage integration with LLMs (AI agents need to connect with remotely hosted LLMs to be able to work) which were otherwise in DMZ (demilitarized zone)
> 2. Making critical systems available on internet to leverage integration with LLMs [...] which were otherwise in DMZ (demilitarized zone)
Honestly, I think this is actually not nearly paranoid enough. Phrased the way you do, it sounds like it's just a matter of setting boundaries in the right places and identifying "critical systems". But that's way, way harder than you'd think.
Here's my For Dummies reasoning behind the AI apocalypse:
1. AI is now at parity with median human reasoning capability and can use people's computing devices as well as the people can.
2. People commonly let AI operate their computers, and can be easily fooled into doing so in any case.
3. Society runs on computing devices operated by people.
4. There is no step four.
Basically any world where there is common access to AI agents (or whatever they end up being called) is one those agents can pretty trivially hijack.
If there is a protection regime that can prevent this, it's not about where the AI runs or what the boundary of its DMZ is.
reply