I personally don't really believe the meme that Anthropic and OpenAI are charging less than the marginal cost (cost of inference on requests). Open weight models have surpassed the abilities of their closed weight models when such claims started, and the inference costs on said open weights models are provably not so high.
Where they quite reasonably might be making a loss is their marginal costs + amortised fixed costs (training).
I predict that eventually the training expenditure is going to slow down. At that point, the large commercial providers are likely hoping they can keep inference demand up, and recoup their investment.
The driver for the training is an arms race between providers and open weights labs. But several dynamics could play out:
* Models will be seen as good enough, with only fine-tuning required, for most use cases, and demand for expensive training of foundation models drops off, stopping the arms race.
* Most of the participants cannot raise capital due to lack of market confidence in them, and there is a 'winner takes all' dynamic with the remaining provider, who can then raise costs.
I personally think the first end game is more likely than the second.
You are absolutely entitled to call it a meme but that is also their own internal reporting/panicked emails. Subscriptions are offered at a large subsidy with the benefit of more training data and adoption.
> I predict that eventually the training expenditure is going to slow down. At that point, the large commercial providers are likely hoping they can keep inference demand up, and recoup their investment.
The big labs can't do this that's their edge over open models. The stickiness is critical for them to keep up. They might try again but it's going to finish badly.
Any exploit might be interactive with a backend component that generates and serves payload components based on interactive responses.
OP (and anyone else who exploits the server) would be able to see the transcript between the backend and the vulnerable server, containing the payload customised for the server, but not the actual code of the backend. This would be useful if it was a 0day vulnerability, but with a disclosed vulnerability getting a payload that doesn't work except on their server, without the code that generated it, might not be that much value over what they already have.
That said, there are other ways to scam by structuring things like the OP (none of this should be taken to imply the OP has actually done this, but rather why this structure as a general approach might be a bad idea):
* This could be a 'stone soup' scam where the OP has the wallet keys (obviously), and puts this up to bait the addition of more funds into the wallet by other interested people. The key might not actually even be on the server, or it might not be vulnerable. Once sufficient other people add funds, the OP could withdraw the funds, recouping their original funds plus anything others added.
* This could be run as a propaganda scam either to falsely sell that it isn't an easily exploitable vulnerability when it is (by making it unreasonably difficult to exploit compared to real-life deployments), or to over-inflate how exploitable it is (by an insider withdrawal from the wallet). Manipulating public opinion could allow an attacker with a working exploit more time, or help to sell services to fix the problem.
I think an alternative scheme that is less vulnerable to scams would be to have a ZKVM (Risc0 or similar) executing vulnerable code, such that the attacker needs to produce a zero knowledge proof transcript of getting the ZKVM into a particular state that would not be possible if they didn't have an exploit. It should also take an Ethereum address as a public input. Then write an Ethereum contract that will allow funds to be transferred to the public input Ethereum address if submitted with a valid proof transcript, and back to the source of funds after some time has passed. Everyone who wants to add to the jackpot could deploy their own version of the contract, so the one proof transcript could claim from all of them, but if unclaimed the funds go back to the source.
Alternatively (avoiding the risk exploits may be so large as to make the cost of producing the proof transcript impractical), they could be built using a public cloud provider's trusted computing offering (e.g. Google Confidential VMs). The person making the challenge could make it so the derivation (NixOS for example) to produce the entire image executing on the cloud provider in a deterministic way (with publicly known source, and hash codes of each stage publicly available). Then, from the code with the known hash, they use the vTPM to generate an attestation that a key was generated while the vTPM PCRs were in a known state (corresponding to the publicly reproducible hashes), and ensure the only way to access the key from the instance is via an exploit. It would also include logic to generate and wrap a Bitcoin wallet key, and emit the public key (with the attestations and a signature linking back via a vTPM attestation to the cloud provider's key). This way, unless the cloud provider is also in on the scam, people could be assured that the OP isn't scamming them.
Maybe a better analogy: They can ban you from going through their drive through on a bicycle, but can they sue bicycle manufacturers because bike riders are going through their drive through?
In this analogy, it seems the originating court said yes, they can sue the bicycle manufacturer (in the analogy), but on appeal the answer was no, they cannot. That seems like the appeals court got it right to me.
I'm not familiar with the case, but to me it sounds like the bicycle manufacturer is encouraging people to use bikes in the drive through, maybe even providing specific tools explicitly for that purpose?
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
Note that many countries who have socialised healthcare and/or accident compensation also collect taxes or levies for higher risk activities as part of the base to pay for it all.
For example, all Australian states and New Zealand have fuel excise / taxes where some types of fuel used for motor vehicles have significant extra taxes above normal cost. Some Australian states require drivers to pay an annual personal injury levy to a government department per vehicle (usually just shy of AU$1000/year in Victoria, for example), while others require private insurance covering personal injury. New Zealand has an accident compensation scheme that levies people based on the type of work they do (which factors in risk of work-related injury), and Australia (federally) also levies everyone for healthcare based on income and whether they have private insurance.
The main difference from the US is that medical care is socialised rather than the victims of road injuries (who are not necessarily drivers) subsidising drivers.
The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass.
An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it.
PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
> No. With dummy messages you are guaranteed that traffic analysis will not be possible.
I don't think that's true.
There are two possibilities: If you're proposing that dummy messages are in addition to normal messages, then dummy messages add noise to the signal, but over a long enough time, for known noise distribution, it is possible for the signal to be statistically significant. That depends on there being enough signal, but then things depend on signal traffic patterns, there are no guarantees.
The second possible design is that the node-to-node communication volumes for any pair of nodes is truly independent of the signal - i.e. the statistical model decides how much traffic flows from a node to another node, and that fixed amount of flow is either used for signal or for noise, and if there is more signal, it's dropped or waits. Encryption makes it impossible to distinguish signal from noise without the key known between the pair of nodes. That is secure against a passive adversary who doesn't participate in the network. However, if network users are anonymous and mutually untrusting, then the adversary can use active methods without much risk (i.e. lots of users and nodes can actually be the adversary). Every packet/message delayed or dropped also allows inference about network traffic by other users.
These methods can harden a network and make analysis harder, but they don't guarantee it is not possible.
No, you always send a dummy message the appointed time.
If you have a real message, or a network maintenance message, that substitutes for a dummy message. You never send messages other than at the appointed time.
Delayed or dropped messages won't matter, since you only send a message at the appointed time. So the network is literally indistinguishable from one where you have sent no messages to all participants and to even someone with complete control of the network.
Do messages have a visible receiver address of any kind? In that case, you're out of luck. The only way to do make this statistically sound would involve an insane amount of messages, rendering it impossible.
No, of course there isn't a visible receiver address.
You have a list of peers, P1,...,PN, every second, at a specific pre-decided time, slightly fuzzed to avoid timing stuff, you send a message to each of them. If you don't put one in, it's a dummy message chosen at random and encrypted. But if you have a message, you send that at the appointed time.
Consequently, from the outside of the system, from the PoV of someone who does not have the keys, its behaviour is indistinguishable from a system where no one is sending any messagess at all.
reply