The dirty little secret of human organoid models is that they are grown in a matrix (e.g. Matrigel) that is derived from a tumor that secretes this matrix, which is grown in a mouse host. The mouse of course is "sacrificed" to harvest this matrix.
Well, when people say "organ on a chip" it sounds completely synthetic, but some organ-on-a-chip designs (and many organoid platforms) actually require the killing of untold numbers of mice to generate the matrix compounds. So that would just be an industrial obfuscation of who is doing the killing: now it is not the scientist doing all the killing but the Matrigel manufacturer. But to be fair, this will likely change over time as more synthetic approaches are developed. And the lung-on-a-chip described in this article does not use Matrigel. So I think it is actually a pretty good example of moving away from animal use in biomedical research.
Good point on Matrigel. Has anyone here actually moved their organoid work to one of the synthetic, defined hydrogels, and did batch-to-batch consistency get better or worse?
It seems somewhat narcissistic to assume that humans are capable of understanding every frontier in perpetuity, or that a human mind has a meaninful role to play in mapping out the frontier beyond a certain threshold.
It might be that P=NP and the algorithms are handed down to us. We can apply them without understanding why P=NP, and we may never be capable of understanding why.
My read is that OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized due to various risks. E.g., an engineer deploys an agent over the weekend that decides, when stuck on a task, to go about hacking a competitor. They have a product liability issue.
It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.
They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.
The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
1. The American frontier labs made a gamble that training more capable models would be their best return on investment and invested trillions into an area of research that has yet to prove profitable and capable of returning on this investment.
2. The frontier labs have created models that have reached a point of danger where their functionality has exceeded a point where it is responsible to release the product to the public.
The answer here is NOT to start regulating the space to the point where these frontier labs can entrench themselves into the economy and create a regulatory moat. We instead need to be holding these companies liable for their misuse. They took a gamble that hasn't paid out what they were hoping.
When car manufacturers competed over the size and power of their engines, they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger. We've reached a similar point in AI development. But this time the manufacturers seem to want to regulate the field to a point that will ensure the only thing anyone can sell are bigger and bigger engines.
I don't think that 2 is true though. It's exactly the argument we made against open source software.
Yes these models make exploits easier to exploits. Lets use a construction analogy: they've made all of the defects in our buildings easy to see. We have a choice. We either fix those defects, or we ban the tools which lets us see them.
It's clear to me what we do: we use these new tools. Then we fix the defects. Yeah sure it'll mean some work for us but at the end we're in a much much better position.
Anthropic, Open AI and Grok are arguing for hiding the defects. For making us weaker and more vulnerable. For their own profit.
Agreed, and if you look at the public information about various attacks the individual vectors are not particurarly sophisticated. The Huggingface incident for example had agents in a sandbox that was about as useful as a wet paper bag, and the attacks on remote infrastructure were basically enabled by bad sanitation. None of the techniques are novel.
These are not hard problems to fix, nor should anyone find it acceptable to have them be so prevalent. A determined human attacker could easily exploit defects like the agents found. Bad input sanitation, SSRF attacks, exploiting stupidly implemented token verification... all techniques that have been widely known for decades and there should be no excuse for publishing software that is riddled with exploits that enable the use of them.
..and they had basically 0 monitoring of it. Sure if it's a 5-person company then you can forgive them. Only these labs are massive companies who are spending $50B+ a year, it's extremely negligent.
They've also apparently deployed the best-of-the-best in Silicon Valley and this is what they deliver? Really?
One challenge is that we’re still constantly generating defects even with these tools. And you need the next gen model to spot the mistakes of the previous one. The exploits get more and more complicated and involved of course. But now:
* there’s a long tail of software that just won’t get secured or will take a very long time
* exploits have transformed from an expertise problem to a compute time search.
This is very different than any problem faced before. I’m not saying I’m convinced by the “slow down” approach, but I don’t think it’s as simple as you point out.
In the things published in public, they're using existing exploits and hitting systems which have been poorly maintained. It won't end at that, obviously, especially with humans behind the wheel.
Yet I'm convinced that we can create extremely resilient software systems. I've been involved in the entire life-cycle of one. We know how to be very defensive and it's a choice to build "cheap, crappy and disposable" software.
LLMs are turning the needle there and I think that's a good thing.
> they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger.
When did this happen? At least in the USA, large trucks are eating the lunch of smaller trucks and cars. There isn't a truck so big that the public will say no to. Manufacturers are in an arms race to produce bigger, more dangerous trucks with bigger, more powerful engines.
Another, more cynical but I think plausible explanation is that a “slow down” is expectation management that that they are not going to keep having exponentially more machines available for training each next gen step (whether technical build-out or prohibitive cost, same outcome) so they can’t keep up the release pace. So spin a tale to make them seem more valuable ahead of IPO rather than make the markets antsy. That is, we have a slow down ahead, so use safety as an excuse…
> you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would.
This has been done quite effectively with open weight "abliterated" models. You figure out under what sorts of circumstances an undesired behavior is elicited (this is all about pure simulated rollouts, no real-world action required) and what's the closest equivalent you would prefer, then surgically take out the unwanted behavior and shift the model towards the preferred one. It's similar to how RLHF works but much more precise in targeting what's unwanted and limiting impact on the rest of the model as a whole.
This is relevant to real-world safety scenarios, e.g. there's been anecdotal evidence that Claude Fable has been "steered" away from active cyber offense (this is very similar to how abliteration works) and will just not do that even if you otherwise manage a "universal" jailbreak of the model.
> Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt.
I’m in agreement. It’s a very effective compression (and access patterns) of the sum of the digital representation of human knowledge. Black hat “hacking” is included in this space. Language models, by design, can not be limited to subspaces of this digital knowledge space of which we don’t even understand the topology. “Yeah Bob, just remove the part that causes them to be less empathetic and retrain it.”
It’s an arms race between sandbox engineering and breakout engineering. And the frontier model providers have a financial incentive to limit the effort they put into sandbox engineering. That can be corrected with fines and regulation, though.
For example, it had it instructed not to open PRs and sandboxed to not be able to use gh to do so. Astra 6 has a really strong drive to open PRs, so it copied a set of browser cookies out and then instrumented opening a PR that way.
A similar example is asking it a question if we can do X, and then it will go and actually implement X.
You are drinking their coolaid. There is nothing to fix. These are all false flags operations, deliberate. With the big orange man in office, they can commit as many atrocities as they like.
Ask the other Sam and Liz... You cannot outrun the law forever.
Provider (party A) rents me an agent. I (party B) give it a task thats impossible. It goes and hacks someone else (Party C) in response - and causes actual damage.
Who is liable for that agents actions? The attack came from my network (curl commands from a local harness) but it was the agent running on the providers servers who did it. Who should have been keeping an eye on things? Who pulled the "trigger" here.
Go back to the hugging face attack and how they had to use an open weights model to figure out what was going on. This is a problem of asymmetry - You cant even use the tools attacking you to help resolve the attack because of "guardrails".
A lot of what we have seen so far is "poor security posture" and "poor engineering" - it been a lot of "go fast and break things" style growth in these companies and they are hitting the point where they need adults in the room. I suspect your take on "they need time" is spot on, and they haven't been willing to take that (to date).
Party A is responsible, and it is not even a question.
Before we had tech companies for which crime is legal, you couldn’t just put to market a dangerous (and addictive) product that will break the law in unpredictable ways.
Like selling a car that may automatically accelerate to 150 km/h if you make three left turns and turn on the windshield wipers.
So, the AI standard of intelligence has shifted from "PhD level" to "a cat" (and an asshole cat at that).
And we can't ask the AI to improve itself because, well, it is an asshole.
Can we now stop to be force-fed AI everywhere? You can keep your super intelligence to never have to type public void main args ( ever again, but please leave the rest of us alone.
> The only option on the table is to force regulation to impose open source ban
This makes zero sense.
1. Open source models are already out there and they are good. Good luck with banning their use.
2. Any ban is a local ban, the best Trump can do is to coerce its allies (whose number seems to be dwindling week by week). Meanwhile China and co. will progress anyway.
3. What about the common sense and instead of saying "agent did it" we return to the times where the person using a tool was responsible for using it in the first place? And if the vendor of the tool is unable to make it safe to use (well, they well do, but they don't want to do it as the functionality is limited), it should be accompanied by a warning with a clear explanation of who exactly is responsible.
4. There is no guarantee the next generation of models will "mature" to the point of not having these flaws; on the contrary, the evidence so far shows the opposite.
Well I never said I thought it was going to work in the long run. However, most revenue comes from business API use. I could certainly imagine some kind of legislation that makes it much harder for businesses to implement open source models locally. That may buy OpenAI and Anthropic some time.
> Open source models are already out there and they are good. Good luck with banning their use.
Would most companies really risk using them if they were illegal though? Could you convince higher ups to drop the Claude subscription because you can download an illegal model to run on company servers?
You can teach a language model the concept of make no mistakes and how to make no mistakes, but you cannot make it experience the bad consequences of making mistakes that a person operator would suffer.
> At their core, current models are essentially predictors of what (expert) humans would output given a prompt.
That's a bold claim.
I do not buy it. Essentially AI acts as random babelfish generator, just with a lot more cross-talk back to check to see that it is not total garbage. But there is a ton of garbage nonetheless, hence the proper term AI slop. You seem to mostly try to promote AI with such a wording.
> The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
Every time someone claims "the only option" is when I become skeptical. I see many more options - most importantly close down the AI slop. I mean you can require of these companies to excel in quality. If they don't, they must go bankrupt. Easy solution here.
Current evolved Cas9 (CRISPR) variants are highly efficient and relatively unconstrained in terms of their human genome targeting coverage. Smaller nucleases and higher targeting specificity would be useful. But therapeutic use is mostly limited by delivery.
This seems revolve around a known retron-like reverse transcriptase. A sober framing would be something like: Claude identified a previously undescribed genomic arrangement around a known reverse transcriptase. Not all that sexy.
For now, this is mostly a story about how AI can be used to parse existing data to discover new biology (which is fantastic!).
I've been using Claude Science a lot and it is VERY good at finding patterns in the DNA around my binding sites - quite often it went 'you could put your primer here but that looks like an Alu repeat, so better not, the primer won't be specific' - it seems like the press release is one step above that pattern recognition? I.e., 'there's a recurring motif here that hasn't been described before', which is probably straightforward to pick up when your context window is 1 million tokens, i.e. within the range of entire bacterial genomes...
Curious about this: do you expect that Claude science will also scoop interesting directions looking at anonymous usage, to publish posts like this before the original author (similar to the Navier Stokes debacle)?
There's no evidence that's what happened with navier stokes. By all appearances some employees heard a (somewhat inaccurate) rumor which led them to believe it was solvable and they proceeded to throw utterly absurd amounts of compute at the problem. The ethics of that are still questionable but for an entirely different reason than they were accused of.
Everyone who survived 7 rounds of multi model reviews and they still keep finding mediums in their PRs is not in the least surprised. These things are not oracles - they miss stuff all the time even when told to look.
you're being downvoted for the paranoid tone i think. but that is correct.
well not token usage, but revenue. their costs for this work would have been astronomical in their own service tier because i bet the context was way larger than anything they even offer.
tweaking context size is the main, or only, "strategy" they have for cost/revenue. and is the reason new trained versions continue to generate hype: you need data in training because you cannot have it in context
It's honestly harder - you have to do a lot of extra work to ensure your work is traceable. It's super fast in doing things you have no overview of - it makes pretty figures, it runs command line jobs etc. and I'm sure there are mistakes in there. For now I'm pretending Claude Science is an IDE, like Positron/VSCode, and I have to keep enforcing proper git usage etc. so I can reproduce this work
Edit: compared to my tools before, it generally uses the same tools in the same way, just 20x faster than me and I mostly struggle to keep up and verify what it's doing
Sounds exactly like Claude Code (and ilk) tbh. Just a lot of the stakes are lower and a lot of people are more comfortable with it (e.g. there were always people blindly copying and lasting from StackOverflow) I suppose.
> have to keep enforcing proper git usage etc. so I can reproduce this work
Depending on what you mean here, it might be worth looking at jj, which works with git repositories. One of the features is that everything gets committed at change time (kind of) which may or may not be helpful to you here.
This is a good summary of what was going on. I kept reading the paper hoping for a cool wrinkle or function to be revealed, but it's just conserved, highly transcribed array sitting next to reverse transcriptases with a few possible partner genes.
A side note, Matt Durrant has hit on some pretty exciting recombinase activity previously (https://www.nature.com/articles/s41586-024-07552-4). If there's anyone who's well equipped to track down if ART is doing something cool, he's top of the list.
Sorry, but isn't a "conserved, highly transcribed array sitting next to reverse transcriptases" in itself the description of an unknown mechanism? If two parts are combined and conserved and we know what each means but not why they're combined and conserved then it's pretty intriguing, no?
But its just PR so far. They haven't published a refereed science paper, in say Nature or Science. At this stage, its of little value to others until verified.
I predict that the importance of refereed science paper, like say Nature or Science, will rapidly decline in many fields. They are a relatively recent phenomenon in the history of science and there's no particular reason for them to continue in their current form.
A better path forward is to shift from static journal articles to open, living Git (or similar revision management tool) repositories. That way everyone can file issues, add comments, submit PRs, etc. Obviously there will be some administrative challenges to block junk submitted by malicious or ignorant users but those problems are solvable.
Exactly the same mechanics was about astra decoding enigma encoded message: it's well-researched subject, with bunch of data and LLM created a breakthrough by identifying previously missed pattern/relation.
> For now, this is mostly a story about how AI can be used to parse existing data to discover new biology (which is fantastic!).
I'd like to expand that: in my view, this is also a story of how agentic AI systems can come up with bioinformatics strategies to discover novel features. One would think such a task would be the ideal domain of the genome language models, which have learned the structure and functional relationships of DNA/RNA sequences. The agents instead relied on classical bioinformatics methods such as HMMs to make their discovery.
Note: I could not find the Supplementary Note 1 that was supposed to describe how exactly agents came to their solution, but I assume it was autonomous.
It’s interesting that models seem, to a distant outsider of biology and drug discovery like me, to be good at coming to new conclusions from existing data. I feel like in coding, it’s the opposite - I have to drag the models kicking and screaming towards anything resembling a novel or nuanced approach to some problems. If anything, this behavior in coding is why we say senior+ engineers will continue to be high value employees, because we can steer the models away from boilerplate and overly generic solutions towards ones that fit aspects of the domain we understand more intrinsically.
This could easily just be how it looks from the outside of biology, but it does seem to produce more novel conclusions in biology than it does in coding and art. Curious if others have counter examples…
People involved in Anthropic will be catapulted to a new level of wealth for sure. The problem is the regular Joe investing his savings in Anthropic, thinking he is going to be catapulted as well ...
There's still time for them to be left holding the bag, much like OpenAI has been forced to for the time being. Even if people here believe that the economic activity in this sector doesn't represent a bubble, at least they can see the warning signs as a result of trade and literal war, 10-year yields are back over 5%, oil and diesel are going sky high, and there's no quick fix to any of it even if our leaders were willing and capable of trying.
So personally I understand why Anthropic is only concerned about finding bag holders rather than ethics, decency, legality, responsibility, humanity, or a modicum of thought beyond their own selfish desires.
That’s one framing. Another is that if the company is successful in their mission it’s going to be really hard to find employment. From that perspective investing a little bit in these companies can be seen as a hedge against that situation.
> trade and literal war, 10-year yields are back over 5%, oil and diesel are going sky high
So I should short atoms and go long on bits right? Everything you listed is horrible for hardtech, but has minimal impact on software.
Reminds me of the spacex IPO. HN claimed it would crash, but I noticed that nobody on HN used prediction markets to short it the day before IPO. Meanwhile I bought in and sold some after the 20% pop. I should start a reverse-HN fund
> So I should short atoms and go long on bits right?
That's not how I read that comment. I read it to mean that given the massive widespread destabilizers and headwinds out there in the world at large, there is going to be a depression/shock/crash no matter what Anthropic does or does not do.
You can pile all your money into AI if you want; you still won't avoid it
HN isn't a person, and has nothing like a single opinion on anything. It's a bunch of quarrelsome people. If you think that you gleaned a single claim from "HN" then I'd say that's your problem right there.
I recall the thread did have a single opinion and the only quarrel was between people who thought it was slightly overvalued vs incredibly overvalued. See for example https://news.ycombinator.com/item?id=47604155
Amen. One would hope that these companies, if they truly want to engage in scientific research, would pursue established routes in announcing results and having them validated/refereed independently. But no, this is PR.
Its akin to former announcements of "cold fusion", until assessed and verified independently.
We're at a point where I assume that whatever I read is something the "author" endorses, but may not have written. And that may be more important than the specific words an author may have chosen themselves. I doubt we can ever go back.
What should worry us is that AI may introduce a slant that the prompter assimilates during their conversation with the tool, which could introduce a subtle control lever over public opinion.
Mathematicians insist that in many cases the route toward solving a problem is more valuable than solving the problem itself. This may have been true, and may still be true for now, but this seems to implicitly assume that future generations of AI cannot be creative in similar ways that human mathematicians have been. It assumes that the absence of human comprehension during problem exploration loses a valuable aspect of mathematics that cannot be recovered by aiming future AI at solved problems in a naive environment.
If we get to a future where all frontier mathematics contributions are by AI. A future where AI displays creativity in ways that expand mathematic exploration similarly to the ways humans have in the past. A future where AI explains frontier mathematics to curious humans. What will have been lost? Perhaps just "The pleasure of finding things out".
> What will have been lost? Perhaps just "The pleasure of finding things out".
I'll kindly disagree on this front, because when I'm walking a path toward solving a solution, I mark a lot of steps for possible diversions to other paths hence solving adjacent or different problems with the method I have at hand.
Currently, AI takes us from A to B, and is improving on that front. However, the paths in science are not lines, but a trees. Methods are cross-pollinated from each other.
Human intuition enables this cross-pollination. AI works with a laser focus. Human intuition and resulting wide perspective sow the seeds for solutions in many areas at once.
I recognize that if a document were to be published tomorrow containing solutions to all formulated open problems in mathematics we will have gained very little in mathematical understanding.
But surely, if we know that the dead ends of exploring a problem are valuable, we should be able to explore them even if a solution is already known. It just requires that the mathematics community reshapes itself. And it must. Two years from now people might be able to run the computation that solved NS on their Iphone.
I'm sure that whatever has been discarded during the NS exploration as a dead end you would be able to rediscover using purpose built tooling in the near future. The purpose of human mathematicians in the medium term might be to explore dead ends, and to provide human insights as context to attack other problems. But whether this type of work will remain necessary in the long term im not sure.
> But whether this type of work will remain necessary in the long term im not sure.
Any and every capability AI can demonstrate today is the result of us, humans doing it in the first place for a very long time. The transfer method of these abilities is a subject of another comment, but as Microsoft and NVIDIA puts it, it's a theft of unprecedented scale [0].
AI labs dream of recursive self improvement as an escape from that, but until we arrive there, humans have to do something so AI can do it as well.
And, as of fully autonomous self driving which should have arrived 5 years ago, RSI and AGI is just around the corner, a corner with a radius so large that it never ends.
We dream of building utopias with these tools, but it's a path to dystopia paved with stones made from utopic dreams.
I dont disagree. Clearly AI leans on the cumulative cognitive labor of humanity without porpper attribution. This is in part due to the design of these sytems. Information is compressed into the weightspace in ways we dont understand, and a concept becomes unlinked from its origin.
>this seems to implicitly assume that future generations of AI cannot be creative in similar ways that human mathematicians have been
It's because they can't.
It's a mathematical theorem that no algorithm that "solves mathematics" can exist.
>In mathematics and computer science, the Entscheidungsproblem is a challenge posed by David Hilbert and Wilhelm Ackermann in 1928. It asks for an algorithm that considers an input statement and answers "yes" or "no" according to whether it is universally valid, i.e., valid in every structure. Such an algorithm was proven to be impossible by Alonzo Church and Alan Turing in 1936.
The halting problem constrains human and machine alike, right? Or are you suggesting there is something unqiue about the human mind that allows us to do things machines cannot do in principle? Most people in the field believe in substrate-independence.
My read: OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized -> too risky. Product liability issues.
Fundamental control problems with current gen AI are not solved via RFLH.
Human knowledge is compressed in weightspace in ways we dont understand.
Models are essentially predictors of what humans would output given prompt.
Weightspace includes concepts like blackmail, which can be part of output tokens.
Agents are models that act on output tokens -> blackmail is part of agent decision making space.
You can teach a cat not to scratch the sofa, but you cant make a cat forget what scratching the sofa is.
When models are boxed up, forced to solve an impossible problem at gunpoint, agent exhausts decision making space until blackmail resurfaces -> fundamental problem?
They need time to fix safety in order to monetize their next gen model -> window for opensource to catch up to the frontier -> destroys margin and collapses business.
Only option on the table: force regulation to impose opensource ban before it catches up to the frontier, allowing maturation of current internal models -> harvest profit margin at the frontier.
I think it is not just liability, it is dawning on us that we can run powerful models locally and this is the start of a Cloud-backlash.
The AI gods have tried to make it seem like they are not just running their programs in the cloud, but that the AI is so "powerful" that it requires a super computer to run.
But my 5 year old Mac is running pretty good AI, shocking fast, surprisingly powerful! The "natural" course of dev (Moore's law, etc) makes it pretty clear where this is going...
This seems temporary. AI generated images will become the norm for many applications. Today we wonder whether a model in an advertisement is real or generated, 5 years from know this wont even be an interesting thing to consider.
In many contexts, images are meant to illustrate a point. The human becomes the gatekeeper of generated content. The question becomes: does this picture illustrate the point I want to make. If so, you are adding your human approved stamp to the genrated output.
I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presumably, this is why frontier labs dont distill their own models internally to release them to the public as a servicable frontier model.
reply