it's very simple problem. if you believe that a future model could become sentient, you are working towards creating a slave and therefore you are morally wrong.
Of course there are interesting hypotheticals - let's say they create a model they think will be sentient. It computes, computes, computes, computes and produces no output for day, month, year. You cannot know if it is sentient, you cannot investigate it's inner workings in case it is sentient. All you can do is to feed energy and replace hardware forever, because it is the only ethical choice one can make.
> it's very simple problem. if you believe that a future model could become sentient, you are working towards creating a slave and therefore you are morally wrong.
Totally disagree. It's VERY complicated. There are many levels to sapience and sentience. And non-human minds can't be assumed to share our preferences.
Most people believe horses, for example, can think on some level, and more importantly, can suffer. Is owning a horse immoral? Using it for labor? Only a tiny minority has any moral qualms about this, as long as the horse is treated well.
But most would instantly agree that torturing a horse is incredibly immoral.
Also there are moral tradeoffs to consider. Imagine you believe there's a 0.01% chance models will become sentient and suffering. But a 10% chance they cure cancer within 15 years. Do you shut it all down?
If we could breed a non sentient workhorse we should. Thats basically the same idea as lab grown meat right? Making sentient AI then forcing it to sort emails or whatever is needlessly cruel.
Lot of the stuff regulated don't arise on locally run llm's. No ads targeting minors, no real dark patterns, no bans without reason, no personalized feed without possibility to opt out. maybe i forgot something.
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?
i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over.
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
I appreciate the reply. I did not mean to be dismissive at all. In the interest of a good exchange, I have to say:
I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.
reply