you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
as a developer that had a brief career in academia, i don't think your last comment is right at all. 99.9% of what i work on as a webdev, even if it's challenging and unique at the margins, is not really novel. concerns about job security aside, i don't really think of an agent as stealing my ideas because it's good at writing CRUD APIs.
collaborating with ChatGPT on a novel solution to an unsolved problem, getting 90% of the way there, and then being "scooped" by your AI collaborator (or rather by the company behind it) is a totally different situation. were i in the same situation as these researchers, it would be extremely hard to take OpenAPI's explanation + denial of plagiarism seriously
But that is exactly what I'm implying is the core reason, whether people realize it or not.
I totally agree that the vast majority of software dev is not novel. I have even made several comments to that effect. The same can be said for a lot of creative work as well. Yet many, many devs and creators are very unhappy with AI, and a lot of their complaints are variations on accusations of plagiarism.
And note, I am not saying it is wrong, it is completely understandable, but we need to be clear about where this turmoil is coming from.
If I were in the same situation as these researchers, I would publish all pertinent research work and chats so that the rest of the world can see how close the model's work is to my own. It's been scooped anyway, so there is no reason to keep it private.
Maybe not money directly, but pretty sure it's about economic disruption. These models directly undercut the value of one's skills and labor, regardless of whether this value is measured in hard cash or abstract self-worth.
I am working on two applications using ChatGPT and Claude.
I have no illusions these people won't steal/copy whatever you want to call it, "train their models".
Yes, I keep unticking the boxes that allow it, that they so kindly tick for me.
But what happened to these math researchers is something else and I am not sure it's about the money for them. You don't do math research to get rich, but to get acknowledged by your peers. Yes, we live in a capitalist world so obviously you need money to feed yourself. but for some people, that is secondary.
OpenAI stole their thunder, and that's just fucked up.
It's not equivalent to cranking out a CRUD app for profit.
Definitely, they can also greatly assist you and that is the silver lining that I choose to focus on to prepare for the future. But most other people are focusing on the negatives because, understandably, they are immense.
As to OpenAI stealing their thunder, from all I can tell that is not what they intended. If we step away from the drama, it's low-key hilarious what happened: OpenAI heard somebody had already solved a much bigger problem -- which in fact they had not -- so they set their latest model to work on it... and it actually solved it!
Now if they had stolen the researchers work this would be a very different matter. This is something I myself have called out as a risk in the past: https://news.ycombinator.com/item?id=48839896 -- so I'm particularly sensitive to this aspect, but as far as I can tell this is not the case here.
i don't think everything that comes out like this is marketing. however, i do think that these companies are largely staffed by "true believers" (anthropic especially) -- people who are so lost in the sauce and embedded in very specific, very peculiar, sf-based rationalist circles where the ai apocalypse is a foregone conclusion.
i understand that these models are powerful and pose certain risks. i use them daily for work and the pace of improvement has been pretty remarkable. that said, i don't buy for a second the borderline-religious proclamations coming from some of these researchers, even if i believe that they are making these claims in earnest
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs