> You’ll take these low wages and be happy about it or I’ll import a foreign worker who will be happy about it. And if they complain, I’ll have them deported.
Exactly . This is even when they have setup shop outside US they bring in foreign employees on restricted visa to softly threaten local employees this keeping wages low. Mean while locals can hate foreign employees and foreign employees can feel unsafe.
Quite honestly I am in amazed at this ingenuity. Art vs artist type of awe
I was digging into this recently with ChatGPT. I’ve loosely followed the progression of ML and NN over the past 20 years, but struggled to put it into context of where an LLM lives. The big inflection point was the 2017 Attention Is All You Need paper [1].
Artificial Intelligence
|
+-- Symbolic / rule-based AI
| +-- expert systems
| +-- search / planning
| +-- logic / knowledge representation
|
+-- Machine Learning
|
+-- classical statistical ML
| +-- regression
| +-- decision trees
| +-- SVMs
| +-- Bayesian methods
|
+-- Neural Networks / Deep Learning
|
+-- computer vision
+-- speech
+-- Natural Language Processing
|
+-- Transformers
|
+-- Large Language Models
|
+-- chat systems
+-- multimodal models
+-- tool-using systems
+-- agents
My interpretation is that before the transformer, most everything under the domain of 'AI' was either an academic curiosity or only applicable in very narrow fields. GPT-3 was when the 'magic' that people had always dreamed of with AI began to emerge, and it's only really this year that we are starting to be seriously confronted with the possibility of a general intelligence emerging from LLMs (albeit, not quite the same thing as 'true' AI which would necessarily be more of a biological exercise).
I've really started changing my mind on this. "There is no AI, only I" is the position that I've really come to adopt.
Part of it is from Michael Levin's quote "Humans only can really see intelligence at human scales an immediately discount anything that doesn't exactly match their experience". The other part is most peoples immediate assumption that for something to be intelligent it has to be alive. Lastly is there may be platonic intelligence, that some kinds of intelligence may arise from the very structure of our universe when accessed.
We have really entered an age where thinking that human intelligence = intelligence is an anti-pattern that tells you less about the world and blinds you to what is actually occurring.
Biological intelligence, human intelligence, electronic intelligence, algorithmic intelligence are all subsets of intelligence set theory and even where humans like to call themselves a general intelligence it's distinctly likely that we're less generalized than we expect.
This sounds right to me as well but the point of this thread here is about how amazed one should be that this page was written in 2016. I'd say moderately, because maybe you woke up to AI being a thing this year or the last few, but the ideas obviously go back far and in surprisingly prescient ways. But this is not something unique to these AI labs. Obviously most laypeople will see the press releases and the media announcements as the big milestones and flagposts, but the academic research was inseparable from it. The AI labs obviously took on PhDs who trained in academia and had deep roots from those ideas. The story that some genius Sama invented this back when nobody thought about AI is the same exaggeration as Bill Gates inventing personal computing in his garage out of nowhere.
Honestly, I was always one of the skeptics who thought AI was a probably-not-in-our-lifetimes moonshot (imagining that it couldn't be done on existing computer architectures) and that stuff like this article was just a bunch of puffery from people who read too much sci-fi.
But in the end, a lot of it was proven right, even if it wasn't quite in the way we imagined.. A lot of pre-LLM interpretations of AI imagine it as some sudden 0 to 100 breakthrough, like one day someone writes an AGI program in their basement and takes over the world with it. There's shades of that in here too, talking about worries of organizations secretly developing AI capabilities and needing to track public data for patterns to discover it. In the end there was a 'magic program' in the transformer, but it doesn't seem like they really foresaw how the program would be useless on its own, and the 'AI' would come from ingesting as much data as possible, a process that has built incrementally over years and been very much exposed to the public.
Exactly. What wasn't really foreseen is that it would come not through some big modeling insight but mainly through massive data scale. People much more imagined some clever general, compact learning algorithm, not quite just gradient descent, but something more intellectually satisfying, and something where you'd feel like "you cracked the mechanism" and you'd see clearly that some critical missing piece had to be invented that unlocked the "understanding" in the model, maybe some kind of fancy Hofstadterian self-referential loop or something. But it just turned out to be data and compute and of course engineering the algorithms to be efficient (which I don't want to discount of course).
That's basically the bitter lesson. Academics only reluctantly swallowed that pill and still aren't satisfied with this answer. It's ugly and feels like it shouldn't work because intuition would say there are too many combinations, curse of dimensionality, etc. But it turns out it's just line go up, extrapolate Moore's law and don't worry too much about philosophical-level breakthroughs just count the flops and bits. Ray Kurzweil's scifi extrapolations turned out closer to the truth, whether deservedly or by luck.
Yes, but that idea + scale + SFT + RL made a much more general problem solver that can solve difficult verifiable tasks irrespective of human observers or the Eliza effect.
AIAYN was very influential but as its title implies its contribution was mostly about simplifying the architecture, and "attention" blocks were already known before, but their message was that you can build a model pretty much by just stacking those (and MLPs). The parallel trainability vs the rollout needed with recurrent nets (like LSTMs) made this much more scalable. But besides the architecture, what was equally important is the increase in available data, and compute. The other inflection point before that was around 2008-2012 when GPGPU (general purpose GPU programming) took off through CUDA (before that, GPGPU was much more tedious as you had to formulate your task as a graphics task about 3d meshes and pixel shaders, but people did that anyway, I had a college class on that in the 2000s).
Also a lot of the vision and speech ideas cross pollinated with the NLP field. One big trend that enabled faster progress is bringing all this onto a common platform. First via Deep Learning and backprop, formulating everything as some vector input, some model architecture, some vector output, some loss, and then gradient descent optimization. This replaced the specialized optimization tricks people used to develop for their own little niche tasks. Before DL, papers usually derived their own math for how to solve their own specific formulation of a task, so it was hard to reuse ideas.
(Reuse was also hard because platforms like GitHub didn't exist, the Python ecosystem wasn't nearly close to what we have, code sharing wasn't as common, and anyway the code was some mess in MATLAB, not in a sane language.)
The second thing that allowed converging these fields was the transformer architecture that allowed turning everything into tokens and throwing it all into the same transformer architecture, making multimodal models that can learn from everything and do everything, instead of having to make specialized models for each little task.
Essentially because attention introduced a way to scale un/self-supervised learning to the level of data out there, and learnable inference time 0-shot feature selection. Impressively in a autoregressive, unidirectional manner.
Having recently read both of Annie Jacobsen’s books — _Nuclear War_ and _Biological War_ — I’m much more concerned about a bioweapon than a nuclear one.
Her conclusion to the latter book comes out and says it, paraphrasing here:
> Nuclear war is terrifying in its destruction, but the chain of command limits the risk of a rogue actor. Whereas bio weapons can be designed and manufactured in relatively low tech environments with equal or greater magnitudes of impact.
Not sure how much Annie Jacobsen goes into the exact dynamics in Nuclear War, but it may be worth reading Ellsbergs' Doomsday Machine before you find comfort in the "chain of command."
People keep mixing up the fact that it's very hard for a virus to naturally evolve that's both extremely lethal and extremely contagious with the totally distinct claim that it's hard or impossible to create a virus that's both extremely lethal and extremely contagious.
The former is true for obvious reasons. The latter claim is not true at all. Nothing prevents the existence of a virus that's both of those things.
If HIV had been or become an airborne virus when it entered the human population in the early 1900s, there's a pretty good chance it would have been an extinction event for the human race. I suppose there might have been a few isolated pockets of humans left in 2026, but I would not personally ever use the words "good news" to describe an outcome that reduces the human population to a few thousand people.
Nukes also cannot kill everyone. Nuclear winter was proven to not be based on solid evidence, the earth remains very habitable after all nukes have gone off.
Show me that proof please. So far the counter evidence has not been convincing. FWIW I don’t think nuclear war will annihilate us. But it won’t be pleasant.
Actually, I've seen the take that the previous poster has made too. The wikipedia page is pretty good IMO. Proof of a hypothetical future event is pretty hard to get, but here's the wiki rabbit hole (if you're anything like me you'll find this fascinating and spend the next couple hours clicking links - you're welcome!).
Annie Jacobsen is awesome, her book about Area 51 was riveting, but I also cannot unsee "Capitalist Realism" and the idea that we keep picturing civilization-ending catastrophes because we don't want to go to work in the morning, and walking/biking everywhere and living in a small community is amazingly appealing to us at a base level. We yearn for the lifestyle of the apocalypse survivor, but none of us want to go through the apocalypse. Generally, I think we kind of say, "oh, it'll be the end of the world" because most of us (myself included) lack the imagination to really look beyond linear extrapolations of the status quo most of the time.
This is what I'm saying constantly to people, I recognize it makes me fantastically obnoxious, but also, I'm gesturing wildly to the things that are happening and yelling, "THINGS DON'T HAVE TO BE THIS WAY."
Then again, I'm also building my cabin in the woods.
Apollo has native support for virtualized displays, which is handy for a headless server… I just switched this weekend from Sunshine so I didn’t have to go buy a $10 dummy hdmi dongle to fake a display
After refreshing the page, I’m stuck with the whole numbers in the dropdown. I’d like to type my own number
reply