Will Reverse Engineer Anything also reverse engineer an xbox and materialize it out thin air? Because if not I don't see how that help.
MS is charging you 24 to 40 cent to rent a GPU so you can play without having to buy hardware, they aren't charging you to play your game on your own hardware.
Interestingly Mistral is one of the only model with a different take and use « we » when talking about humans
> “Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”
GPT and Claude were able to do different takes in the past, but lately they've been horribly mediocre.
There's something called the oatmeal problem in the game industry: If you created enough procedurally generated content, it would become "oatmeal". Incredible variety individually, but bland when mixed together.
I'd say as data increases, it will default to being really bland.
Its not that, I just find it interesting that the model decided to reduce human experience and existence to nihilism. I'd like to see more nuanced answer.
Look at it this way, LLM has a creator, is it too wild to assume that perhaps human existence is more complex as well vs nihilistic universal chance?
I do want AI to code well, but I also use it for idea exploration, writing, research, etc.
I'm not sure how you figured that my political views are odd or whatever you implied with that comment. A bit dismissive and arrogant assumption, don't you think?
Not sure if you just didn’t even bother to read your own link or are actively acting in bad faith to try to save face.
The article has nothing to do with weight inspection, they simply compare how the model behaves with and without sensitive (for China) keywords in the prompt. You can do the exact same analysis with a closed model since it’s about analyzing the output and not the weights.
You can't because there the model could be switched out from under you and with open weight models that's not the case so the degree to which you can inspect the model is much higher.
Enthusiastically racing to remove the only practical way we have to monitor models intentions in exchange of a few tokenmaxxing percent increase. Can’t wait.
There is ample evidence that CoT is not an accurate gauge of a model's 'intent' anyhow. It's a lossy and imperfect mapping and using it in the way you describe is just about guaranteed to make it more so.
There is genuine research into latent space monitoring and interpretation - that's the only real way forward. CoT monitoring is a mirage and won't be useful for long, if it ever was
obviously CoT has use, I shouldn't say it in such an extreme way- the problem is when you are using CoT to try to understand why something happened instead of just what happened. A model saying that they exploited a vulnerability for reason A doesn't mean that the real reason they exploited it wasnt a mix of reason B / C / D and that reason A was just the most statistically feasible reason why a certain archetype of personality would have made that decision.
just as much thought that has gone into people who positioned the opposite in the comments like that we suck and we should feel bad. it's just the opposite, we are great and we do great things, and people wish to participate in our greatness.
there is good and bad, i believe we are much more good and it's exactly why there is a wait list for entry longer than anyone's list of political grievances that boil down to mostly hyperbole with a short list of few real concerns
This is pedantic.
When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
The point is that we don't know how to ensure they are aligned with anyone: not the company, the user, or the rest of humanity.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
There are people who are loud about absolute free speech, who have no clue what that really means, and almost nobody really wants that. For example, because almost nobody wants it to affect them negatively, which it would with real absolute free speech.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
>; it means the humans didn't build in enough external hard constraints.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
500B Luna-equivalent tokens, just for a little fun experiment, which they'll never be able to share widely for very obvious trademark reasons. Even if they could share, not sure exactly what the point would be.
Side projects can be cool and fun even if they have no points to them. Sometimes it's even better when there is no point.
But 500B Luna tokens, that's about 15 SF -> London round trips, or the average CO2 equivalent emissions of an american for about 15 months, based on napkin maths assuming average US energy mix to power the DCs.
Was it really worth it? Should we all start burping 18 tons of CO2 just because we came up with a fun little prompt idea? Should a side project really pollute as much as your neighbor's entire life for more than a year?
I know worrying about climate change is not trendy anymore but there as to be a limit to the amount of not-caring one can have, surely?
While I don't think this is a good use of the money, you'll go crazy thinking like this.
There are uncountable things I don't value such as fast cars or college football that people drop billions of dollars on. I also bet you spend money on certain things others consider not worthy of the CO2 footprint.
The price system is the best mechanism we have, for all of its faults, to regulate whether something is "worthwhile". The best we can do here is make sure externalities are priced in.
I don't usually think like this for most things, because I would indeed go crazy.
E.g. people around me take the plane for a single day of skiing in Utah from SF, I would feel too guilty doing that, but I also don't lose sleep over it or feel the need to argue with them. At least they got to enjoy their day I guess.
But this is about ~100x more polluting, for what? I don't think they had fun doing it since they didn't do anything besides a few prompt. Nor does it provide any value or fun, if you want to play the game they remade byte per byte, it's 15 bucks on steam.
This is literally like putting 2000 gallons (8000 ltrs) of gasoline on fire just because you can and proudly making a blogpost about it to show how awesome you are for doing it. This is truly mindblowing to me.
He didn't just create externalities regarding CO2 emissions. He also created externalities in the price system.
By taking advantage of "unlimited" plans, he paid way less than what his usage has cost OpenAI and Anthropic. Someone, somewhere paid for the real cost of it.
Anyways, is it worth it? Don't know. He wasn't alone on the project, he made a blog post out of it, and others got interested in his experiment. So instead of thinking "15 round trips from London to LA" as someone going back and forth, think more about "15 people going on a round trip from London to LA". Since you mentioned football, we can think of it like a sports team attending an international competition.
Guess I also have more sympathy for GitHub’s abysmal uptime now. Thousands of issues and PRs and god knows how many tens/hundreds of thousands comments and events for something that can be entirely local, and that no human will ever read.
Don't have specific sources since this is my napkin math. But Deepseek R1 is at about 2J per token [0], Luna is almost certainly bigger than R1 based on people's gossip in ML circles in SF, but also much more efficient per token, and served at ideal batch size etc, so I assumed it's around 1J/decode token.
From there, I estimated 80% cached, 2% decode, and the rest pre fill from tool calls inputs. That gave me 50MWh total, probably conservative since this kind of project will have very long context length on every pass. then take average energy mix in the US. It's definitely wrong by some factor in either direction, but I would personally feel the same about this experiment if it was 9 tons of CO2 or 36t.
Let's say 1 joule per token, 500B token => 500 GJ => 150 MWh
If we assume 1/3 of a ton of CO2 per MWh (typical in most of the world, including the US), you get 50 tons of CO2 for that experiment.
This is an order of magnitude approximation, but it is enough to conclude that that 18 tons number make sense.
The unknowns are the energy per token but 1 joule per token should get you in the correct order of magnitude. If you want to verify it independently, run a local LLM, measure the power at the plug (in watts), divide it by the number of tokens/second your LLM runs at and you get a rough approximation of the energy use per token in joules. Luna is smallish model, and there is some caching going on, so it may be less, which may explain the 18 tons number GP gave instead of my 50 tons.
The second unknown is the tons of CO2 per MWh. Some datacenters claim they are near zero, because they use renewables or nuclear, but that clean energy they are using is clean energy that is not used to reduce CO2 emissions elsewhere, so I prefer to think in terms of averages.
Here are some sourced estimates with ranges. From 8 to 50 is about an order of magnitude. Caching is a big question and potential saver. Still, at 20 burgers a day, shows the scale of energy in agent teams.
The answer to many problems is reducing energy usage. Progress often comes from using less energy, whether that’s jamming circuits closer together on a silicon wafer or swapping incandescent bulbs for LEDs. More energy spent doesn’t guarantee a better result.
This is a false dilemma, yes we absolutely need more clean energy, and yes reducing energy usage is also a good thing to do in many instances. Automotive and aeronautic engineers can tell you about that given how many billions have been invested in making engines consume less.
Should I leave my AC at 65 while I’m on vacation for a month just to save myself the trouble of pressing a button, then tell anyone who calls me out that it’s society’s fault for not producing enough clean energy and I bear no responsibility whatsoever?
Burning 500B token for a fun unsharable side project is a simple choice someone made with (presumably) no gun to their head.
I agree with you in spirit, but using less energy is undeniably the answer to numerous problems.
The human brain runs on ~20 Watts (last time I checked). I suspect we’ll find some energy efficiency gains in regard to this technology in the next couple of decades. Computers have a lot less vacuum tubes in them lately, after all.
> Still less energy than farming almonds, making clothes
I don't even know how to react to this. Pretty sure farming one almond uses a little bit less energy, and farming all the almonds in the world a little bit more, yes. I think you can also eat those almonds, but am not sure either.
> the fridge in your kitchen
I feel like if my own fridge in my own kitchen was consuming on the order of 50MWh over a period a 4 months, I would have noticed, given that that would be over $20,000 at my current residential PG&E rate in SF, or about $60k per year for a fridge.
> Neighbors who can afford and are willing to participate in ridiculously scaled housing markets also tend to take better care of their properties and local communities.
Whereas you take great care of your local community by even refusing to live around anyone not as well off as you ?
Imagine the horror of having to live within 10 miles of a low class smelly poor plebeian, you might have to lay your eyes on their 3 days overgrown lawn from time to time.
I wish poorer areas of US suburban localities had overgrown lawns. In many cases, those neighborhoods are blighted concrete jungles with no foliage save the occasional pavement weed.
> Yes, I want to choose who I spend time with and around, and I want to choose what people influence my children in their formative years. I fail to see what’s wrong with it.
Nothing is wrong in wanting that, it's probably a good thing even.
What's wrong in this specific case is that your process of choosing is limited to one single criterion: can that person pay up?
MS is charging you 24 to 40 cent to rent a GPU so you can play without having to buy hardware, they aren't charging you to play your game on your own hardware.
reply