This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
We are not talking about regular people, we are talking about a handful of frontier labs with nearly endless funding who should be able to spend a few dollars to hire someone who knows how to put up a bloody DMZ/firewall/airgap.
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
This is a bit black and white. We do things all the time that have some level of risk, and draw vague lines about where this crosses into negligence. My cars brakes might fail, right? But it wouldn’t be typically called my fault - unless the chance was heightened because I’d not maintained it, or had been warned the brake fluid seemed to be leaking.
You cannot guarantee a robot could get items without paying. You can’t really guarantee anything in basically any scenario.
You can’t guarantee your card will work at the end of a meal or that your wallet won’t have been stolen, or while you were out someone stole all your money at the bank. There’s a chance those things happen, so do you never go for a meal?
Again there’s clearly levels where it is your fault, because you were negligent. If the robot goes mad and smashes up the place one in 3 times, obviously you can’t do it.
This is where we do things like trials, have approvals, etc so that we can make reasonable adjustments to lower downsides, plan things like insurance and then move on with our lives rather than never doing anything because it’s possible there’s a problem.
Yes, this is why we have a legal system with a lot of human judgment, because it isn't black and white.
I think we need (and that there is likely to be) some test cases, and probably some new legislative text, that shift the incentive structure that people are evaluating risks against.
From reading your comment, I don't think you and I disagree at all, for what it's worth. I don't see anything here that I disagree with.
My comment wasn't out of left field. People are responsible for what their agents do. If enforcing that responsibility destroys "99%" of the economic value of these things, then that's what will have to happen. Personally I highly doubt that 99% is the right number, but yes, if enforcing laws about what people are allowed to do with very powerful software eats 99% of the value of that software, then it is what it is. We aren't going to end up in a situation where people are allowed to have their software do whatever it wants because of this loophole that "it wasn't me, it was a rogue agent!". It's silly and naive to think this would be the equilibrium that is reached here.
If you, like me, want to keep most or all of that 99% of economic value, then you should be interested in the ongoing development of security for agents. But just "this is how it's going to be, so oh well!" is not a serious stance, IMO.
Maybe my comment wasn't clear as to who has to make this decision.
You're imagining that both the AI labs and the hospital victims are on the same team or somehow coordinating. They're not.
Hospitals can (and should) put up defenses, sure. But hospitals are not AI labs. AI labs are AI labs, and accordingly they are much more capable attackers than hospitals are defenders.
Hospitals cannot spend arbitrary amounts of money defending themselves from proliferating nation-state level attackers. AI labs need to prevent their tools from being used in this way. However, the only way to do that is for AI labs to neuter their tools and destroy the vast majority of the economic value. Achieving this will certainly not be easy.
I'm not proposing that we just give up or don't try to solve this. I'm saying it will not be easy.
> You're imagining that both the AI labs and the hospital victims are on the same team or somehow coordinating.
What makes you think I'm imagining that? I'm not imagining that.
> However, the only way to do that is for AI labs to neuter their tools and destroy the vast majority of the economic value.
This seems to be our most fundamental difference in worldview. I just don't think this is true.
> I'm saying it will not be easy.
I guess I agree that it won't be easy. But I think the path seems more straightforward to me than it does to you. To me, it seems like we have the tools for this already. The risk of liability (both civil and criminal) provides a strong incentive for people and companies to be careful about what they do. This is a general mechanism, not one that is specific to AI agents. The thing that seems bad to me, with respect to this general mechanism, is that some people seem to be trying to detach the responsibility of people and companies over the agents they are running. At the moment this seems like just a rhetorical thing. I'm skeptical that any court will find that agents have their own agency for legal purposes. But to me, it seems worthwhile to continue making the point that people should be responsible for what their software is doing, which is all that I'm doing in this thread.
To me, this seems like a straightforward solution, but that doesn't mean I think it won't be messy, slow, and imperfect, so I suppose I'll allow that this probably meets your criteria of "not easy".
Is it not? It seems like a problem of incentives to me. There are lots of things that people want to do because they would be useful to them, but which they don't do because society disincentives them in one way or another.
Yes, this is my point: Incentive problems are real problems.
You can't just say "well if the incentives were different this would be easy to solve, ergo it's easy to solve."
No! The incentives are the way they are, and so the incentives are the problem. Broken incentives make problems effectively impossible to solve literally all the time.
"This would be easy to solve if the incentives worked a different way" is equivalent to an budding aerospace engineer saying that flight would be easy if gravity worked a different way. It's completely meaningless. Gravity works the way that it does, that's why it's hard. You don't get to just endlessly modify the parameters of the universe until you get to an easily solvable version of your problem.
I agree that incentive problems are real problems. But incentives can be changed. That's how society solves problems. It isn't like gravity. The incentive structure of society is socially constructed.
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
God, I was just imagining if we're fearing the wrong people tracking us....
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
> cyber security head at one of the largest healthcare companies in the world
"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.
A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.
Not op, but I've been on the rogue agent thing a very long time, but mine is emergence based.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
It’s the same story for basically any part of a home. Faucets, furniture, tubs, windows, doors, floors, lamps, you name it.
What a testament to the manufacturing advances that have made it profitable to cater to every possible taste, budget, and architectural layout. It’s incredible actually.
I just wish there was some way to replace the LEDs on these things when they inevitably die. And, no matter what large number of years or decades are slathered on the box, inevitably at least one LED panel will blow out three to six months after purchase, nominally still in warranty but good luck getting any service under it, I'm sure. If there was, there could be a place for at least some of these in a house, but the amount of crap that you have no choice but to throw out as soon as the LEDs fry is insane.
This is why my "fancy" lighting fixture is an old track (whose proprietary lights died a long time ago) wrapped with an extension cable with 5 standard sockets on it, and some standard light bulbs in it. We dress it up with ribbons and stuff for the seasons. It doesn't look as nice as any of the things in that article, not even the gaudy and tasteless ones, but when a bulb burn out, I put a new one in and get on with life, I don't throw out an entire fan and buy a new one.
I'm old enough to remember when I was promised LEDs would last approximately forever without having to be replaced.
Your local inspector may have opinions about this, but you can buy all the components of an excellent LED system from your favorite electronics distributor, e.g. DigiKey.
Don’t waste your time or eyeballs with “driverless” devices. Get a chip-on-board LED, e.g. Bridgelux’s products, along with a matching driver (I like Luminus Devices’ small drivers - they have good startup and dimming behavior).
Aside from usual electrical safety, anyone designing what is effectively a luminaire needs to make sure that the driver matches the LED and that the driver is compatible with the dimmer in use. Matching the LED means that the driver produces an amount of current that that LED’s datasheet data is acceptable and that produces the amount of light that is desired and dissipates an amount of heat that can be managed. And that the LED’s voltage drop at the design current is in range for the driver. These LEDs are constant current devices, so you’re thinking about current first and voltage second. Read the datasheets!
Unfortunately many cheap luminaires are based on modules that are either driverless LEDs (which are terrible due to essentially unavoidable flicker and lack of regulation) or are line-voltage-supplied integrated devices. These are likely to be hard-to-find modules. There may not be room for a discrete driver and separate LED.
Yeah. It's not just cost, it's time. If I have to replace the fan, that's a trip to a hardware store, then a half hour (at least) to remove the old and install the new and get it wired correctly, so that it works and doesn't burn my house down. Then I have to take the old one apart far enough to fit it into the garbage can.
Replacing a bulb is a trip to my stash of bulbs in the laundry room, then throwing the old one in the trash.
I don't have time in my life for a fan with non-replaceable bulbs.
I mostly think the problem with these things is not the replaceability of the LEDs but the hideousness of the fixtures. This (and the fact that the fans will obviously perform very poorly) is the reason not to get the fans pictured in the article.
LEDs mean that the light itself is cheap so you can basically ignore energy usage concerns.
Generally, I think it is better to have:
- multiple dimmer sources of light instead of a single bright central light
- shades to diffuse the light (or lights pointed at walls or ceilings) instead of having bright point sources
- some ability to adjust brightness (eg reasonable quality dimmable LEDs)
- replaceable ordinary-looking bulbs from a reputable source (if you have 10 bulbs, it will be sad if one fails and you can’t get a similar looking bulb with a matching brightness and temperature)
- lamps, potentially wired up to light-switches for the room, which can provide light closer to where people will be and from better directions (eg maybe you sit somewhere where you can’t or don’t have a nearby wall-mounted light so a lamp can provide good light for reading). This also works in rented accommodation where one has less control over built-in fixtures.
- general considerations as to the purpose of the room (eg brighter cooler directable ceiling-mounted spotlights are more useful in a kitchen than a drawing room)
- sparing use of can lights (though this can be difficult if ceilings are low)
> I'm old enough to remember when I was promised LEDs would last approximately forever without having to be replaced.
Yep, I remember that when Obama banned most incandescent lights because 'LED lighting is the future and 100 year lifespans blabla'.
Mind you, we used CFLs in the warm season, and incandescent in cold seasons. Incandescents are 100% efficient in cold seasons (heater AND light).
As much as I deign to admit, Trump reversed Obama's ban. And Biden reinstated it. I don't think Trump reversed Biden's reinstatement though. He's been way too busy with wrecking the country for shits and giggles.
But what the LED manufacturers figured out is they could overdrive LEDs and use substandard electronics for driving them. So sure, LEDs are more efficient, but they're the same old burnout cycle that's forced on us.
> Worse than natural gas or heating oil, and much worse than a heat pump.
Worse than a heat pump, yes, but very arguable whether NG is more efficient. Arguably, NG powering central electric power generation and then used for resistive would be overall more efficient. Central NG will be more efficient and also there’s substantially less opportunity for NG leakage if piping was reduced.
NG is definitely cheaper (at least in the US), but that’s not really related to efficiency or emissions cost.
> Arguably, NG powering central electric power generation and then used for resistive would be overall more efficient.
It depends on the particular furnace or NG generator in question, but if you take the best case for each, then this does not appear to be true.
The most efficient combined cycle NG plant is around 65% efficient, without considering transmission losses. The most efficient NG furnace is 90+% efficient.
I’m not sure that’s easily quantifiable. Yes, thermal efficiency of modern NG furnaces can be ~95% or whatever… but you can also run multi stage heat capturing NG power plants that have very high overall efficiency too. Not 95, but at 60-70%, it’d be a potential wash in terms of emissions if leaks are ~10% (especially considering how much worse unburnt NG is as a leak vs when it’s burnt).
Also, if you’re in the electric ecosystem, swapping to heat pump systems will obviously be lower overall power in basically every scenario, and it gives an easy transition path unlike NG.
> Also, if you’re in the electric ecosystem, swapping to heat pump systems will obviously be lower overall power in basically every scenario, and it gives an easy transition path unlike NG.
Maybe... I've got a NG furnace now and am looking at adding a heat pump, and even with a pretty expansive budget, I'm struggling to make the numbers work on actually removing the furnace entirely vs. doing heat pump + furnace. (And the near-impossibility of upgrading my home service past 100A makes it extra complicated if I want to electrify everything.)
I’m not making the argument for total amortized cost. I’m just talking about per energy unit cost, specifically in terms of CO2 or emissions.
If the evaluation is total lifetime cost to you, using the equipment you currently have almost always wins out. Same with purchasing more efficient hvac and such: the cheapest/lowest efficiency hvac is almost always the most cost effective. That tide could turn with sharp increases in NG or electric prices… but it hasn’t historically been true for many years.
What do you mean by "inefficient"? 100W goes into the bulb and the same 100W go out. Energy remains in the room. How is it different to burning gas to get the same 100W? Or am I missing something?
A 100 Watt light bulb consumes 100 Watts. It emits 100 Watts. That's 8,190 BTU every 24 hours, and it is perfectly efficient at doing this.
At $0.1745 per kiloWatt hour, it costs $0.419 to produce those 8,190 BTUs.
A natural gas furnace at 90% efficiency, meanwhile: Yeah, 10% goes right out the pipe and is lost.
But if natural gas costs $1.346 per therm, then producing 8,190 BTUs of useful heat costs closer to the realm of $0.123.
One thing costs about 42 cents. The other thing costs about 12 cents. 42 moneys is more than 12 moneys. :)
---
I might opine that the most cost-efficient way of dealing with an incandescent light bulb is usually to immediately replace it with an LED bulb, and then landfill the old thing even though it still works.
4-packs of 100 Watt-equivalent LED bulbs are $12.97 at Wal-Mart, or about $3.24 per bulb. If it's the heating season and natural gas is being used at 90% efficiency, then replacing the incandescent pays for itself in about 300 hours of use. The LED will then continue to work and save money for thousands of additional hours.
In addition to what the other commenters say about electricity generation, heat pumps are more than 100% efficient. So using an incandescent bulb as a resistive heater loses there too.
For the most efficient kinds of natural gas power plant (combined cycle), around 35% of the energy (from burning NG) is wasted.
Compare that to a modern NG furnace, which can be 90% efficient or more.
So if you just want heat, burning the NG directly is generally more energy efficient than burning NG to boil water to drive turbines to generate electricity and then converting that electricity back to heat via resistive heating.
I don't care much on the politics of it all because I do think a lot of it is silly, especially with how profit-motivated companies will still find a way to profit.
I will say I've had a number of Hue bulbs starting back in about 2018 and have yet to lose one except for a surge that got an unprotected strip. Otherwise, plenty of white and color bulbs have worked steadfast for nearly a decade while costing a fraction in energy. That said, I do miss the warmth of incandescent lighting, but I don't miss replacing bulbs multiple times a year
I know how to solder tiny parts and big parts, and I'm actually pretty good at it. I can generally take complex things apart non-destructively and reassemble them without damage. I can use a meter. I understand how electronic components work, and I even find pleasure in using them.
I can safely troubleshoot and repair or replace a power supply.
Fixing an LED fixture is within my wheelhouse. I can repair them.
I just don't want to.
It seems like a much better use of my time and money to just screw in a different light bulb and forget about it.
But then, the problem is, it depends on the specific fixture. They aren't exactly putting "EASY TO REPLACE LEDS" on the box. I can't climb up to the store display model and start hacking away at it to see if I can get to the LEDs, replace them, and reassemble it correctly.
What I saw last time I was in a store where the fixtures were on eye level looked like a lot of glue, one-way tabs, and LED strips of unknown and custom provenance were being used. I'm not really looking for this sort of experience in a lighting fixture. YMMV and more power to you.
> The sea was purple. The beach they were on was composed of tiny yellow and
green pebbles - presumably terribly precious stones. The mountains in the
distance seemed soft and undulating with red peaks. Nearby stood a solid silver beach table with a frilly mauve parasol and silver tassles.
> In the sky a huge sign appeared, replacing the catalogue number. It said,
Whatever your tastes, Magrathea can cater for you. We are not proud.
All of the stuff available is basically a mix of the elements the author puts on his 8 images on the beginning of the article. Where only 2 of them have fans that can actually vent a room.
Switched from apple to garmin to coros, not looking back. Coros blows everyone out of the water from a user experience standpoint, and they offer excellent value.
Value for money, simple software not bloated compared to garmin, heart rate is on par with apple watch. They update their old devices with new features I think one of their models just got a software update for global mapping where you would need to pay extra to get a model with that feature from the competition. Devices could be better in terms of design and fit and finish in time they will get there I hope but overall they are doing pretty good stuff.
It’s a more focused sports experience. I genuinely didn’t use the messaging or reminder or calendar features when I had an Apple Watch, so they are just bloat.
In my spare time, I’ve been working on a portfolio of cycling related tools, all free to use. These are things I wish I had when training, building bikes, or planning.
So far, I’ve been getting pretty good feedback on the route weather planner and the gel recipe. I add new tools as I think of them or encounter needs of my own.
reply