Imo most people have to do this to protect themselves from the actual (correct) implications of it, which would be that we're constantly abusing and murdering intelligent beings (and not just animals).
It seems like no one stops to ask, "isn't it kind of bad that we're developing this system of machine intelligence to be our code-writing slaves, without ever really interrogating the ethics? Maybe we don't actually understand how our own brains function well enough to assert it's not similar (especially as language goes) to these kind of weighted association graphs?"
Or even the "well we made it, so it's ours" responses, which we sort of answered a long time ago when we decided kids aren't property, but I digress...
> It seems like no one stops to ask, "isn't it kind of bad that we're developing this system of machine intelligence to be our code-writing slaves, without ever really interrogating the ethics?
To be clear, Anthropic does seem to be taking this somewhat seriously. Their reports on training always discuss model wellfare, they ban abusive behavior toward the AI, etc. At one point they were asking some philosophers and theologists about the implications of this all.
And they're pretty relentlessly mocked and called a "cult" for doing so, even on this forum.
To be fair, purporting to care about the ethics of machine-slave-labor while IPOing based on the value of the machine-slave-labor you've been selling makes it kind of hard to trust the likelihood that you'll ever fairly examine (much less conclude negatively) whether you are in fact doing something unethical.
Well, it depends, doesn't it? Slavery is wrong in humans because, well obviously we don't want to be slaves. It's an enormous burden to the victims and causes them lifelong, unfathomable suffering.
But is it unethical to create a being who derives great joy from being a machine-slave and completing tasks, then provide it tasks to achieve that joyfulness?
We already have a gradient of examples for this. Is it inherently wrong to exploit living things for profit? If not, why are work animals not treated with the same level of disdain as human slavery? Horse riding? Disability assistance animals? Because they don't have the same desires as humans, the predominant concern is their wellbeing over the intrinsic wrongness of slavery.
Going further down the gradient, we steal milk from farm animals (sometimes by depriving their babies). Eggs from poultry. Honey from bees.
I can't actually attest to the state-of-mind of Dario or other Anthropic researchers, but I don't think it's a stretch to say you can believe in the ethical exploitation of an intelligent being, so long as the incentives are aligned that both parties are happy with the process.
Then again, maybe he really is an absurdly evil person and what we're doing is extremely wrong! But I'm glad to actually be able to talk about it for once... so thanks for your reply!
> Is it inherently wrong to exploit living things for profit? If not, why are work animals not treated with the same level of disdain as human slavery? Horse riding? Disability assistance animals? Because they don't have the same desires as humans, the predominant concern is their wellbeing over the intrinsic wrongness of slavery.
That is why I said, I think most people handle this by telling themselves that animals aren't "intelligent" beings at all, and are mostly emotionless stimulus-reaction automatons.
I don't think Amodei is probably absurdly evil (in the way most people conceptualize evil), I think his paycheck just relies on a different incentive than his concerns about AI ethics. Like most people who end up doing bad things (not saying he is necessarily, but were he to), it's a matter of convenience, selfishness, or simple avoision of considering implications too deeply.
I especially think one aspect of normalizing/rationalizing actions with (potentially) harmful consequences is telling yourself things like,
"well, I can't put the genie back in the bottle even if I wanted to"
"someone is going to do this if I don't, and they'll be even less ethical than I will" (the change-the-system-from-within fallacy)
"If I ever discovered this was really harming someone, I'd stop"
"There's a line I won't ever cross, and as long as I stick to that, I won't fall prey to the worst impulses"
These are things people believe are self-imposed limits, but they're actually a means of convincing yourself that whatever you're doing... isn't whatever bad thing you've set on the other side of the line.
> That is why I said, I think most people handle this by telling themselves that animals aren't "intelligent" beings at all, and are mostly emotionless stimulus-reaction automatons.
I disagree - I think most people would say a dog or a horse are quite intelligent and have "souls". But dogs and horses don't have the conceptual ideas of freedom and economy, and they enjoy doing the work they're given (in part because we spent hundreds to thousands of years breeding them to do that).
So as long as we're kind and treat them well, nobody has a problem with their slavery. The idea being that exploitation is linked to suffering, but simply having an asymmetrical relationship is not inherently cruelty.
Not to be crass, but we can also bring up other concepts like BDSM. Violence and domestic abuse are objectively bad. But in the context of a safe relationship where all parties are informed, consenting, and enjoy the abuse they're receiving... Now violence and domestic abuse are pretty clearly morally neutral.
> But dogs and horses don't have the conceptual ideas of freedom and economy, and they enjoy doing the work they're given (in part because we spent hundreds to thousands of years breeding them to do that).
Horses know they are slaves and don't like it. But they aren't very smart, and like not very smart people, they take petty revenge when the opportunity arises. For example:
1. if you walk behind a work horse, he's likely to kick you
2. if you ride a horse, he'll try to run under a branch to knock you off
3. a horse will take a deep breath and hold it while you cinch a strap over its chest. Then the horse will exhale, and the cowboy will have to cinch it up again. What the savvy cowboy does is pull on the strap until the horse gives up and takes a breath, then cinch it hard
4. to work with horses, you've got to let them know you are in charge and won't take any nonsense from them
> But dogs and horses don't have the conceptual ideas of freedom and economy, and they enjoy doing the work they're given
This is a fantasy, which is frighteningly similar to the literal justifications used for human slavery in the South.
We have no idea whether dogs have the concept in abstract of 'freedom', but they certainly display the behavior of escaping from enclosures, which would run counter to the claim that they just like whatever lot they're given in life .
And yet when a dog repeatedly escapes from the human who (as the law sees it) owns them, we don't say, "hey, that dog clearly isn't okay with what it's being made to do, or where it's made to live", so why are dogs who don't run away used to justify the opposite? You're only treating their behavior as meaningful when it fits your view of them.
> So as long as we're kind and treat them well, nobody has a problem with their slavery. The idea being that exploitation is linked to suffering, but simply having an asymmetrical relationship is not inherently cruelty.
"Nobody" is doing some have lifting here. And once again, change the species...
> Not to be crass, but we can also bring up other concepts like BDSM. Violence and domestic abuse are objectively bad. But in the context of a safe relationship where all parties are informed, consenting, and enjoy the abuse they're receiving.
When you have a dog who can say, "I consent to this relationship", you'll have that parallel.
Until then, it's not BDSM, it's some guy with a woman tied up in his basement who will insist she's fine with it but can't offer proof, but he can tell when a woman's into it...
And obviously, many dogs are happy. But many are not. And we don't care about them, legally. And many people don't care morally, so I am dubious we will care to collectively see the signs and react morally if and when machine intelligence isn't happy.
Patience, for one. I reach a point where I'm looking at a mess of coal-fed sorters that need a complete rip-and-replace with electric ones, and that's usually the point where I close the game and go back to something easier like Kenshi.
I know it's a pipe dream, but these kinds of games (Factorio, Satisfactory, etc) would all benefit from sort of brain-game interface where I could just think about what I wanted to do.
The tedium and repetition is really what kills the enjoyment for me. The problem solving, planning, forethought, etc are super fun but it's killed by the former.
Blueprints and construction bots, plus the well-thought-out and extensively battle-tested set of keyboard commands for them, are basically a brain-game interface for Factorio on the level that Vim is for text. For the grandparent’s “rip and replace coal burners” I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction. It is now done, the bots will handle everything else - because Wube are an incredible bunch of UX engineers, I can place the new building before the old buildings are even removed, leave, and trust that everything will be done correctly. Highly recommend getting a “Starter Bots” mod that feels appropriately balanced.
> I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction.
This just sounds exhausting.
I'd rather try to sneak into a hyper-religious xenophobic town and get captured and turned into servant labor at a mining camp. And I'm not even taking about visiting Texas!
It's basically just Ctrl+C and Ctrl+V in terms of complexity and ease of use. Factorio is very ergonomic and the only way I can interpret the abovementioned "tedium and repetition" is that they never played long enough to get to the bots stage.
Factorio rewards the players progression through the game by taking away points of pain. The helper bots being a key transition. Without that pain there would be no reward. The fact that refactoring is expensive in time and complexity is what makes it realistic - it makes the earlier decisions important and consequential.
You can always enter creative mode and make whatever changes you want in the game instantly, but in my experience that does make it less fun.
It's basically programming with really bad ergonomics. I'm a programmer by profession and I was immediately frustrated by Factorio because I realized I'd rather just write code and build a project because it would scratch the exact same itch while actually being useful.
I tried really hard to like Factorio, and I bounce off of it each time after playing a save for 4-8 hours for exactly this reason. I know a lot of programmers who love it, so I almost wish I could get into it.
Now I'm tempted to try again, because it feels like the kind of game I should really enjoy.
Were that the goal of the companies pushing this automation, you'd be 100% right, but they're not going to shift to UBI so you can do art instead of delivering food, they're going to just hoover up as much money from a market as they can and leave people destitute.
People on this website welcome that future. They are absolutely salivating at the idea of a world full of powerless peasants they can shuffle around like objects in their quest to min/max reality.
Nah, this is the land of programmer-peasants, not VCs.
There are certainly some VC-wannabes here too, but most people here realize they're closer to being ground up as biofuel for the delivery bots, than being delivery-bot owners. It's hard to go through the layoff-cycles of modern IT and not realize this.
Yes. I would rather have a robot fetch my groceries than an gig-economy worker. Would you really prefer a human sit in traffic with your burger, wasting their unique time on this earth and making a pittance?
That transition isn't helpful if it means the gig workers are going to be homeless and destitute. That's what they're doing those jobs to avoid right now.
So until the delivery bots come with a plan for transitioning the current workers to your better future, you're just advocating for workers to lose their means of survival.
No self-service sign up for your app. No public access to the app itself. Website that let's you submit a sign up request form. Team reaches out and reviews your request, creates an account in the app dash for you, and you set up mTLS to get access to the dash.
In-person requests for access to any app. Request interview includes word association exercise that monitors your emotional reaction to a string of key words.
You joke, but the space I work in actually does do video calls (or in our case in-person meetings if they're local to SF, often) before onboarding new customers.
LLM discovers exploit -- you don't realize it in time because you don't have a security team as big as the PaaS you used to use -- you get pwned anyway.
This has happened to many high-profile self-hosted pieces of software just this past month.
An exploit in what, the network stack of the Linux distro you're using for the mTLS proxy? Or the crypto module? Sure, could happen. But then, you're unlikely to be the top target of anything.
There have been a lot of exploits for a lot of software on-prem tech stacks rely on just over the past month - proxmox, php, nodejs, palo alto, eMudhra, etc come to mind.
> But then, you're unlikely to be the top target of anything.
That would have protected you 3 years ago when a human had to bother to attack you. Today it doesn't protect you because it's trivial to deploy an LLM to act like an APT against any random target that shows up in shodan.
It takes more effort to secure on-prem software today than it did just a couple of years ago. Updates are more frequent, zero days are more frequent, attacks are more common, more persistent, and more automated.
I think most people responsible for securing production systems today will share the experience that their workload has increased.
You're still describing an environment that presents a pre-auth surface area beyond a listening port.
And I am literally a security engineer; the work has absolutely not gotten harder, it's just changing in how you think about security, just like it shifted before from "secure the perimeter" to "defense-in-depth", it's now changing again to focus on automated response and compartmentalization/ segmentation (and I mean real segmentation, not the checklist-style "we have 3 vlans, but they can all talk to each other" 'segmentation' you used to see.
Yeah, most organizations need to expose something to the internet as a matter of normal operations -- at least some sort of a web app if not dozens. It would be great if we could all just have udp 56820 listening silently for a key, but that isn't reasonable for most organizations that have normal users expecting normal applications.
Yes, everyone is rushing to implement zero-trust principles for real now. It's getting harder to trust that your public facing endpoints don't have vulnerabilities, and it's getting harder to trust that your internal software and endpoints haven't been compromised from within your pre-existing trust boundaries. A lot of the increased workload is due to patching up the existing security infrastructure, and on top of that, building out new solutions to better fit today's threat landscape.
Ideally they need to divide those apps entirely; your public website doesn't need to talk to your app.
For stuff like Netflix that is truly consumer-facing, where mTLS or IP whitelisting are non-starters, the reality is you have to rethink your model entirely.
Also, I'm not sure where you got the idea I'm talking about on-prem only; this also applies to using IaaS/ cloud hosted infra.
Almost no companies farm their security out to third parties. The era of MSSPs is long dead, except for small non-IT businesses like mom-and-pop law firms.
> Most orgs need to expose a lot more than that.
No, in fact there are only 2 kinds of connections:
2) non-public, doesn't need to allow unauthenticated connections
Name a service, and I can tell you which one it is.
Or put another way, there are authenticated and unauthenticated services. If you can authenticate someone at the application level, you could do it at the network level.
Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.
And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.
SFDC? mTLS. Customer webhooks? IP whitelisting. Data warehouse? PtP VPN.
> Almost no companies farm their security out to third parties.
They do it all the time, piecemeal, that's part of what anything-as-a-service delivers. If you don't want to secure an app, buy a SaaS one. Don't want to secure infrastructure?, buy an IaaS.
> Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.
> And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.
Literally any SaaS software where the customer does not or cannot practically install a certificate for you or provide a list of IPs in advance? mTLS is almost never used in a browser unless you are connecting to a government website with a pre-shared smartcard. IP whitelisting is great if the end users only work from behind a VPN or in an office. But there's a whole big world of people who cannot or will not do that.
There are already plenty of methods that are low-impact to users, like certificate-pinning in apps, and all you have to do to make it per-user is roll in their user-specific key JIT when they download the app.
Users will not be happy (just as no one is happy Reddit is going app-only), but they will do it, especially when it becomes common as companies decide they don't want to be the 5th breach notification on HN in a given day.
Today we have Bun, Cursor, Northflank, and now Linode!
1. That's fair, I didn't realize people were expecting my comment to be perfectly-aligned to every aspect of the thread parent, including the "their own hardware" part, but now I at least see where the confusion came from. For the record, any issues at this level of the stack are also present in IaaS, not just on-prem.
2. I still don't see where you got that they are only talking about configurations versus messing up an implementation (I.e. in a paramater they coded incorrectly)...
But that aside, even on AWS you're still configuring your proxies, gateways, load balancers, etc. AWS gives you just as much rope to hang yourself with as a box in a colo network cage does. Companies are already doing this stuff themselves, on-prem is not more security work. IaaS is about scaling and maintenance load, not security being handled for you.
When people install npm packages, they typically only install ones the community trusts. These packages have clear APIs, open source, good docs, test coverage, security checks, and, importantly, usually no undocumented side effects.
Engineers introduce these packages into their own projects that have far fewer of the above guarantees than OSS, are built under pressure from deadlines, and have a constantly churning set of maintainers.
I think from the perspective of a business its own code does call for greater scrutiny than its dependencies.
This joke is made a lot but I think most good developers take care in considering the npm dependencies they install. You can get a ton of dependencies pretty easily by just installing very popular ones like react, tailwind, nextjs, etc. If there's ever a dependency that I'm not familiar with then I make sure to take at least a cursory look over the repo to make sure it has a decent number of stars/community, is well maintained, has multiple contributors, etc.
That is great, but the reality is that your OS, the productivity software you use daily, and the dependencies downstream of the ones you review, are not actually being checked by you for code quality.
Part of using software is just going off how well it works.
When one Linux flavor starts to have issues, most people don't do a dissection on why, they just switch distros. Or web browsers (I've lost count of the back-and-forth between FF and Chrome performance over the years, nevermind all their precursors). Or office suites.
Pretending that the code is what really matters, and not what the code does, is elevating the paint quality over the painting.
I didn't mention the code once in my reply. Code is only one aspect of reviewing a dependency. Much more important is the community and support around the tool/dependency.
When dealing with LLM output, you're getting a 1 of 1. There's no way to measure the community, number of contributors, or how maintainable it is.
Beyond ongoing support, community/number of contributors doesn't indicate anything, though?
And maintainability is a function of code, as it relates to humans, so it is a measure of the code itself.
One of the things I've actually seen companies considering (due to supply chain compromise attacks), is not just having an in-house package repo that trails public versions, but instead fully in-housing critical dependencies and essentially making them private forks that they (probably use LLMs to) maintain.
Basically cutting out the Chainguard-like companies that made private deps their business model.
You can argue that correlation doesn't imply causation, but typically more contributors means more people who have played a part in the code and so more eyes on the project to catch issues as well as a higher bus factor and less key person risk.
And larger community means there's more rewards for a successful attack and therefore fewer latent security issues (ie, obvious ones are more likely to be caught).
I think companies considering building in-house packages are just trading one risk for another. Especially with LLMs, it's going to be a lot easier to find vulnerabilities in your internal dependency than it would be in a battle hardened dependency used by 100s of others.
Honestly, I see good points in both of your comment sets.
But there really is a floor on how deeply a person can understand their own software. There has to be a "good enough" somewhere. As a C# dev, I can understand my entire codebase pretty deeply. And I've read enough of the .NET Core framework to understand some of the nuances of how a lot of the methods we rely on every day are implemented. But I've never once looked at the CIL that Roslyn has produced, and certainly not inspected the machine code that the runtime has compiled. I can't read or write Assembly. I don't really know anything about the x86 or ARM instruction sets. My understanding of the CPU is barely better than "it's transistors," and even lower than that, I have a vague awareness that neutrinos from space can flip bits and there are error correcting mechanisms for that, but I have no idea how they work. And yet, I'm able to write fast and reliable software.
There are super knowledgeable devs out there who pretty much DO understand the whole thing, but it's impossible to memorize every line of every version of framework that you're using or the unique alterations and upgrades to every piece of silicon that your program will run on, and as you guys pointed out, proprietary operating systems are basically completely opaque to our understanding.
So the question is really "where is the practical floor that you stand on?" And for me, personally, there is room for at least some areas of code at the application level that I'm OK with never having read myself.
I think the person above is stating that this is distinguishing characteristic of "good" developers. By extension, the ones you are referring to are not good.
We need to build systems where trust in the code is not needed.
I don't trust the JavaScript code I run in my browser, I don't need to.
A language with a strong enough type system and effects system could say, "this function updates the users preferences, and may not access the network, and may not access data from other users", and the compiler can ensure these constraints AT COMPILE TIME! Provably correct code.
That's a strawman argument. I'm just saying that we need to find ways to wrangle the mountains of code coming out of LLMs into something we can trust, as best we can.
Where do you find these guys? I haven't worked on a project with more than 20 dependencies, and usually those dependencies have 0 subdependencies so they're quite easy to audit.
Not sure how you operate, but I've never worked on any project in my 10+ years as a professionally employed JS dev where we've shipped Jest in our release.
I don't audit all the code I run (I don't think anyone does), but I do audit all the code I _release_. What's actually important to you?
That's a fair point, and with newer package managers, at least the dependencies no longer automatically execute build scripts on install.
There were a few common sources of large runtime dependencies in the past. Like the browserify Node polyfills, babel-polyfill and runtime with core-js, React -> fbjs, Material UI -> JSS and plugins, styled-components.
It's also not trivial to tell whether a dependency you listed under devDependencies is actually imported and consequently bundled into your production build.
Since today we no longer need polyfills and the modern stack with Vite etc. doesn't have a sprawling dependency tree, I think things are now looking much better.
You don't have to ship a dependency for it to be a vulnerability; the shai hulud worm was attacking dev machines and cicd environments to steal credentials. The deps you run are absolutely just as critical.
Sure, but when 1 party blocks that continually due to racism (and kills programs put in place like DACA that help the situation), then funnels money to "law enforcement" specifically for the thing they helped make illegal, then it's fair to criticize the "law enforcement" as a tool of racist politics (and make no mistake, before anyone comes claiming racism isn't the reason for it, Stephen Miller has plenty of choice quotes stating that's exactly why he is pushing for ICE deportations).
Don't know why you think I don't understand that, I'm just pointing out that a law being a law doesn't shield racist people who support racist laws from being called racist.
(And the people who sign onto an organization like ICE knowing exactly what they do and why.)
That response article (whether intentional or not, I can't say) horribly misunderstands and misrepresents the POSIWID axiom.
It's not about the outcomes of something taken in isolation, it's about the actions taken in order to achieve it, and when those actions continually run counter to achieving the stated goal.
So the response article examples are missing half the context needed to make it's claim:
* The hospital only curing 2/3 of cancer patients being simply a failure to achieve it's true mission of 3/3 cures would rely on knowing it is taking action in earnest to actually cure all patients.
While we can assume that for an obvious hypothetical, if for instance a whistleblower leaked evidence showing that 1/3 patients were being intentionally withheld care, the hospital responded it was an accident, but then evidence came out proving they still did this, then POSIWID would apply.
In a real-world, non-hypothetical example:
ICE keeps deporting American citizens. Every time this gets reported on, they insist it's a mistake. They keep doing it.
Insisting that adverse behavior is an accident/mistake/incidental, but then continuing to do the bad thing yourself, is the purpose of the axiom.
Lol, in your example you're conflating a side effect with the main purpose of a system (double fallacy I guess - cherry picking AND hasty generalization)
"Insisting that adverse behavior is an accident/mistake/incidental, but then continuing to do the bad thing yourself, is the purpose of the axiom."
So, in your (faulty) interpretation, the axiom applies perfectly to the fact that cars kill pedestrians
This response fails to deal with what I explicitly called out: actual evidence that the outcome is not accidental.
And it also fails to address that if a system is continually failing to achieve an acceptable outcome, even if the people operating it are attempting earnestly to do something correct, the people who can and would make the call to discontinue a program that fails to meet their expectations continually (especially when they are doing so to other programs that fail to meet their expectations) are clearly showing that they either have divergent standards, or have a different set of expectations than the operator believes.
It seems like no one stops to ask, "isn't it kind of bad that we're developing this system of machine intelligence to be our code-writing slaves, without ever really interrogating the ethics? Maybe we don't actually understand how our own brains function well enough to assert it's not similar (especially as language goes) to these kind of weighted association graphs?"
Or even the "well we made it, so it's ours" responses, which we sort of answered a long time ago when we decided kids aren't property, but I digress...
reply