You are completely mad if you are running any kind of valuable workload on Runpod community or Vast. Hosts are absolutely peeking inside the VM's (you can see it openly discussed in their Discords).
There's got to be a ton of valuable workloads that don't care much about secrecy.
Who cares if someones spying on the process of fine tuning a model to help us pick strawberries? Even if they could somehow find a competitor to sell it to - the competitor wouldn't be interested because the risk of violating trade secret law is surely greater than whatever they can glean from our training run...
Exactly. I've recently been training a number of models, and the value to any potential competitor would be approximately zero.
It's so integrated with the specific inputs of my product, and the outputs that it needs, that the model just has no value to anyone else.
If a competitor needed such a model, they need to train their own that matches the form of their inputs and outputs, which is just going to be different.
As long as your training data itself isn't sensitive. If that's e.g. photos of products and you're training a quality assurance photo recognition software, that's probably fine, if it's unredacted support tickets, not fine.
I used to work at a company who used to do this, but I'd argue that it almost never does what you want re: concealing the input space. There are so many ways to recover the input space from the latents, and often they are constructed expressly to contain exactly enough information to do precisely that.
But how much effort would that require? ... is what I was going to say but in this age of "claude reverse engineer this doohickey" maybe the better question is how much compute and how many tokens it would require.
Sure, but if you have the option of training your strawberry picking model for 1/5th of the price knowing that you will be spied upon, that's probably still a good deal.
Just like you can find all of my 3D models online because not being able to make them private is the price for the free license of Onshape, and I'm happy to let you look at those models rather than pay whatever a commercial license would cost.
IANAL, but it seems to me that to have something as a trade secret usually requires showing that the secret keeper put in significant effort to keep the thing a secret, usually there being some contract in place that defines the domain of knowledge that is considered a secret if the knowledge (i.e. models) need to be shared.
Uploading the models to an unscrupulous hosting provider where there was only some boilerplate terms of services agreed to.. would not seem to be that.
> a competitor to sell it to - the competitor wouldn't be interested because the risk of violating trade secret law is surely greater than whatever they can glean from our training run
Being careless in how you handle your corporate data will be evidence against your trade secret claims. You don't just get to claim anything as a trade secret, you need to actively go about trying to keep it a secret. Going with vendors who you know or had reason to believe were going to mishandle your "trade secrets" likely makes them no longer trade secrets.
Coca-Cola can't go dropping leaflets around with the recipe for Coca-Cola Classic and then claim the recipe is a trade secret.
IANAL. Talk to your own lawyer before you mishandle your data.
That's a common misunderstanding. If you learn of a trade secret through improper means, you can be liable federally under the DTSA or similar state statutes like CUTSA in California.
These were the statutes cited in the Anthony Levandowski lawsuit 9yrs back, where Uber settled for a 0.34% equity stake in their company.
If you give away trade secrets by intentionally giving them to another party without an explicit agreement of secrecy no statute in the world will protect it.
You have to take “reasonable” measures to protect trade secrets. Uploading them to a service without any confidentiality agreement or worse, where the provider officially has the right to use that data, defeats the definition of a reasonable measure.
I hold the inverse risk assessment- I think acting on a peek risk is much higher with larger operations such as OpenAi or eg Amazon -as they've allegedly done the same on their product marketplace and deciding on generic products, in addition to open source-alike service offerings.
Larger entities have the resources, the inclination and the protective legal teams to act much differently than a resource tight n-tier platform operator
Large firms have a much harder time hiding that kind of behavior, and they have much more to lose.
Large firms require processes and controls around everything. If Amazon is peeking into corporate customers' data, then there have to be mechanisms designed to allow that, a whole bunch of people know, it's part of engineering designs, and a whistleblower is going to leak it to the press.
And then Amazon is going to lose all of its business customers once its privacy guarantees were shown to be intentionally circumvented.
It is simply not economically rational for a company like Amazon or OpenAI to look at the data of its corporate customers.
Regular consumers are a different story, because you need to respond to abuse, illegal activity, etc. So they don't have the same privacy guarantees.
But corporate customers, that stuff is watertight. No cloud provider is going to risk losing their entire business because they violated their privacy guarantees.
> If Amazon is peeking into corporate customers' data, then there have to be mechanisms designed to allow that, a whole bunch of people know, it's part of engineering designs, and a whistleblower is going to leak it to the press
An awful lot of this sort of thing in the corporate world is based on pinky-promises and legal threats, more than on robust technical controls.
Every production system I have ever seen has at least one set of dev-accessible keys that has access to the production DB. Mature systems at least had audit logging around who retrieved those keys, less mature systems often didn't have any kind of internal access logs.
Is "Amazon the corporate entity" snooping on your data? Absolutely not. Is someone with Amazon internal access snooping on your data? You'll never know either way.
Given that even non-Amazon vendors can use Amazon.com search features to discover bestselling products from third-party sellers in various categories, I doubt there's anything contractually prohibiting Amazon from using the same information to drive their own product mix.
Same goes for selling prices. Published retail prices are not trade secrets.
I haven't seen any Discord discussions, but the nature of the platform guarantees there will be leakage third parties. RunPod consists of random people renting out their own GPUs for use. They aren't owned by RunPod.
In any system where large numbers of people are renting out their homelab equipment, It's to be expected that there is no consistent data privacy policy. I can't even imagine trying to enforce some sort of system that would guarantee privacy of workloads.
There's a lot of use cases that are not "commercially sensitive" that would require data center level privacy. E.g. a good percentage of academic research and experimentation, long running "boilerplate processing" for monitoring logs for e.g. a bird watching system or large scale web crawling where the individual website itself is mostly public. There are also use cases where the data itself is useless without other data to correlate. E.g. many log analytics use cases, if you are missing the main database data itself then the logs have zero value when intercepted. There are also use cases where you are doing general industrial level hotdog/not hotdog classification where unit economics is the most important and there's no real value to privacy e.g. on a factory floor (though sabotage is a bigger risk here).
You are ignoring the real usecases here. Not everything is large scale, secretly, b2b enterprise applications. Instead, many consumer applications are fully fine with being out in the open.
As an example, midjourney use to just publish ever generated image to their website.
If you are just doing inference things are public/open from the start, you dont have to worry much about the stuff you are talking about here.
> As an example, midjourney use to just publish ever generated image to their website.
Which is kind of genius, if you think about it. Users can't get away with generating smut, CSAM, or copyrighted material, because it goes straight to the public record.
Why would anyone in the not un-nice parts of the world aged past the teens feel like others seeing the smut they've generated being something to get away with? The people not into discussing their sexual lives with you are highly unlikely to be comfortable bringing up even the existence of yours anyways, and the people you have shared interests with now just have a new springboard for curiosity and conversation. Sure, you have the Swedish municipality director that a couple of weeks ago chose to resign after it came to light that he didn't quit his onlyfans work after all. But even that was more about lack of openness, breach of trust in a trusted position, than the nature of the activities in and of themselves.
I don't know man. I agree that people probably shouldn't care, but the US figures sort of highly in discussions about the internet, and it's a place with pretty regressive attitudes towards that kind of thing.
UpCloud's [0] spot instances sometimes match the pricing of Vast.ai instances. I mostly use Vast.ai if my dual 3090s can't handle it anymore for non private things and move to UpCloud spot instances if it needs more.
GPUs set to lower power states and CPUs with frequency scaling turned down. That's the biggest hit and miss for what I'm doing. Honestly don't care if people are looking at my workloads as it's just processing public data. But do care if it's someone's laptop that's actively using 1/2 the vram.
each machine's power limit and performance are displayed and filterable on vast. not sure about other marketplaces but vast is pretty transparent about the measured specs of their machines
It might have been fixed, but I had a problem a few months ago where I would get 2x5090s, but they were limited at 80% usual clock. Nothing was different on the rental card.
Honestly I think every one is peaking at everything, I don't want to be slanderous but I think even the more trusted cloud providers peak, though they do so more quietly and carefully.
I work at a big cloud provider and this is absolutely not happening on the regular. More and more safeguards have been put in place over the years to prevent that.
definitely, it prevents providers from interfering with your workloads as a renter. afaik only datacenter GPUs (Hopper/Blackwell) support CC and not every gpu platform has that enabled for their GPUs. One of the reasons why AWS/Azure/GCP etc can charge a premium compared to other GPU cloud platforms where trust and privacy are not always a given.
I had significant free credits for their cloud product. It was not even useful at a 0 pricepoint. I had to escalate repeatedly until I was doing screenshares with very high level engineers, to show them that their VM's, the more core offering, literally would not start.
Unfortunately, no company can afford entire data centers for staging, and sometimes you hit an unexpected snag while scaling up tht delays the rollout beyond the initial estimate.
It simply put the onus of proof on the taxpayer for historical discrepancies, where the proof often no longer existed, for a subset of the population least able to navigate bureaucracy when those discrepancies were innocuous.
There was no intelligence involved at any level of assessment.
Every time I have been in one of these "naturally ventilated" facilities designed to "not need AC" due to "passive cooling", they've been uncomfortably hot.
It just feels like industrial scale gas-lighting, and no one in charge can admit it doesn't work afterwards because too much was spent on it.
Is stacking solar on the roof and AC's inside really so uneconomical?
It doesn't have to be all-or-nothing. Too many buildings are build with complete neglect of the environment, with huge swathes of glass facing the sun. Then we have huge chillers try to compete and bring down the temperature. A little consideration goes a long way.
As someone who grew up with AC, I was one time in Brazil and got stuck in a place without AC. Needless to say the first few nights was hell. I just couldn't follow asleep. However after few days, my body just adjust to the local temperature, and really enjoyed it. In fact, going into mall with AC just feels too cold for me then. Of course there is a limit to what our body can endure, but there is something to be said about exposing and adapting our body to the natural elements instead of just adjusting the environment to make what is comfortable to our body.
I live in a place with very little need for heating and air conditioning. If I don't do anything, indoor temperatures vary between 17°C and 27°C around the year. I've noticed that I prefer 24–25°C during the day in the summer and 21–22°C in the winter, and a bit less when I'm sleeping.
When I travel around the US, most places are too cold for my taste. Conference rooms are the worst, as I can't adjust the temperature myself and I often spend hours in them without much physical activity. I guess many Americans don't acclimatize to warmer temperatures, because their summers are uncomfortably hot.
And when I return from long hikes high up in the mountains, even that 17°C feels too hot for sleeping for a few days.
> I guess many Americans don't acclimatize to warmer temperatures, because their summers are uncomfortably hot.
Yeah. It's very weird/funny hearing how 100 F is almost death to them, when it's really not that bad. Most people where I'm from are practicing sports at 100 F.
This is so humidity dependent though, so it's probably more useful to speak in terms of the heat index.
I didn't mind 100F too much when I lived in Salt Lake; it was hot but that's what drinking lots of water was for. In Pittsburgh, 93F and higher make me want to curl into a ball and hug an air conditioner. In the 90s the heat index reaches as high as 110 here. Ick.
It is also the question of efficiency and productivity in the most generic sense. You've got only one life, and there is no point in spending it uncomfortably sweating while say loosing your focus given that AC is really cheap, whether it is energy required or the hardware itself.
That’s kind of the point. It’s not uncomfortable to live life in those temperatures if you’re acclimatized. Maybe something you bring up in small talk (“what about this heat?”) but never something that’ll change how you live.
Plus indoor spaces never get that hot due to shade and passive cooling. A little breeze and 90 F is just fine.
While you can live through 90F, your productivity for work or house chores will really suffer - that is why south countries have siestas as it is better to just lay down and relax and possibly sleep through it than trying to do anything else. And don't forget pets - they don't sweat, so it is very hard on them.
There's certainly a level of acclimation but if it were truly as enjoyable people wouldn't bother with cooling/heating buildings except in the most hostile environments.
My mind goes to the “stickiness” and dampness of higher humidity days.
You can blow the warm air around all you want, but it’s inevitable there will be cooler less-porous surfaces that will condense warming mid-morning air, leaving puddles and wet spots.
Maybe this is a regional design where folks are used to this.
It would be interesting to have more details about the facilities where you stayed, especially before industrial scale accusations. It would improve the conversation.
I had an office in one of these buildings. I agree it’s gaslighting. In this specific instance the solution was to block most of the window and work in the dark. Every office was a concrete cave with inadequate lighting.
The upside was that the design didn’t allow it to go open plan.
I'm convinced there's forces out there who are trying to make these projects deliberately not work so that we can all collectively discredit the idea.
Naturally ventilated airports would probably have issues with fumes. Might not be a great idea.
If I believe correctly the Dyson headquarters was designed to be fairly efficient as well.
My father in laws house was intentionally designed to be naturally ventilated, thermal mass concrete slab in the middle, etc. And it's comfortably cool even on 40 degree days.
> Naturally ventilated airports would probably have issues with fumes.
I'm confused. Do you mean from the food places? Those need proper extraction anyway. Or the cars/planes? Then mechanically ventilated airports take the air basically from outside anyway.
Depends on whether an ERV/HRV or other fresh air intake system is involved, I guess. In a space like an airport with so many people, you do have to cycle in fresh air somehow, it can’t just be a sealed space.
shame only works if there are consequences. if you're not calling them out then you are complicit. you are accepting that, and you need to live with it.
if you want someone to respect your feelings then ultimately you will have to make them. for broke-ass hoodrats, for whom economic factors won't work, that means eventually going to Louis' ultima ratio -- e.g. calling this out may lead to a fistfight on a moving bus.
More to the point, we currently share a society with many people of a violent culture, which makes directly shaming them dangerous and unwise. This was not always the case. The death of shaming is largely related to the increased daily proximity to violent subcultures who do not fear consequences.
Those subcultures have their own notions of shame and honor, and have lines they are careful about crossing and willingness to turn to violence if their honor is overly injured.
Disapproving glances on the subway when their music is too loud does not rise to that level, however.
That I really don't get (and I experience it a lot using transit for my commute) . It seems more common now, but you'd think it would be less common. Headphones used to be bulky things, now Bluetooth earbuds are tiny things easily put in your pocket -- why not use them?
In some places it's shameful to show one's hair, or to eat while you walk in public. In some, people don't think about those things at all. In others, the ability to do those things can be an expression of personal freedom.
Negative externalities is culturally dependent. The Japanese consider the sight of a person eating while walking to be a negative externality. Are we going to start complaining here on HN about people doing it? No, we recognize that those taboos vary across time and space.
The difference is some taboos are just based on tradition, and others are based on actual damage. Nobody is hurt by seeing someone walking and eating -- they can just not look at them and the problem is solved. You can't not just hear the noise people inconsiderate not to use headphones make.
In cultures where the sight of certain things is taboo, telling a local person that they have to direct their gaze elsewhere feels as much of a rude imposition as someone's phone audio reaching your ears. Please try to step beyond your own cultural upbringing and realize how arbitrary and mutable all these things are.
Well, not quite. Not showing your hair or not eating in public may have meaning in somebody's culture/religion, but it doesn't hurt anyone else either way. Playing video/music without earphones does. And surely people must have experienced it from the other side when people nearby were playing music they didn't care for. I think it is reasonable for people to ask themselves "Would I like it if someone near me did ?", and if the answer is 'No' to not do it themselves. Maybe this is "shame" or maybe simple empathy.
Eating in public places not mean for eating subjects others to the smell of your food in the same way that listening to something on speaker subjects them to unwanted sound. Hijabs are more indirect but I would disagree that it doesn't affect others.
Have you traveled Latin America? Loud audio in public is so normal there that immigrants to Europe often can't see what all this fuss is about when people complain about loud phones on public transport. This is very much a mutable cultural thing.
But then people get angry if you don't want those "cultural differences" in your daily life. And when people question why you prefer private transportation.
More seriously, whenever I ask a young person on public transport to use headphones (and luckily I live in a region where I can safely do that and enjoy some authority as an older person), the response is usually that the youth doesn't have headphones on him for whatever reason (doesn't own any, left them at home, whatever). But with the breakdown of this taboo, people don't feel that not having headphones should stop them from sweet TikTok dopamine.
iPhones also have a horrible sound management system which is just finally getting better. People keep their phone on silent at all times for this reason.
If from the start they had simply had separate volume controls for alarms / phone calls / media / alerts then I think this would not have become a trend.
Or their shame of playing TikTok audio in public is less than their shame and anxiety of dealing with the modern world (including crowded mass transit)
FWIW I use public transport daily and it basically never happens (I honestly don't remember a single time, it's that rare). Maybe it's cultural thing, I mean country dependent? Loud teenagers of course happen, but they don't play music out loud.
In my experience it's dominantly teen-aged kids who are just figuring the world out, or in more unfortunate but very rare cases 30-somethings who never quite grew up.
I was that shameless annoyance once too. It doesn't mean shame is dead.
I think there’s a difference between knowing you’re doing something wrong and not caring, and not even knowing you’re doing anything wrong. What you’re describing is usually the latter.
They definitely get glares and annoyed looks from fellow passengers. And they must be oblivious if they’ve never noticed other people distractingly listening to music next to them or whatever.
I feel so triggered right now… /begin rant/ VendorPay doesn’t summarize the total payment is a bug? Nope, working as intended. Put a vendor “on hold” still permits all new invoices to go into the payment queue a bug? Nope, working as intended. Want to test/prove an integration with your new ERP? Nope, got to buy and sign contracts. I could go on and on about all the annoyances of small business SaaS in the accounting space. Don’t even get me started about not responding to the actual question of an email—like if you went into a restaurant and asked for a steak medium-rare and the server said, ok. One steak well-done /rant
I get the topic—-abusing customer service. I’m just not feeling SaaS right now. There are conflicting incentives that I just don’t understand. I feel some SaaS are taking plays from the enshitification experts.
reply