I think the conclusion is more like mandating home and business web access is proxied at a national level and most other protocols disallowed. Use of VPNs without a VPN tax stamp and key escrow is prima facie evidence of violation of the Counter-Terrorism and Security act. Use of non-standard protocols is assumed to be criminal intent. Check with your approved internet provider for how to access the Internet safely.
The midlife crisis is probably better described as a change or unease in the self and not knowing why you get out of bed in the morning anymore and that entire adventure, whereas this anomie is the social environment you're moored to breaking down.
I think the strongest point out of this was that the US populace is a dangerous vector for our adversaries.
In 5th gen warfare, the battlefield can be kinetic or information, and might be all around us and omnipresent.
What we're guessing at without this decade's version of a Snowden leak is what the equities of the US military are in this. Are we playing defense, offense, and where? Is the answer just, "yes"?
What about our adversaries, or more importantly, _who_ are our adversaries?
Consider just the usual 3 who are in the news: Russia, China, and Israel:
There's so little that's public about these wars and so little traction in the mindshare of the electorate, you'd wonder if it was real if you just talked to random people.
If you believe some of the emerging comments out there - try "Danfoss CVE" - you'll see that some of the suspects are 15-20 year old controllers like the Danfoss SM800 (EOS 2021) and SC355 (EOS 2016).
Why don't they just--- no margin.
But there were CVEs-- no margin.
The laptop class doesn't always grok that the shift-work labor class has nothing in terms of luxuries like security reviews and gear replaced on cycles and other basics taken for granted.
But I'd setup Tailscale and-- would you, the grocery store manager running on 3% margin, really? What if you're part of a chain and it's centrally controlled elsewhere, like all the other decade-old tech you use but have to call and wait for a dispatched tech to handle?
Secure-by-default laws will eventually help, but there's always going to be groaning infrastructure particularly in commodity industries, and we still haven't onboarded most of the disaffected men to the Internet that will commit innumerable more hacks and disruptions.
> The laptop class doesn't always grok that the shift-work labor class ...
I think it's more than this. Industrial control system upgrades are serious business and (at least if you're doing them properly) require a lot of planning and consideration. Particularly when the subsystem being replaced is decades old already.
I think the overwhelming majority of people (including on hn) have no understanding of how complex industrial systems are - even as-built, but particularly after decades of evolution. Doing proper engineering in these spaces is expensive.
You can reasonably blame supermarket chains for being insecure, but we certainly aren't currently enabling them (or anybody else) when it comes to fixing these issues.
Is it more expensive to keep your systems upgraded and maintained or to have all your product get destroyed one arbitrary day and then have to pay to do the upgrade anyways so that the arbitrary property destruction doesn't keep happening?
In before "but insurance"; yes insurance can in theory mitigate some of this, but I feel like people on HN have no understanding what it's like to deal with a multi-million dollar insurance claim that the insurance company 100% does not want to pay out. The legal battles over such things take years and are on their own incredibly expensive.
The thing is that many IoT devices aren't simply updated. The industry doesn't work like software development countries.
They usually think in projects. Some project manager, some funds and some workers are assigned to a specific project. And than this project is followed through. Eventually the project is done (finished or dismissed). And all the allocation shifts to the next project.
This is e.g. the reason why most of the Android phones have so bad security. Even Qualcomm does the Android stuff on their chips only project wise. Is the project done, then you'll get an Linux kernel from 2021 with this specific device. There will never be an update, even if this outdated Linux kernel has 20 known CVEs. The reason is simple: no PM allocated, no funds allocated, no workers allocated.
And such project-thinking also exists for other hardware chips, e.g. network or WIFI chips. Lots of development to get the thing "ready to market" --- which isn't necessarily to get it good --- and then almost nothing.
And this is the hardware the industry automation uses. Except that there some intermediaries in between, e.g. Kontron that make industry-grade boards. But even at that level you have the project structure. These boards are then used by the manufacturers of the industry automation which, who would have thought it, also mostly think in projects.
And this is the reason why the S in IoT stands for security.
People at various levels would need to switch to a process mode, away from project mode. And people at various management levels need find a way on how to sustain this financially.
> Is it more expensive to keep your systems upgraded and maintained or to have all your product get destroyed one arbitrary day and then have to pay to do the upgrade anyways so that the arbitrary property destruction doesn't keep happening?
It happened once. Did you do the maths here? Can you tell us which one was Kore expensive and what percentage chance there was of this hack happening?
Someone did the math. Cost of continual maintenance, versus cost of product loss. They determined it wasnt worth the maintenance cost, and accepted that risk. And that risk manifested.
This happens all the time. Unfortunately, in my experience, most of the risk acceptance happens without even really believing in the cost of the risk, and people just accept everything. it is highly likely someone like that accepted this.
The problem with security maximalist[0] ideology is that it does not accept reality for what it is: shit will happen. The correct way to handle risks is to mitigate them to some extent, and then insure (or self-insure) against the risk that remains[1].
Then, when the risk actually materializes and mitigations prove insufficient, you write off the damage and carry on with your day - and yes, perhaps even reward the executives who set up mitigations and insurance that allowed the business operations to continue undisturbed.
Only then, mitigations MAY be upgraded and insurance premiums adjusted, as an event happening is an important new evidence in risk estimate for it happening again.
That's how adults in the room handle security.
It's only the general public and cybersecurity people who see random events happening and decry that somehow it affects them and Somebody Needs To Be Accountable, and Someone Must Do Something About It.
--
[0] - Or should I say, cybersecurity maximalist, because in every other field except software, this is well understood.
[1] - Talking about regular security, that applies to 99.99%+ of businesses; the risk to life and limb is a special case with extra considerations, though it fundamentally doesn't change much about the framework.
I think it's fair that the price up until now has been mostly zero. On the other hand you can 10+ years of examples on HN of people saying these types of threats with IoT devices and internet connected industrial hardware are not theoretical and it's shocking they weren't being exploited more often.
The thing I can‘t wrap my head around: why does a freezer need internet access at all? My freezer at home works just fine without. I bought it 20 years ago. Worst case in the supermarket: a freezer dies on Saturday night, you throw away the goods on Monday morning and call someone capable to fix it. Lowest TCO solution.
My random guess: freezer need temperature audits and franchise supermarkets have to prove they're properly managing the food. If the central monitoring service can also alert/dispatch staff within minutes to deal with issues, it becomes a pretty attractive proposition.
Something that might not be obvious, but temporary dips in temperatures also need to be monitored as it's enough to spoil the food. Worse case scenario isn't to have to throw 50~100 kilos of food, it's to have spoiled food still looking fine and consumers ending at the ER, with the whole franchise name on the news.
> My freezer at home
That can't be your frame of reference for a business entity part of a franchise.
There are better (and offline) solutions to detect temperature dips in the freezer (e.g. a thermograph, or cheaper, solutions that change color). And as a former trucker delivering frozen and refrigerated goods I can tell you the most dangerous part of the journey to the customer is not in the freezer of the POS. That being said, I admit I don’t know the legal requirements today. But I doubt someone will be sent out at night to salvage a few pounds of frozen spinach.
On better solutions managed locally, yes of course. Small shop's also deal with the same requirements without keeping their stuff online.
> someone will be sent out at night
We're talking about France, so that's an even lower chance of happening for a freezer's worth of goods.
The two upsides I'd see are the ability to throw away the food next day before opening the shop, and the franchise being able to force that procedure even if the local shop would have chosen otherwise.
Local tenant are usually a lot more lenient than their franchise manager (for good reason usually, but there's always that tension)
this is sad. its like tragedy of the commons. you could have lived in a village without thieves and you dont need a lock. but as soon as there is one thieves, everyone in the village now need to have locks. you could have all your servers without security in a village with social trust but the moment you have evil actor everyone must now have security expenses that do not add functionality just to fight that single pathogen.
so the expensea are related not to amount of adversaries but to amount of things you need to protect. because its enough to have one bad guy to inpose expenses on whole village.
That is a dangerous game, especially when affecting food supply. Widespread cyber attacks on critical infrastructure are deemed grounds for nuclear response in most national doctrines. This obviously is way out of scale, but concerted widespread efforts by decentralized groups could end up triggering ww3 by accident. Not when they try to play a game of chess, but when they shut down a water treatment plant, and some other unrelated group shuts down gas pumps, and suddenly there are millions of people with no fresh water and no fuel to get it to them...
It's psychology. One option is a guaranteed upgrade bill that's due right now. The other one is a speculative what–if that may never happen.
And that's even assuming anyone who runs these sorts of systems have "hacked to turn on defrost mode" on their bingo cards at all, which probably isn't the case.
On top of that, upgrades carry risks of their own!
It was „what–if that may never happen” before agentic AI.
Nowadays you can be pretty certain it will be exploited.
It is not just run faster than slowest guy to get away from the bear, now it is dozens of bears per runner and bears are stupid enough to chew on anything but also smart enough once they bite something tasty that works will just continue biting that.
In my recent experience (last 3 months) with a cyber claim getting insurance to pay out was actually pretty straightforward. It was about a $250K USD loss on a phishing attempt (someone hijacked and rerouted payment details for a wire)
But you don't need to replace entire system to massively reduce surface area for the attack. Just do what was done for decades, and put the vulnerable devices between VPN and/or additional authorisation layer.
It's far less work than touching anything on controller side (at most changing some network settings) and requires attacker to at the very least penetrate your network perimeter and snoop around to find it rather than "run nmap on a controller's default port over country's IP range"
An enterprise grade VPN is pretty damn safe. A government could probably get in but then a government probably isn't going to bother hacking your fridges to defrost.
> I think the overwhelming majority of people (including on hn) have no understanding of how complex industrial systems are - even as-built, but particularly after decades of evolution
I may not have experience, but it is standard security practices that monitoring and control must live in separate channels for security. This is not a "planning and consideration" concern, this is conceptual design of systems.
A supermarket may have 30 freezers. You don't want them to beep loudly if someone forgot to close the door. Instead you connect them to a central monitor. A local employee can see on the monitor which freezer has a problem and go see the issue.
The monitoring is as well a proof that the cold chain was kept.
This network was local to the market. Then one day it was found better to collect these data to a modern dashboard, and so the local network ends on corporate network.
> You don't want them to beep loudly if someone forgot to close the door.
Why not? It works as an alarm, is simple in design and requires no external input.
They need to be checked 3x per day anyway for temperature so someone will notice eventually. And checking temperature must be product core temp so no remote monitoring. Unless haccp changed massively over the last 20 years to "we don't care anymore".
This is remote management, not just monitoring. Say you want your fridges to integrate with weather predictions, or prepare for a change of stock in a specific freezer across 100 sites.
Features are appealing, but hopefully LLMs can make these cowboys finally secure implementations that they have never been forced to before.
An audible alarm encourages people to tamper with it when it breaks and starts misfiring. Sure, they should put in a maintenance request, but they frequently take things into their own hands, even if maintenance is good about these things. And if the audible alarm just dies and there is no central management, how would you know? Stores don't really want to have to pay people to stand around with a stopwatch to determine if a time-based audible alarm still functions properly, and they also don't want to scare customers when things go wrong or when the alarm correctly goes off for a door just slightly ajar (and thus it wouldn't be immediately obvious which one needed to be closed).
With centralized monitoring, this all becomes much easier, and you can inspect a whole row of freezers in less than a minute. You can even do it remotely, just watching when customers open and close the doors on a security camera and checking that the control system registered it correctly.
See also: dump trucks that have alarms if you start to move while the bed is up. It's a great safety feature on the highway, but it's annoying on the job site if you can't just dump everything out while standing still, so it frequently gets disabled by the workers, especially if it starts to misfire. This is how you end up with dump trucks going highway speeds with their beds up taking out overhead signage and slamming into bridges. The stakes are much higher, yet workers would still rather rip out the alarm.
How do you measure core temperature of, say, a block of frozen fish without destroying the packaging and contaminating the good? Do they throw it away after measurements?
Apparently the freezers needed to coordinate defrosting. Evens out grid load or whatever. Making everything Internet controllable is one way to achieve that.
This slowness also means it sometimes has a hard time understand new risks and a lot of industrial systems are not operated by people who understand those risks and do…very stupid things.
100%; I've worked on many a project that was implemented without proper concern for cybersecurity. It seems like this hack in France was triggered via a missed update (according to the manufacturer of the system, who I am hesitant to believe tbh). Security updates aren't always done automatically, and if a system's implemented by a consultant and the owners of said system don't know how to operate it, then you wind up with a bunch of spoiled food across however many supermarkets.
The best systems are implemented either in-house or by a team of consultants working hand-in-glove with the in-house IT or ops teams, then maintained in-house. That's been my experience. Unfortunately your average municipal wastewater plant is probably run on decades-old systems by the same IT dept. that fixes the city council's printer.
Sorry, it was a busy day. I was agreeing with everyone else here. The move fast and break things attitude is one we can't afford when building control systems for critical infrastructure. I used to work in big tech, and if I pushed an update that didn't work or wasn't tested properly, the worst thing that could happen was we revert the change and there's a minor inconvenience to a user. If Facebook goes down for an hour, the company loses money, some advertisers get mad, and their users get frustrated. If an industrial Siemens SCADA operation goes down for an hour, the best case is the operation (be it a factory, airport, wastewater plant, etc.) shuts down for an hour. The worst case is people get sick or die.
It sounds dramatic but every update or change to the system rocks the boat and exposes it to a whole new tranche of dangers. So we spend most of our time observing. There's no such thing as a casual QoL update in SCADA.
I think for the big players, it often is, my impression is the more systemic risk is smaller players (like very local utilities) these are sometimes run by very unserious people.
Most SCADA systems date from the 1970s and a lot of them don’t really think about network security or even have a concept of it. So, they don’t think about the consequences of connecting it to the Internet for remote access. It’s just a feature they want.
Operators understand how important having them connect to PLCs and various bits because failure can mean you can take out parts of whatever it’s controlling, the power grid, oil pipelines, etc. So, you need to be very aware of what you are doing when you do these upgrades. There’s a lot of process to ensure you don’t break things.
But, you tell them about network security and or having to have Modbus messages wrapped in TLS and some of them will tell you to shut up and get out of their way. These systems have literally been just ported from the equivalent of a PDP/11 to Linux. So, not the most robust things when you come from a network security perspective. They really are 1970s systems and it’s really annoying, but I don’t think there’s any money in really doing anything about it and a lot of people make good money consulting around for these things.
I don't work with many of the smaller players, but you're right. They're far more likely to not have their own teams building and managing these systems, so they're susceptible to the Lyle Lanleys of the industry who come in and show them how they can control everything from their phone. They get sold off-the-shelf products and told that they're plug-and-play, but they're usually not.
That too ¯\_(ツ)_/¯ then you end up with a bunch of civil engineers who build a monitoring and control platform in Power BI and expose everything to the open internet, which then gets exploited, and they wind up working on a Siemens product anyway (I don't work for Siemens, I do custom development). It's really frustrating to watch.
Plus the lifecycle on these systems may be decades not weeks. At some point it needs to be segmented physically (more cost).
In 2017 I was hired at a place to support their ERP system and regular SysAdmin work. My first month was the last patch for their 10 year old ERP system. That was key software running the entire company and it was deployed before I finished middle school. The floor was run of scanners using WinME terminals that lost config on power loss.
That business is still profitable. I did not stay long.
Yeah the margins are tiny because private equity has spent the better part of the century destroying American dynamism and outsourcing all externalities including getting hacked to the people.
This is only partly true, historically before private equity was a thing grocery was always a razor thin margin business that operated on volume. PE just made it worse.
My in-laws had a family grocery store back in the 60s and 70s. Privately owned and operated, no way they'd have been paying any attention to security concerns like this.
Their only chance would have been if appliance companies wouldn't be dumb enough to put a simple refrigerator on the public internet in the first place.
As the IT Director at an independent grocer that's currently overhauling it's HVAC system (super fun for me to see Danfoss in the news this way) let me assure you that no one working in this space day-to-day has much of a clue wtf they're doing. When I inherited our systems, all the controllers were statics set within the subnet's DHCP lease pool which was shared with the office network at the time because, and I quote, previous IT guy thought VLANs were scary. Every couple of weeks a new DHCP lease would be assigned that would overlap with a microcontroller's static and kick the communications to something like the main AC governor off the network.
The guys deploying the new system ($2 million for a single location) are having HVAC schlubs terminate CAT5E and leave the gateway controller just hanging, unmounted to the wall or anything. Lo and behold, the RJ45 port has been broken and you have to send me a new gateway.
All the replacement controllers (for a system that is supposed to be better than Danfoss) are being set to 192.168.0.x. Each individual device, based on the "technician" descriptions are running individual DHCP servers inside them. And they just deploy them, no asking IT about approval or correct network config, just deploy and then make it our problem on a random day when they realize nothing is talking to each other.
They are just now in the late 2020s realizing what a VPS with an API is - truly, nothing to worry about with baby's first public IP address when baby doesn't understand what the implications of multiple DHCP servers on a single network might be.
The people "designing" and deploying these systems are HVAC guys with a little bit of software and networking knowledge. The networking (and security lmao) are the lowest priority part of what they're doing, they're more concerned with crimping pipe and moving onto the next project.
The people doing the project management and buying decisions are likely not IT at most operations, but the Facilities guys, and they'll slap a microcontroller anywhere until IT catches them or tells them they can't do what they want as easily as they'd like. There are no security reviews before deployment because the people heading the project don't even know what security is if it isn't petty theft and they don't want to hear that it will take longer than the contractor is promising them.
For additional context here, given everyone talking about the tightness of margins in retail grocery, until 6 months ago our network backbone switches in an environment where we need true on=prem servers to run operations, were still 10/100 Cisco Catalyst 2000s. Previous generations had fucked up the networking so hard with such little documentation and wacky misunderstandings that it took me several years to unfuck bits and pieces slowly and then schedule an actual maintenance window (because those mean potential lost sales, can't do that!) to cutover the rest to a more reasonable series of 1Gbps switches. We do mid 8 figures in a single location and I had to go through an entire song and dance to get budget approval for scheduling the 4 hours of OT for my sysadmin to do a maintenance window with me at midnight. Granted, we post mid 5 figures in actual net income most years, so yeah, margins are incredibly tight for smaller players and those of us actually trying to do a good job in any of the different sectors of the business are constantly trying to fix messes that wouldn't exist if there'd been better understanding or honesty about the long term costs of short term savings.
In any case, I'm sure there are more business segments out there like grocery HVAC where the networking is a simple means-to-an-end and the people deploying these networks barely understand the depths of the waters they're swimming in. There is absolutely a need to coordinate all the constituent subsystems to keep inventory stable until it sells, but yeah, if I'd even have been in the room during decision time with the contractors I'd have shot down any plan with external access without very precise deployment specs and plans. But I wasn't, I'm just the guy finding out some just certified HVAC chucklefuck is deploying DHCP bombs on my network 4 months into their contract and thinking they're doing me a favor. Without a doubt, a sufficiently informed and motivated attacker could knock out many SMBs that don't have a suitably knowledgeable IT department or god forbid an MSP charging them extortion rates for little real value.
Really love that I'm finding out our current HVAC system is getting exploited in France and at US Army bases on the first day of my vacation. Honestly think my only winning move here is to either buy a completely separate ISP ingress for the HVAC system or set a new subnet with only enough address space for the handful of microcontrollers needed for this new system. Time to start pushing on when the Danfoss is being retired and with how much gusto.
I’m curious, which system are you ripping out, and what are you replacing it with? I sell and run HVAC controls installs and do business with JCI, Siemens, Honeywell, Automated Logic, and Daikin. Sometimes we install Schneider Electric, Alerton, or Distech for independent dealers.
Doesn't sound all that convincing to me. There is no need for network controlled fridges in the first place. And it's not like setting up that network control was free. This is not about saving money, this is about ignoring what IT people preach since multiple decades now, spending more for the attack vector and then loosing a lot of money when that attack vector gets used. Margins? If it were about margins the problem wouldn't exist.
(I can see the need for remote monitoring, that the freezer actually worked. Controlling the temperature is the issue.)
Do you say that as a supermarket manager, or a random person on an Internet forum? Because if it's the first, I'm very interested in hearing more on how you actually manage all the devices in your store and which ones require remote automation and which ones don't.
They are all being sold on the idea of granting external access to a cloud application so that a third-party contractor can know when the system goes into disarray and pay for one less maintenance guy's salary.
Could the same thing be done with SMTP way more securely than opening up your firewall to some random AWS EC2 instance? Yeah no doubt.
Do you think grocery retail execs actually understand computers past Microsoft Word and Excel? I report to one, they can barely listen to me say the word WAN failover.
Like all things in IT, experience is the best teacher. Organizations will fail to invest in their technology, security included, and only then will the execs start to trust that the wizards in the backroom are telling them truths instead of asking for new toys - if they're lucky enough to still be the same exec that heard they should take security seriously before the Bad Thing happened.
In that case it's just a tradeoff between convenience and security, it isn't because it would be too technically difficult to isolate the systems. Whoever installed them could isolate them if there was no reason for them to be connected to the internet. I doubt most grocery store managers understand the precise engineering behind their freezers either, but they still buy them.
Very likely they were sold on these internet-connected systems without being told about the drawbacks, which is sad.
The people selling the systems often don't fully know the downsides themselves either and in a sense they're incentivized to not know as it makes it easier to sell the lie that there are no drawbacks to having your HVAC system exposed to the wider web. But you are also overestimating the average skill of an IT person in these environments and the capacity for their IT or MSP team to step back and evaluate the security of all the systems they have to manage. There's only the ability to evaluate convenience vs security if you actually have the time to formulate the question - if you're drowning in ticket submissions you will likely fail to properly catch every such issue. Certainly for these larger EU corps they should have the capacity to make those determinations, but who knows exactly how the management of these systems were delegated. My bet would be it was the responsibility of an under-resourced team with too much to do already.
I'm catching issues as I can as the head of IT at an org replacing our Danfoss systems with something else, but it wasn't until about 1.5 years since the initial decision to replace them that I'm finding out from a sub-contractor of the contractor that there's a cloud app our HVAC network will need to talk to - which is not-so-funnily enough a requirement by the state for us to receive associated grant funding for a portion of the new system cost. This was never mentioned to me at all as the head of IT. It's not even a convenience vs security issue, it's a misaligned corporate communications and approval channels issue. In other words, the capacity to secure your technical systems is downstream of political organization.
> They are all being sold on the idea of granting external access to a cloud application so that a third-party contractor can know when the system goes into disarray and pay for one less maintenance guy's salary.
Even in that case there is zero reason for the devices to be reachable from outside the network rather than the devices reaching out to known hosts only.
Controlling the defrost was the issue here. The only way a cooler or freezer gets to 120F is by running the defrost overnight. And networked communication for sequencing defrost cycles IS essential so you don't run out of compressor capacity and/or run up outrageous electric demand charges. With the same communications you can sequence compressors and evaporators to get more stable temperature control, longer product shelf life, and may cut down on the compressor capacity you have to buy up front.
Then there is refrigerant leak detection, and automated calls and remote troubleshooting for the refrigeration vendor.
But since we've already got off-site access, why not throw in temperature controls so you can implement demand-based setpoints, schedules, and so the corporate facilities and product quality teams can keep an eye on things. They are required to keep detailed logs of each refrigerated space after all.
Any vendor selling grocery equipment without network communication is going to have a hard time staying in business.
I know one German retailer from their storage warehouse point of view (not from the actual stores!). I made the operating system of fork lift terminals and visited quite a lot of their warehouses. And inside their warehouse there is zero connection possible to the internet, they are completely shielded. Their firewall ONLY allows connection to their (locally running) warehouse software.
A warehouse usually have LOTS of outdated equipment, security wise. Barcode label printers 15+ years without any firmware update. Voice dictation systems based on Windows XP. Things like that ... but as they closed down any contact to and from these systems to the outside, this is of little interest to them.
That's how we do it here, because we have decades old production machines running Windows XP and other vintage software. There's a completely separated production network, wires only, no WiFi, that doesn't have a connection to the office network, and is managed by a separate department.
The threat model of an insider is wildly different to an externally facing vulnerability. That doesn't mean there is no risk - but removing a class of attacker is an extremely useful outcome
I'm not network expert - but isn't this limited to the one case where somebody plugs in phone to charge from local computer at work, that is infected in specific way and can lockout the whole warehouse?9
I know somebody that works in HVAC. Two or three years ago they asked me about exposing the web-based controls for a meat processor's refrigeration system to the internet for remote administration. I advised them not to because it was an old janky-looking turd that hadn't been updated in something like a decade and suggested enlisting an IT person to stick a cheap OpenWRT router with Tailscale in front to protect it. Of course there was no money to do this so they asked what would happen if they did it anyway. The first thing I thought of was a disgruntled former employee turning the system off to spoil six figures of inventory and stink up the building. They paused for a moment and said "hmm... yeah, it sounds bad when you say it like that".
These aren't struggling grocery stores. They are massive businesses collectively running several thousand stores and employing hundreds of thousands of people. They have IT departments.
Revenue tells you scant little about the health and profitability of that business. Care to share profit and gross margin numbers? Grocery stores are notoriously low-margin.
Those numbers aren't available, but they're irrelevant to the point anyway.
The post I replied to (currently the top post on the article) is trying to paint a picture of some harried manager of a struggling grocery store having to fire up networking tools, which is ridiculous. A company doing tens of billions in revenue employs or contracts some of the "laptop class" (to use that post's terminology) to do that, regardless of what their profit margin is. They don't get a "shift-work labor class" employee to take a break from stacking shelves and check the firewall config.
I agree; you post some revenue numbers to try and prove a point that theses businesses have a lot of money. When someone reminds that revenue is not the same as profit, you state the numbers aren't relevant.
Companies try to cheap out all the time, in incredibly stupid ways. Especially when they are thin margins, fat ones usually try and protect the cash cow.
Revenue and profit are indeed different, and revenue is the relevant number here. If you need an explanation of why, perhaps someone else can furnish it. It's 2am here and I'm going to bed.
> Companies try to cheap out all the time, in incredibly stupid ways.
Yeah, and sometimes that results in someone turning their freezers into ovens. I don't think we disagree on this bit.
Having had a wonderful 4 hours of sleep, here's the free lesson I couldn't be bothered with last night:
Grocery store owner A's revenue last year was €500,000. Its profit was €0. It does not have an IT department because it's a corner shop that employs 2 people and has one computer.
Grocery store owner B's revenue last year was €50,000,000,000. Its profit was €0. It has an IT department because it's a huge fucking corporation with a thousand massive stores, multiple office blocks, 150,000 employees and all of the networking and IT infrastructure an operation of that size needs. It's considering opening a bank.
The number that tells you the difference between the scales of these two companies is the revenue, not the profit.
I need to stop wasting my time on the pit of ignorance this site has become.
I have worked at several different companies, of various sizes. One place sold equipment, 1 machine was between 100 ~ 250k, they usually sold 10 or so a month so revenue would be ~12 million. IT department was 1 guy, he eventually was able to get a helper.
Another place I worked at used to have IT on site at every location, they cut that as a cost savings. It became a small team at corporate HQ, then was outsourced after that. Regional offices and factories worldwide.
A story about their policies; They decided to ban USB drives in the name of security but there was equipment that you needed to get data off that could not be put on the network. Their solution was "Get a computer that we not controlled by IT, Login to Teams on that computer and use it to send the files to yourself."
This is a company that has over 5 billion dollars in revenue...
I think neither revenue nor profit can give you a good idea of what a company's IT ability is.
Myself, I have no idea why the fridges would be on the network/ internet but wouldn't be surprised that the utilities requested it be done.
I don’t have profits numbers but, as a customer, what I can tell you is that the large majority of those supermarkets are doing fine. They are renovated like brand new every couple years, they pop everywhere, they have money to innovate (E.Leclerc for example now have automated drive-throughs where your groceries arrive to you on a conveyor belt, serving dozens of clients in parallel directly to their car).
We also have a struggling supermarket chain (Auchan) and I assure you you can tell just by entering the supermarket that there isn’t a money issue.
Thin margin ? Sure, probably. But if there is enough money to build and develop brand new factorio-like contraptions, I’m sure they can handle fridge controllers well.
And yup, I worked for one of those three actors in their tech department and they are as much as tech-savvy than the other startup. And they have the same tech culture as your favorite startup : no real money issue and not a lot of care given as long as it is shipped.
I was talking to these guys when the faulty fridges got installed (early 2000's).
The IT director was usually a consultant with no say at the executive committee, let alone the board. Reporting to the "Secretary General", along with HR.
It took them a LOT of time to realize that digital was strategic.
I don't get it - it is not like the people working in a supermarket are expected to pay for securing the hardware in it. It is up to the owners who clearly have the money to run a supermarket chain not to keep insecure shit exposed to the Internet.
There really ought to be a market for a little device that goes between the IoT/industrial device and the rest of the world and that acts like a fancy, self-updating proxy.
Conceptually, you have a physical plant and a PLC and maybe an HMI. Quite sensibly, no one wants to modify this once it works. But you also have a remote access system, and all the remote part should be through a proxy that is continuously updated and only grants access to authorized users. And preferably logs everything and also can act as a sort of web application firewall.
Tailscale is fairly close, and they should consider selling a secured (attested boot, tamper resistant) device with two Ethernet ports (PoE on the uplink side) and WiFi client device capability. Stick a serial number on it, let it be added to a tailnet, and make it impossible for anyone to add anyone else’s device to their own tailnet.
BOM cost would be maybe $20 - we’re talking higher end ESP32-class hardware plus the Ethernet hardware. 100Mbps would be adequate.
Hmm, startup idea: anyone could sell this. The device has a little bootloader and two modes. In provisioning mode, it attests to the mothership (mothership is the vendor, but enterprise plans give an air gapped mode where the vendor can provide a signed config blob and the device can fetch it over TFTP, PXE-style) for configuration. In provisioned mode, it does what it’s configured to do, and it can attest to its firmware + config for purposes of authenticated to, say, Tailscale.
A conventional firewall requires either lots of complex configuration or installation on the protected device or both. I’m talking about a physical object that could be shipped to a facility and plugged in, and no one at the facility would need to deal with configuring it.
Ubiquiti cannot do what I’m describing. Frankly a plain firewall cannot do it securely.
I want the protected device to be on its own subnet with no plain IP packets passed through. It gets access through an authenticating proxy (speaking TCP and maybe UDP) and maybe through a VPN specific to that device.
Conventional network gear can fudge this by putting each protected device on its own VLAN. This does not scale. It’s also vulnerable to anyone who compromises basically any network device with access to a trunk port.
I would expect my device to be considerably more expensive per port than UniFi. (A VLAN capable UniFi switch port costs about $10.)
The cost of shipping a preconfigured Cisco firewall to some location, finding an appropriate place to put it, configuring it and maintaining it will be orders of magnitude higher in all respects than a bespoke device. And it still may not support the TCP-level proxy features that it really should have unless that’s all offloaded to some remote concentrator, which makes for either a not-so-fun HA configuration or an unfortunate single point of failure for access to the entire fleet. UniFi won’t be any better (you’ll be stuck setting up each node as its own “site” and you’ll be stuck with maintaining a bunch of sites on the ever-changing UI platform).
I'm literally suggesting that some company could make a little device designed to be inserted inline between an IoT or industrial device and the LAN/Internet that would, effectively, retrofit a security policy onto the device without interfering with its normal operation or requiring rework of the device. You're saying that plant operators could use an off-the-shelf firewall, too. I'm saying that using an off-the-self firewall is both weak and a pain to configure and that I think a bespoke device could do a much better job for quite a few reasons.
Uh, how do you expect configuring these policies would get any easier than literally the systems designed to configure those policies in the exact same situation?
Or be actually as secure?
I’m just pointing out these things already exist. You can buy and configure them right now.
At first you seemed to deny it, then it’s ’too hard’? Then ‘oh, but what about feature x or y’?
I find the case of having enough money to put all of that on internet and centralize management and not having enough of it to set up a rudimentary VPN unlikely. The money is not the reason.
Those systems are not cheap. It's not the case of lacking few hundred bucks to put it between a VPN box.
It is simply just "well, we weren't hacked before, why change?". Back when those were installed that was just not in the field of view and most those people worried about is changing default password.
And there is another case for industrial equipment, even if it is out of support, what you gonna do, trash a machine because of it ? No, it will be running till it falls apart or it will be more cost effective to replace.
> But I'd setup Tailscale and-- would you, the grocery store manager running on 3% margin, really? What if you're part of a chain and it's centrally controlled elsewhere, like all the other decade-old tech you use but have to call and wait for a dispatched tech to handle?
You either pay small cost now, or bigger cost when someone finds out. So, yes, really. We live in times where you no longer have to pay $500+ for enterprise grade VPN box, you can do very acceptable security wise solutions on cheap hardware or on literally "any old piece of junk"
That's part of the problem. By the time the fully analog original equipment needed replacement, the only commercial replacements available already had been built for network access for quite a number of years while none of the purchasing people were keeping up-to-date with it.
Even the less-massive models are so expensive that the vendors can't fail to include the latest technology whether anybody is using it yet or not, because future. These things don't fly off the shelf very often either and lots of analog leaders "just" used temporary consultants to bolt on some networking without ever gaining any unwanted digital headaches and accompanying expertise.
Along with the routers of the 1990's, inexperienced users were supposed to be able to just connect it and have it work.
Lots of them never connected it to the web or any other network, or maybe years later when there should have been orders of magnitude more security considerations they finally plugged in a long enough ethernet cable and it just worked as they always expected it to once the future got here.
It hasn't been that many decades, conventional (lack of) wisdom is it certainly shouldn't take more effort to install than an analog unit would have required.
I’d be willing to bet many more HN users work for companies who are the ones creating unnecessarily network connected smart devices than work in infrastructure, trades, retail-adjacent positions that implement/use many of these network connected devices.
I used to work for a large retailer in Europe. IT and software engineering was counted in thousands, I don't think setting up a VPN is beyond ability of any supermarket chain.
So what you're telling us is that there is tremendous opportunity for a startup to disrupt the space with new Refrigeration-as-a-Service (RaaS) products controlled by AI. That way we can replace those legacy microcontrollers with modern, secure GPUs and vibe-coded AI agents.
Thank you for saying this. It's quite frustrating to read some comments here that have obviously lost touch with the reality of the vast majority of people and businesses across the world.
So that’s the takeaway? “There is no money for this, so it just is what it is”? That’s insanity. So what now? Hope they don’t get hacked again, because by that logic there is still no money for upgrades and fixes, especially after losing a ton of money on spoiled inventory? But I guess that’s okay, only the “laptop class” actually cares. The supermarket lives in the real world with real challenges, none of that pesky security needed. I feel like I’m taking crazy pills here.
I would say the takeaway is that it cannot be expected that the majority of businesses hire an IT security expert (let alone multiple, + technicians) whose job it is to keep on top of CVEs and apply updates to a variety of embedded devices. IMO it must be a legal requirement, which is now coming (e.g. NIS2), but it will take time.
Target learned the security lesson the hard way, but they definitely learned it. When I was there, security was one of the rare parts of the tech org I found legitimately impressive. (But also: I am not a security guy! Maybe their actual practices were compelling but flawed! But the organization as a whole was legitimately serious about it )
I agree the average tech worker is woefully unaware of how tech illiterate the non-tech user is, but I would push back on the notion that basic security measures would've threatened the margins of a grocery store, and I am well aware of how they operate on single digit margins.
Tailscale pro pricing is $18 per user per month. Assuming you had one tech per store even, and 500 stores, that is a $9000 per month expense. Ok, not as bad as refrigerant and electric bill costs, but not great. So let's say instead of having a dedicated IT tech per store, we have 5 techs who work from HQ managing the infra for 500 stores.
$90. $90 per-month is all it would've taken to take these fridges off the public internet. If your margins are so razor thin that a monthly $90 bill would threaten profits, I question whether your business is even viable.
First you need to hire somebody who wants to do it. That costs more than $90 a month. Then they need to install whatever equipment is required to bridge the fridge controller to tailscale. Then finally you have a $90 a month cost.
I agree that corporate probably has the budget to do it properly. But the idea that the costs are just the monthly cost is Tailscale is laughable.
I can imagine that there's already an industrial security company at their door, offering a solution to prevent this from happening ever again, already tailored just for them.
Maybe it's even honestly not related to the attack.
For (just?) this fiscal. By saving on IT costs in previous fiscal years they were perhaps able to book profits.
One incident every (say) decade or so may be determined to be worth the trade of between the costs of 'proper' IT practices and getting pwned (and the resulting spoilage) so they can book profits for the other nine years.
There is more than just fiscal reverie at play here. Real food was lost. You can't eat money and there is not an infinite supply of food. Actually France with this year heat waves saw also devasting results on many crops. Plus farmers have been one of the higher suicide rate for quite some time now, and there are obviously structural reasons to that.
Mercosur ain't gonna save the population from starvation if shipping cost start to reflect actual environmental, energetic and geopolitical impact, as the trend seems to go.
If someone walks in and unplugs your refrigerator and causes you to lose all the stock in it, your insurance is going to cover you. You don’t need to show you locked the power sockets.
It depends if they entered by forcing through some security facilities or the door was full open with a white on red panel "the plug is over here" and a big arrow pointing to it.
I don't know how it is in Europe, but here in Canada, looking at grocery stores in isolation makes little sense when most of the market is controlled by a handful of vertically integrated conglomerates. On top of the stores, they own a lot of the distribution, wholesale operations, real estate, and even some suppliers. Their own financials show record revenue and profit, year after year, and their margins have increased significantly over the last few years too. We can't reduce the economics of such integrated corporate groups to their retail net margins alone to judge profitability. As far as Canada goes, small margins my ass.
> their margins have increased significantly over the last few years too.
Empire (Sobeys et al) has seen consistent profit margin of ~2% for the last 2 decades.
Loblaws and Metro have higher profit margins at ~3-4% attributable largely to their acquisition of pharmacy brands which function as high margin grocer/convenience storefronts.
This is a pet peeve of mine (as a Canadian): I'm not going to write an apologia for the oligarchic grocery system we have in Canada, but when I hear about headlines about "record profits" it's alway about the amount — which of course will go up if revenues are up because of inflation of inputs/costs. But I always ask about the margins: have those gone up? (They generally are pretty steady.)
What would help is not Interet-of-shittifying things that don't need to be.
There may be a sensible argument to connect these to the Internet in read-only mode (and from there, poor engineering choices leading to unintended ways to remotely control it). But there are a lot of devices where there is no good reason but everyone tries to cram WiFi and Apps just because they can or because they hope that they'll be able to abuse their users later to make an extra buck. At least those cases need to be discouraged (e.g. by making the companies liable and very publicly bankrupting a number of them).
I understand what you're saying, but these systems are introduced as cost savers. The problem is that "management" then absorbs these savings, and doesn't invest in keeping them up-to-date. It may be a cut-throat business, but that 3% margin is only for a very little bit depending on the cost of maintaining the fridge's controller. Now the costs may well exceed whatever they thought they'd saved.
Just latching on to this comment to add that it's probably a good thing (all things considered) this happened when it did - it's only one chain, so overall food supplies aren't impacted, and it happened during a relatively normal time period instead of during another crisis. The more hacks happen during peace time, the better prepared we'll be for not-peace time.
Exactly, and "Early reports indicate losses are already approaching 300,000 euros, but could run into several million euros nationwide." - IOW, the attack has cost far less money than it would have cost to prevent it with better security.
But the fridges are developed by corporate (laptop) workers, and the chains that buy them are also office workers. Nobody is arguing that an independent grocer or a shift manager get into cybersecurity, weak strawman
The reason why safety regulations are written in blood - is because no one wants to pay until the consequences become blatantly obvious and impossible to ignore.
It's not entirely clear which comments/criticisms you're responding to when pointing out the lack of margin. Whose excuse is that? And what's it an excuse for?
Security isn't optional just because margins are low. If eg FedEx decided to disregard security because they only made 3% of the value of the packages they ship in profit, they'd lose most of that business fast. But yes, let's not shame folks for failing to preempt theoretical-yet-unseen threats, if that's what you mean.
I think it's that people who've never faced a security threat and never had to think to are among the most vulnerable. And grocery stores are notoriously thin margin businesses who actually strive not to spend unnecessary the most.
Ok, then why were their damned fridges connected to a network then?
"The chain wanted to have Internet enabled control and dashboards about their fridges" or something?
You don't get to say you want the perks of being Internet enabled and then turn around and say that protecting against the downsides would cut into your margins too much. Sheesh
Imagine instead you had 100 fridges and 3 walk-in freezers and 5 industrial compressors. Might you not think, hey it would be great if one person could sit in an office and look at an internal dashboard instead of physically walking to each unit and looking at the temp and other gauges?
Now imagine you had 50 such stores.
MPLS? Multi-site Domain? VXLAN? Absolute fantasies, can't be done.
But seriously, networked fridge systems are one of the most straight-forwardly practical use cases for networking tech to improve logistical latency in a supply chain. There are definitely lots of different solutions for securing it depending on scale though.
> Might you not think, hey it would be great if one person could sit in an office and look at an internal dashboard instead of physically walking to each unit and looking at the temp and other gauges?
Sure I would. And my very next thought would be "how do we secure that"
See also: the McDonald’s ice cream machines by Taylor, their half assed and overpriced vendor locked in system, the raspberry pi based alternative, and then the ensuing legal and commercial chicanery
The profit motive absolutely favors neglect, and may even punish attempts at reasonable maintenance
Of course not. It is precisely because they have to fight for scraps that this is the place where workers are beaten down the most, because the owners are making pretty damn sure they will be the ones to get that margin lining their pockets, everyone else be damned.
In high-margin industries, you can give a 3% salary increase and free coffee and snacks, it is a rounding error compared to the 20% yoy increase applied to your multi-billion dollar compensation.
> Of course not. It is precisely because they have to fight for scraps that this is the place where workers are beaten down the most, because the owners are making pretty damn sure they will be the ones to get that margin lining their pockets, everyone else be damned.
My class-brained friend, margin is what you get after wages, not before.
I was talking about the supposed oppression of customers through high prices, but it’s typical that communists imagine that the purpose of an enterprise, private or public is to provide jobs to union workers rather than to actually useful products or services to customers.
Very little, which is why supermarket owners make everything they can to make sure they - and not their workers - get most of it. You still can't understand that the least lucrative industries are the ones where the owners and managers are the most cruel against the workers?
So if the profit margin is low, capitalism is bad cuz the workers are exploited..
... and if the profit margin is high, capitalism is bad cuz the workers are exploited?
got it!
Meanwhile under communism.. the workers are exploited everywhere and the profit margin is 0 everywhere, except the black market, and the black market is the only place you can find anything.
But you’re going to adopt a platform that meets the needs across your enterprise, so the capability is often there even if one situation may only call for monitoring.
Further, once you have hundreds or thousands of devices, “just call someone to turn it up/down/off” doesn’t scale.
Yeah, and people sewed clothes per hand for hundreds of years until someone automated it with a machine.
You need to save labor somewhere to free it up for other jobs that give you technological progress. If everyone would still be occupied plowing fields with oxen, we wouldn’t have cars and computers now.
>Why does it require a photograph of the employee's face every 3 minutes in order to determine if they're completing their work? Why does that make any sense at all?
AI image analysis, gives managers a Statuspage-like dashboard of labor engagement. Green across the board, boss.
It seems like a faceless attempt to prune back WFH like you said. Even if it wasn't, nothing activates organizational levels of manager antibodies like open defiance by workers. She was cooked.
Dropbox had basically no competition at the time as the idea of cloud files was barely in the market.
Meanwhile, home routers with meshing and security and built-in cloud management planes with easy apps are already cheap.
What would your pitch be when mom & dad would probably just stumble on Amazon Eero or Netgear Orbi? Someone already brought fire back to the home market, you can't differentiate if you're just putting up a site and a device, unless you hit some absurd low-$ cost in an already low-margin industry.
You'd be better off looking for a niche or industry where the edges are sharp, the tooling is bad or inconsistent, and there's no clear winner.
>Especially without offering any advice on how to improve things.
>And apparently it’s not even the point of the article.
It wasn't a pointless rant, you're seemingly primed for something that didn't happen here.
It's a good argument that passkeys are a trap while TOTPs offer the security you need but avoid the absolute control passkeys may hand over, so stick with those.
TOTP/SMS 2FA is a gigantic pain, I've probably lost multiple 24 hour days worth of time added up from entering TOTP and SMS 2FA codes dozens of times per day on websites that mandate it despite having nothing meaningful worth securing.
With passkeys I click once and I'm in like the golden age of using the web with a password manager in 2012. It's a huge quality of life improvement. If people want to avoid passkeys for whatever reason, that's fine but not interested in reading the same slippery slope argument that gets reposted every week.
A recurring issue with these critiques is how they seemingly presuppose passkeys are an all or nothing thing, and losing your passkey means losing access entirely. Which isn't how it works basically anywhere. It's just another auth method, if you can't use a passkey you fall back to whatever recovery method is offered. If passkeys became some user hostile trap overnight I'd just unenroll them and use TOTP, SMS 2FA, email 2FA, or whatever else is supported.
And every critic always attacks this same thing. TOTP isn’t so bad, my password manager can do it. 2FA is fine it’s automatically pulled from my phone.
It’s still phishable. The password is still sent. You haven’t solved anything, you just piled more complexity on top to make it harder. People can still use the same password at 12 websites.
Passkeys never get transmitted. If you hack into a server and steal my credentials, all you get is a public key. You can’t use it at another website. A bad developer can store my password or TOTP secret in a way that makes it useful to a thief. That’s not possible with Passkeys.
No one ever offers an alternative other than saying it’s irrelevant. Despite practical experience showing otherwise.
Passkeys work and make my life better. These constant rants remind me of seeing people argue that the GUIs are a waste of time and no one will ever use them and it’s a fad that will go away and the only reason everyone’s talking about them is companies want to sell graphics cards and mice.
Not the concern, it's whether it's worth the time and energy to expand passkeys as the default and getting normies using them. Some people will always surrender and say they have nothing to hide, so be it. I'm more thinking about the practical rough edges that user bases will cut themselves on once you expand past the extremely-online.
From my experience, it's a nonstarter. The expectation that normies will understand and maintain a password manager layer to store their digital keys in and understand that concepts end such as sharing passwords or logging in on a new device without fully bootstrapping the password manager is just too far.
I think sometime after Q-Day when the dust settles and we've had to reissue all the keys today, we'll have a better answer on whether the specter of remote attestation is truly manifesting and in any event I bet there'll probably be an updated passkey concept by then.
Passkeys work because you always have the hardware for them.
Suppose I had only passkeys and Google Pay. I take a train into town and lose my phone. What then? I'm screwed.
I couldn't just go home because I'd need to buy a ticket or taxi using my phone. I couldn't buy a replacement phone. I wouldn't be able to online shop on a library computer or withdraw money unless I have a physical bank branch. (I do not.) I can't even contact anyone unless I know their number and borrow a phone or find a way to send an email.
My preferred balance of risk vs convenience is: passwords in a password manager, hardware-based 2FA only for making account changes, and confirmation emails (My email password is not in my password vault.) any time a site successfully or unsuccessfully processes a login attempt.
There is no hardware element. If someone phishes my password, I'll know right away from the login email, unless they're so talented or lucky that they also accessed my email or found a way to spoof my 2FA. If I lose my phone, I'm not totally helpless. I don't see an overall downside to my strategy that isn't worse than passkeys.
-----
Edit: One suggested alternative I can fathom is "passkeys managed by password manager". While this does provide better security than "passwords managed in a password manager", my experience has been that passkeys are not well-implemented across all platforms where they Just Work(TM) on every site. I don't know how you'd go about using passkeys in a shared public computer setting (university lab or library) unless you can install browser extensions (and why would you want to, even if you could?). At the end of the day, these passkeys are still secured by the password manager's single master password. Regardless of what I use, I want emails when a login attempt succeeds or fails. It takes one full second to swipe away at my convenience, and I would instantly become alert to an attempt or login I didn't execute.
Taking care of them was historically about calorie and nutrientmaxxing. Subsidized calories have worked too well, worldwide, and there's no response to it within the current Overton window.
reply