Is this something one would notice if they don't have problems with OLED phones? I was a bit worried reading this...and then realized I was reading it on a Pixel 8 on low brightness. Did a search and found reddit comments like: "Google pixel 8 is the worst when it comes to pwm. It has a frequency of 240 hertz which is the lowest in the industry." [1] I haven't noticed unusual eyestrain using this phone, so I'm hoping a new Macbook Pro screen also would be fine, despite its larger size.
Back in the CRT days, it always bugged me when a display's refresh rate was down at 60 Hz, but otherwise I think I'm not too flicker-sensitive.
The article is not about abandoning ssh (client) keys but protecting them from exfiltration:
> This post documents how I’m using various secure enclaves (YubiKey, TPM, Apple Secure Enclave) to hold/encrypt SSH key material and achieve the goal of not having any unprotected key material living on-disk.
Could you help me out and state the goal of your practices as simply as this? I can guess, but I'm really not sure, especially when you get into nftables checks for window sizes.
My method reduces the threat of exfiltrated keys when someone or something manages to bypass the aforementioned methods without the need to purchase anything or use anything proprietary.
- One must connect from the correct address and with the correct TCP options.
So for example I could give you one of my ssh private keys and it would be entirely useless unless you were on my home internet connection using a device that emits my TCP options. (Or this could be a corporate SNAT pool or corporate VPN pool.).
Even the most clever MitM (man in the middle) attacks become significantly harder to perform. Leaks happen, I reduce the blast radius of the damage.
So if an attacker compromises your private keys, they’re already on your computer. If they’re on your computer, what stops them from connecting to your server from your computer? It would have the exact same TCP options, same IP address, same everything.
If I were an attacker that had an agent or RAT on your PC and got your private key, I’d first try to connect to your server on my computer. That would fail, but the very next thing I would do is use my RAT or agent to connect to the server via your own PC, because I would assume you had some sort of IP filtering in place.
If you store your keys in a trusted enclave like a Yubikey, TPM, etc it stops this cold (configured correctly).
So if an attacker compromises your private keys, they’re already on your computer.
They could be but I would not automatically assume that. Many stealers upload keys and do not install a RAT and there is no guarantee the RAT will function even if they manage to install one. If we want to go down every what if path I could make everyone want to power off their data-center and go home. If I showed everyone what SSH can do without any RAT's many would uninstall it. Involves multiplexing and one phishing victim to bypass MFA, HSM's, vaults, bastions, and does not leave a trace by design. No need to sneak past anti-virus.
Most orgs will not buy or implement an HSM so I am offering a soft middle ground that may suffice for some and may not even remotely be an option for others. Some orgs have no choice but to buy an HSM. Been there, done that. It can be a massive daunting task depending on what checkboxes are not yet checked and what requirements B2B financial customers are demanding. One of them required 6 different colored keys that get distributed to different org leaders and have to be put in the HSM to make any changes. They put the fear of diety into everyone about losing keys. One was lost. Thats when you find out that crazy expensive device is just hype.
Oh and ask me how many HSM keys I have seen pasted into Slack.
In security threat modeling, you only factor in things that you control when determining exploitability or risk. So for instance, when you say “a RAT might not work right” or “attackers don’t always use RATs” those aren’t things that you control, those are things that an attacker controls.
One of the principles of threat modeling is that you must assume an attacker is working actively to maximize their odds of success. Saying something like “well it would be annoying for an attacker to do this” is practically universally rejected by cybersecurity professionals especially in the age of AI agents.
I disagree that I am posing a bunch of “what-ifs”. The threat I am modeling here is how companies get compromised all of the time. Living off the land attacks as I’ve described is how companies large and small get hacked.
Furthermore, you do not need an expensive HSM. Most computers nowadays have secure enclaves (TPM) that can be used for this purpose.
Run a poll on HN some time and ask how many people are using an HSM. Out of the 17k visitors (or so on a given day) I would be surprised if it is over 500. That's ~17k not counting bots. Counting bots it's closer to 50k.
I suspect what you will find is that a majority of people are using SSH keys and a very relaxed firewall policy. A smaller number are using a bastion with some form of MFA + LDAP (if that) but it's all moot if the bastion or data-center servers allow multiplexing thus allowing log-less bypass of MFA, LDAP, HSM, etc...
A growing number are at least using a corporate VPN however more often than not those devices have a myriad of 0-days and are very poorly engineered over priced proprietary garbage full of state back-doors so at least buy the hardware made in your country so it may be your intelligence agencies that have access and a small sub-set of the MSS PRC.
Every Apple computer has a built-in hardware enclave. Just download Secretive, or you can install my macOS app, KeyMux, from the App Store, which supports SSH key agent protocol, GPG agent protocol, and PKCS#11. Or use the native command-line method discussed in the article.
That's not very economical. A YubiKey sets me back €50 (already twice as expensive as a smartcard). A MacBook Air sets me back €1299 and requires a €99 dongle for USB.
You generally don’t need to buy a dedicated HSM. TPMs are built into a lot of modern motherboards nowadays. A lot of folks are probably unaware they have access to one.
I enjoy listening to Michael Malice debate people mostly because he is a proud open troll. Sadly the numbers of people like him are dwindling but perhaps he will inspire people to come out of their shell.
Everyone is free to choose what options work best for their needs and budgets. I would never attempt to inhibit anyone's choices. I use what has worked well for me and stifled red-team members.
Yes and you can change the protocol field in your TCP stack to run all connections over ip protocol 200 too, but it doesn’t add any security to your authentication system over and above your private key. What you are doing is the same thing; lots of extra too-clever-by-half complexity that doesn’t actually benefit you.
Key agreement protocols make sniffing the wire for credentials useless. Two seconds in wireshark and your hyper secure tcp options are toast.
I do not agree. I have used this exact method to detect all manor of shenanigans within VPS environments, dedicated server environments and corporate environments. Attackers stumble, fall and make a lot of noise trying to untangle my simple mitigations. I will keep it.
Correct. Thankfully none of what I am doing is security through obscurity. It's actual security with a subtle scent of obscurity. If it was just obfuscation and you knew something, then you could do that something from somewhere else which you can not.
That's not security by obscurity, that's the Swiss cheese approach; which people attracted by purity frown upon but people attracted by practical solutions can live by.
They're color-correcting the board pre-display, and then presumably a second time from the camera's feed (of the full newsroom scene including the display)? Why? It seems at best unnecessary.
I would expect they'd color-correct the board feed in the control room when it goes direct to broadcast rather than through a camera, but I don't get why they'd ever feed that back into the display.
> Why is the input latency so bad on the display? The touch/draw visibly lags behind by 500ms or more, is that an issue with the software or with the hardware?
One likely factor: the video we're watching is from 2016. I think input latency has gotten more attention (optimization effort) since then.
I'm speaking about devices in general. Haven't been getting my election coverage in video form, so I'll your word this board hasn't improved. They may have just stopped investing in it rather than take advantage of hardware and tooling improvements since 2016...
Debian Trixie is supposed to remove the "emSign ECC Root CA - G3" and "Comodo AAA Services root" in ca-certificates 20250419+deb13u1. [1]
I don't see that update available after `sudo apt update && sudo apt upgrade`, and one of the `curl` commands from this thread still succeeds. How long does these typically take from notice to actually being available on whatever mirror one hits?
In the meantime I ran `sudo dpkg-reconfigure ca-certificates` and unchecked those two particular items. That worked.
edit: doh, it was an own goal. I had a `/etc/apt/preferences` that was causing it to prefer `stable` over `stable-security`. facepalm. not sure how I ended up with that, but clearly I did something dumb. should anyone else be wondering, `apt-cache policy ca-certificates` is helpful.
tl;dr: MCUs are space-constrained and often don't include root certs that aren't used by the one host they talk to.
In my experience it's equally likely they just don't do validation to begin with. For example, my Elk M1XEP (physical security system's Ethernet expander) runs a configuration server with and without TLS. They recommend with, but it uses a self-signed certificate, and the crappy VB.NET configuration program doesn't validate the cert in any way, so it can be trivially MITMed. The Elk M1XEP also can connect to a user-chosen alarm monitoring service ("central station") over the Internet. I don't think it uses TLS at all for this. I think there's optional shared secret auth/encryption?
Over a decade ago, I created an AWS ELB to serve a production API. It was HTTPS-only, using an Amazon-signed certificate served from ACM for my employer's domain. Right after deployment, logs showed a regular pattern of 404s trying to POST/PUT to my API at paths I didn't support. Thousands of requests per hour, and they continued for days. It didn't look like a DoS--the paths looked like someone else's API.
I got curious what all the traffic was, so I added a route to the API that saved some of the uploaded data. It was live home surveillance video and photos! Indoors, outdoors, baby monitors, living rooms, etc. It was pretty easy to find which vendor's hardware was doing this (from headers, paths, watermarks) and I reached out to their security contact to let them know what I found. They responded generically but didn't seem to care that much.
In short, because HTTPS was required, and an ELB gets assigned IPv4 addresses arbitrarily, I can only conclude that a whole fleet of home security cameras were sold across the US that hard-coded the IP address of their cloud server and had certificate validation disabled.
This is unfortunately accurate, sometimes due to real constraints and more often due to loose threat models. I am hopeful that more open source firmware options (e.g. Zephyr RTOS) and more software investment from silicon vendors is helping to change the status quo. That being said, we are still a long way away from a "Let's Encrypt movement" for embedded devices.
IMO this should always be pushed back against, because it's not a valid excuse. If the hardware is too constrained to support appropriate security then that hardware is not appropriate for the application (or no longer appropriate for the application, as the story often goes).
Unless you plan to assign your Ethernet expander a domain name, how do you expect it to participate in WebPKI? The vendor similarly has no interest in explaining its buyers how to do that so it can connect "securely" to their particular alarm monitoring service.
There are some goodish options for how we could secure communications with local network devices hosting simple configuration WebUIs, but the browser vendors have overwhelmingly demonstrated that they have no interest in any of them. So what you ship is HTTP, not even the self-signed certificate, that just gets scary messages.
(What the browser vendors seem to want, to be clear, is that your gadget reaches out securely to the Vendor Cloud^TM, and you similarly use your Google and Apple notarized smartphone to reach out to that same Cloud, so you can talk to the hardware sitting a meter away from you)
> Unless you plan to assign your Ethernet expander a domain name, how do you expect it to participate in WebPKI?
Well, it does support DDNS, [1] so presumably some people do assign it a domain name. But I don't expect it to participate in WebPKI necessarily. That's not the only way to secure communication from the config program to the device, e.g.:
* The routers I worked on in 2006 had baked-in keys/certs signed by our own CA with cn based on serial number rather than reachable customer-chosen FQDN. That's not WebPKI but it actually works. Elk's configuration software knows the expected serial number and could use a similar approach. The configuration is not http-based (much less browser-based) anyway.
* It'd be better to just punt and say the communication should be secured by proxy or VPN or something than to offer TLS in a misleading way and suggest it should be directly exposed to the Internet.
As for the Ethernet expander to the central station, that totally could use TLS and WebPKI.
> Edit: I might also be misinterpreting what I am seeing and it's actually weak logic that created the keys, in the sense that one could guess the private key?
As I understand it, the private key is supposed to be a number in the range [0, 2^256), but they actually generated a number in [0, 256) for these 8 intermediate certificates. Once you know there are so few possibilities, you can try them all ~instantly, so effectively everyone has the private keys. (And when I say "you can", I actually mean you, not some hypothetical more skilled version of you: even if you don't know anything about public key infrastructure, if you have access to an LLM, it can do it for you with a single prompt.)
This is the second unikernel article this week I think, and the second that really undersells the benefits of having a full OS. Let's just swap out a couple of words:
> With a unikernel, the [observability] surface is much smaller. If the functionality isn't in the application (the operating system), then the [SRE] is screwed. There's no next hop.
Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value. The article mentions some nice things for observability ("structured logging, OTel") but so much gets thrown away.
Not just that. Unikernel sounds ok if all your linux server does is run one service that doesn't need anything but tcp/ip, and whose users don't need extra resources. But being able to deploy a new binary and other files through any means (ssh, pipelines, git), having localized log files, etc., are more than a convenience. Sure, I could log on another server, run a db on another server, and deploy html/css/js from yet another server, but would that make life easier? And this is just a really simple case. If you have multiple services, you'll need multiple servers with a uni-kernel approach, or build some monster thing. It's horses for courses.
Welcome to cloud computing, where software is cattle, not pets.
There isn't a Linux server taking care of a couple of services.
There are services deployed as Kubernetes pods, or serverless language runtimes across a cluster abstraction, running directly on top of type 1 hypervisors, or minimal kernel images enough to power distroless container images.
maybe we _should_ be thinking about building proper distributed operating systems instead of loose collections of services. we really missed an opportunity to rethink the OS interface for 'the cloud'. or rather we kind of wasted alot time and maybe we should get our shit together and do it properly. no matter how much we chant 'cattle not pets' everyone is still installing individual nodes and logging in and puttering around.
Agreed! There were many promising distributed operating systems, most got stuck at the research phase however. But the cloud pretty much killed them all.
Not that many people keep large distributed workloads anymore, and SLURM-like schedulers on top of regular operating systems were deemed good enough for supercomputers.
not always. Asci-red had a distributed version of OSF/1. this wasn't a great design because it was built on distributed shared memory, so it had fault tolerance and performance problems. the MTA had a shared memory monolithic BSD, but that was almost a forced choice.
The Cray T3D had a really nice distributed services model (really kind of a distributed exokernel) that would be worth emulating
on some machines there was an interactive ganged run facility like 'mpirun'. I think even this distinction where there are 'login' or 'service' nodes and anonymous runners would be at least a little structure. I guess you could argue that k8s is this, but it kind of falls short and seems to require a lot of continuous propping up.
I worked on a single system image project that didn't ever get shipped. it turns out that a syscall proxy layer isn't that difficult to implement, and that leaves you with an undisturbed user space.
anyways, there are a whole host of design options that don't look like k8s or slurm. and having spent some time recently on non-hyperscaler clusters, that bottom level of bringup (provisioning, networking management, reporting, fault tolerance, distributed filesystems) is really a nightmare. I think we lost a lot of traction there by raising generations exclusively in someone else's operational context.
Yep. We say cattle not pets, but even cattle often need personal attention for one reason or another. It doesn’t mean you’ll never need that option in your back pocket
if you didn't have a multi-user operating system running on every node that basically demands this, because of its complexity and fragility, and you had the necessary introspection to look at memory usage and open sockets, and disk bandwidth, and everything else you might need to mange those processes, then the utility of your escape hatch - which also is complicated and fragile, becomes negative.
Let me know when your local OS does multi-machine scheduling across heterogeneous hardware and application constraints, networking, observability, service accounts and a finer grained permissions system, in a way that’s far less confusing and piecemeal.
Also, this argument only works if you just broaden the definition of “OS” to “anything that runs that isn’t your apps”.
In general, the more specific tool you have, the more tailored to the goal approach you may have, with all features optimized for that goal. If you need those tools from the full OS, you might have their specialized versions in a unikernel.
Take this with a grain of salt. I haven't run unikernels in prod, so it's all a figment of my imagination.
> Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value.
But doesn't their value lie in getting you access to the core of the OS that you wouldn't be able to touch otherwise? What if your network stack was a library you could instrument just like you do with the rest of your app code?
Also, I don't need a tool to tell me I'm being punished by the IO or CPU scheduler if I am the only one using the CPU and IO :)
> But doesn't their value lie in getting you access to the core of the OS that you wouldn't be able to touch otherwise? What if your network stack was a library you could instrument just like you do with the rest of your app code?
Yes and no. The traditional separate kernel exposes one set of interfaces that these tools use. A library version might expose a different set of interfaces that accomplish the same thing. Your application could then use those to provide similar observability. But maybe not—that consideration was mostly absent from this article. And with new tools, you need to retrain your developers, SREs, LLM agents, etc. If your new tools are that much better, it might be worth it. But these articles have fallen far short of making that claim.
> Also, I don't need a tool to tell me I'm being punished by the IO or CPU scheduler if I am the only one using the CPU and IO :)
Disagree. Embedded programs still have IO and CPU scheduling problems, including priority inversions. You don't need different programs to have CPU scheduling problems, just different threads, or different tasks with an async scheduler. Any time you have concurrency you need to consider scheduler observability.
It's also not like you are stuck with a full OS when you are running Linux. You can remove features from the kernel you don't need, and you can configure a bare-bones busybox to be your userspace, or just run your application directly as the init process.
In a type 1 hypervisor running a JVM on top, you would use Java Flight Recorder.
The tools exist, and in a container world no one should be running those tools directly anyway, it is all about rootless immutable containers, ideally without OS layers for deployment.
If your app, is the only one running, and the whole kernel is a library, you can simply have said library instrumented with as much observability as you need, directly and you now no longer need to worry about which processes are using cpu and IO because…it’s all you.
> Tools such as `strace`, eBPF, `lsof`, `ss`, etc. have tremendous value. The article mentions some nice things for observability ("structured logging, OTel") but so much gets thrown away.
Trivial to implement. I have a full custom (from scratch written) debugging/benchmarking implementation written both for Linux and two of my custom kernels, both of which are REALLY unique when it comes to architecture. My custom debugger also supports numerous inline scripting languages.
I'd love to read more about your debuggable/benchmarkable/observable unikernel architecture and interfaces. Much more interesting than these "unikernels are the future" articles which make little reassurance they've replaced what they've dropped.
I wonder if they ever considered a table-driven approach for `#[derive(Debug)]`, as `facet` [1] does. That would have been my first instinct for something this formulaic where binary size and compilation time matter more than execution speed. But my impression is facet hasn't quite realized its promise on those fronts, so maybe the table-driven approach in std was similarly tried and rejected.
You're never going to get promoted with that attitude!
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
Back in the CRT days, it always bugged me when a display's refresh rate was down at 60 Hz, but otherwise I think I'm not too flicker-sensitive.
[1] https://www.reddit.com/r/PWM_Sensitive/comments/1f9j3fi/pixe...
reply