It's always an interesting thought that I had meetings about the log aggregation dropping stack traces that would take ~3 3.5" floppies to store. Each.
Zabbix also confuses operators at this point. I recently had to increase a cache. I chose to go from a solid 4MB to a luxurious 256MB.
I know high-performance Java, so just a little jab there, but you don't size JVM parameters like that, except maybe code caches. And before Rust comes in, we also have a Rust application running without good resource control and it wouldn't like those resource constraints.
4-5 java based microservices, all of which logged their own stacktrace, and also included the remote stacktrace in their local stacktrace... which was in turned included in other remote stacktraces.
Add in all of the usual unpruned Java middleware calls, multiply by 4+ and you end up with oodles of things.
Out of which the dev needed the first 5 and the innermost 5 lines. Croak & Carp did that better. But I'm old.
There are a few guns that exist that could achieve the necessary velocity (just shy of 4,000 mph) but they’re not handheld. But yes, it would be possible for the right gun to fire a bullet that went around the moon and hit you in the back of the head.
Funeral for the Sun also came out recently. I'd call it more guess-y than Obra Dinn or the Golden Idol, but also very fun.
Other smaller games in this niche would be "Death at Fleming Manor" (it's a bit laboursome to play) and "The Seance at Blake Manor" (Took me a bit to get used to, but honestly very interesting), if we're collecting. Which we should. Maybe I need a blog.
I can very much recommend looking at games developed and published by Wadjet Eye Games.
You can see the learning throughout the years, with the Blackwell Chronicles being a bit linear, and some games in between being a bit of a miss.
But "Unavowed" or the new "Old Skies" are excellent games I consider on par with the old golden age. "Technobabylon", "Resonance" and "The Excavation of Hob's Barrow" are also very far up.
For some other smaller devs: "Elroy and the Aliens" was also a really good one. "Kathy Rain" also came out as a directors cut recently. There are some fun changes in there. "An English Haunting" was also fun. "If Fusion were that easy" was also quite neat.
There is a bunch of cool adventure games in the old style coming out recently. It's just hard to find them because "Adventure" and "Point & Click" are just such overloaded tags on Steam. Pretty much everything is tagged as "Adventure"
I always recommend to use weird numbers for different tasks. That way, you can see "oh my system is under severe load at *:13 (or 17, 23, 42, ...). That must be this cron job"
I do a similar thing with ping payload sizes to be able to fingerprint the traffic: I’ll use prime numbered packet sizes so that they can be detected even when the packet sizes are summed.
Btw, one of those numbers is not like the others (has more prime factors that evenly divide 60).
I don't think a UAF on its own allows control hijacking in an exploit. You will need further logic bugs or the ability to legitimately or illegitimately control memory, either via further requests / function calls, OOB writes and such. And it will always be extremely specific to the application vulnerable.
For example, I remember an exploit workshop around reuse of passwd entries. The vulnerable would validate a passwd entry for a non-root user, use it a bit, and later on free and zero it out for security purposes... except it then looked at the passwd entry again in some conditions. Suddenly, the uid was non-zero at first, and zero later on, so later decisions assumed the user was root.
In a more concurrent situation, a possible vulnerability was a shared cache of these passwd entries. It wouldn't zero them out anymore, but you could create a race condition of starting requests as user A, and then hammering requests as user B, so you'd end up mixing up the permission contexts of these users. Because the passwd entry that had been released back to the pool was overwritten, and then re-read with the value from another context.
The necessary line of thinking tends to be very different from thinking about program code regularly.
And even with a mutual agreement - default LLM output tends to be very hard and inefficient to read. We've spent quite a bit of time at work to setup a few writing skills for Claude.
For example, mine is a technical writing skill based on the old book "The Art of Scientific Writing": Front load information as much as possible - the first paragraph of a document should be a summary, the first sentence of a paragraph should summarize. Short sentences building on each other. Always show evidence.
Someone else spent quite a while optimizing a standardized PR description template, with a section for intent, expected changes, risk, rollout coordination and such.
And strictly speaking you cannot trust the pip upgrade of pip with the vulnerable cert in the trust store due to certifi, which you also cannot upgrade safely with pip for the same reason. All of these are at one out of two compromises from being MITM'd during the upgrade.
So strictly we have to burn everything python to the ground and reinstall entirely.
Heh. This is part of why I always preferred getting my python runtime and all these core libs from my linux distro packager. But python die-hards look at me like I'm insane when I suggest this.
Even in today's LLM-addled world, I still think it's a better tradeoff for smaller system/site operators. Get the distro-level testing and regular auto-patching, rather than telling yourself it is a good idea to "freeze" your python world at the pace of your slowest app/system integrator.
It's the same reason I think docker is an anti-pattern for these small groups. You really cannot safely use these strategies to bake an image and divorce yourself from the maintained distro layer unless you are going to adopt CI/CD that does the same thing. Keep tracking all these patches to the entire runtime with all the dependencies, rebuild, re-test, and redeploy.
It's total cargo culting to adopt the freeze/image techniques and just abandon apps in place as if that is robust!
But, it's also more than just automatically updating everything. There is an editorial aspect to distro management to help secure the supply chain to your app. Most smaller orgs cannot feasibly do this as well as the regular linux distro maintainers. Some large orgs can probably do it better, but that doesn't matter if you aren't doing it large org style with dedicated security folks tracking all this against well-developed policies and procedures...
> But python die-hards look at me like I'm insane when I suggest this.
They have a point. The distro is only carrying a small fraction of what's available on PyPI, and will have out-of-date versions that may lack security patches (and for projects like yt-dlp, are highly unlikely to work at all). Realistically they aren't applying security backports; there would rarely be a reason not to just use the new version, and the resources just aren't there for that scale of patching anyway.
Further, pip will interfere with your system package manager's tracking of the system environment (PEP 668), and any attempt you make to install from .deb/.rpm/etc. to a virtual environment is a hack at best. Maybe you could symlink individual package folders to the system environment while installing a superset, but ugh. Oh, and if you're a developer (like most "Python die-hards" are) then this isn't very helpful, as you aren't testing the installation of your package the way that your users will do it.
The mitigation is to ignore nearly all that stuff in PyPI, realizing it is barely a step up from copy-pasting out of stack overflow. Same goes for, e.g. python-whatever RPMs that look like they were drive-by contributions.
So your trusted base is the Python runtime and maybe a few choice libraries that actually have engaged package management. Sadly, it is a trap to think you can put things on the internet as a small player by leveraging lots of third-party stuff, when you don't have the resources to actually maintain that large dependency base.
This also may mean you want a base OS like Fedora and not some crusty "enterprise" variant that tries to bundle a frozen Python from 15 years ago. This is getting a little better in recent years, with some enterprise platforms having optional package streams for parallel install of newer runtime versions, etc.
Edit to add: on the Fedora-derived platforms, pip doesn't really interfere much anymore. The platform python-* RPMs go into /usr/lib/... while pip installs into /usr/local/lib/... or your ~/.local/lib/... hierarchy. The only mysterious behavior you get is if you manage to install in both places and then you have to understand the path search order. Just like if you do strange things with user and virtual environments.
Eh, it's a personal choice and a not unreasonable position to take, but there are a lot of things on PyPi that aren't packaged by e.g. Debian. Database clients, immensely popular frameworks, scientific computing libraries ...
Basically once you leave the realm of the 5-10 most popular building blocks in an area (which OS packagers do distribute), you're still left with plenty of packages that have 10ks+ of users, sane maintainer/security/ownership models, and code that would be very difficult to just copy-paste a la stackoverflow (e.g. native/compiled deps with complex wheel build tooling).
Yes, that's why I left wiggle room for a few choice packages you track yourself.
What isn't really tenable for a small team is to drag in scores to hundreds of different third-party packages and rely on them all when you have no staff allocated to really evaluate and monitor so much upstream stuff.
I agree about getting libs (binaries especially) from my distro, and it generally seems to be regarded as the "wrong thing to do". Freezing anything that might mean you get security updates slower seems to be a mistake to me.
Or - you can just upgrade pip with a SHA hash pin, or set the magic env vars to use the system bundle.
You do need to be careful about creating new venvs with the old version of pip bundled in ensurepip in older Python releases. `uv` kindly patches that to newest pip, but not every Python binary does.
I had a pretty fun time with it myself. Rather than a system package for pip, or the `ensurepip`-provided one, I've been using the copy managed by pipx. I had to:
* Update system certificates
* Locate that pip and directly invoke it to self-update with hash pinning, rather than going through pipx
* Fix a few stray venvs that contained their own pip
* Use the fixed pipx to update uv (although I guess the uv self-update should work even for a pipx-installed copy?)
The legacy-certs hint is neat. I did it the first way but would rather not have since I'm on Mint.
Yep! I tested it with my own vibe-coded MITM proxy that hijacks pypi installations, which I created for ~$0.30 spend on DeepSeek API.
$ python -m pip --no-cache install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
Collecting pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
Downloading pip-26.2.2-py3-none-any.whl (1.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 570.0 MB/s 0:00:00
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70 from https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70:
Expected sha256 fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
Got 8c448177668e735cdde7d6178694b90c084ba18559348cc394c4b1546eeec89c
The second one works too:
$ PIP_USE_DEPRECATED=legacy-certs PIP_CERT=/etc/ssl/certs/ca-certificates.crt python -m pip install --no-cache --upgrade "pip>=26.2.2"
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082) - skipping
ERROR: Could not find a version that satisfies the requirement pip>=26.2.2 (from versions: none)
ERROR: No matching distribution found for pip>=26.2.2
As compared with a naive installation through the proxy:
Oh nice. Then I have to thank you for a possible non-horrible upgrade path. If you were close by, I'd invite you for some icecream or a round of pool :)
Yep. Per the main thread on the Python forums (https://discuss.python.org/t/information-for-python-users-re...), this now happens even with the outdated pip because the built-in `ssl` module has been patched to reject it. The relevant commit is https://github.com/python/cpython/commit/7dd8b4736a1419750b9... , featuring a rather amusing workaround for security scanners that might mistake the blacklisted certificates (included for testing purposes) for "leaked" certificates accidentally included in a source package.
From code comments, because I found it interesting:
* This is an extraordinary measure and last line of defense. Removing trust is
* primarily the job of the platform root store. This list only blocks keys while
* that update is still propagating to the OS, container images, or root CAs
* bundled in Python packages. The threshold for adding an entry is very high.
*
* See https://github.com/python/cpython/issues/160040 for the eMudhra discussion.
As a Mint 22.x user, I wonder if and when Debian or Ubuntu will pick up the change and release another patch for the system Python (basically, to my understanding, a base of 3.12.3 with security patches from other patch releases backported). Since they've already updated ca-certificates it might not seem necessary, at least as long as the pip package gets updated (although I don't use that anyway).
BTW, I think I saw your comments in other places about the 30-cent vibe-coded MITM demo proxy, unless someone else had the same idea. Thanks for helping educate.
Edit: apparently (https://discuss.python.org/t/avoiding-compromised-certificat...), the relevant Debian and Ubuntu packages already patch the distro-provided pip to enforce using the system certificates. (Indeed, `ca-certificates` is listed as "Depends" for the pip package, rather than "Recommends".) But maybe they'll still be interested in the key blocking, on defense-in-depth principles.
> All of these are at one out of two compromises from being MITM'd during the upgrade.
This is not true; pip supports hash pinning, so an attacker would have to be able to defeat SHA256 as well (or compromise both the official Python discussion forum and PyPI itself, so as to forged the published hash). Other hashes can also be used. Instructions are provided on the Python discussion forum to self-upgrade pip with hash pinning: https://discuss.python.org/t/information-for-python-users-re...
Zabbix also confuses operators at this point. I recently had to increase a cache. I chose to go from a solid 4MB to a luxurious 256MB.
I know high-performance Java, so just a little jab there, but you don't size JVM parameters like that, except maybe code caches. And before Rust comes in, we also have a Rust application running without good resource control and it wouldn't like those resource constraints.
There is truth to this article.
reply