I looked into this and was tempted as there is less read locking and better RNG performance but they stripped the SSL_get1_curves() API which I depend on for bot detection plus I would have to recompile nginx and all its modules everywhere so in the end it was not tempting enough for me. Perhaps I am just lazier in my older age.
We ran a number of tests, including sending a percentage of real traffic to endpoints running AWS-LC and Wolfssl. Technically yes, OpenSSL 3.X is slow, measurably so, but not enough that we wanted to deal with it. That's for our use case, I can easily imagine there being others, where maybe you have different hardware or way more traffic, where it would be worth while. For basic TLS termination in haproxy, it wasn't really worth it to us.
The infrastructure is already in place for a global great firewall. One need only to compel Google and Cloudflare to block a site in their DNS and most of the planet will not be able to reach it.
Did any of them move to a .onion? I would expect commercial sites to cave and some small sites to just move to Tor which probably accomplishes the same thing as far fewer people use the Tor Browser or Brave. I could also be delusional.
Thankyou for taking the time to detect and validate/verify those hosts. Is there a way you could make an outbound connection from them (maybe curl) to a site like ifconfig.me [1] and start logging which providers they are on then post a report of them on HN?
That doesnt prove FranTech itself was compromised, but its a strong cluster. AFAIK nobody has contacted them yet, if no one does, I plan to give them a heads up in the coming days.
That is useful information. Historically I have blocked BuyVM on my nodes due to their pro-privacy pro-tor-exit policy. Now I just put anyone arriving from them into a moderator review queue for my silly hobby sites.
It's a good policy to permit Tor exit nodes but it means their network will egress some of the most untoward behavior so that is expected and I would not default to assuming they are compromised, rather have some of the highest concentration of Tor exit nodes. There are pros and cons to such providers and you found one of them.
This is just my jaded take but I suspect that whomever is good today may be gone in the not too distant future. I have been watching registrars get gobbled up by investment groups that have no knowledge of or interest in DNS. They seem to be milking these registrars and leaving behind broken husks.
There should probably be a deadpool board that lists all the registrars and lets people bet on which will be the last ones standing. I am going to guess a couple of the last ones standing (that still have something resembling technical staff) will be Amazon and Cloudflare and to be clear I did not say anyone can reach their technical staff.
Whatever the consensus will ultimately be in this thread this week will probably be registrars we perform libation for in the coming years.
still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
robots.txt can used as a reward mechanism. If an automated visitor that we would normally block begins by accessing robots.txt that's a good sign and we give that a plus score or whatever. If subsequent accesses from that visitor comply with robots.txt, we let them continue (perhaps with some rate checks).
Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will.
So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism.
In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else.
Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with.
I have been writing bots and scrapers for almost 20 years.
I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.
This implies the solution is every actor in the world willfully complying which doesn’t sound very realistic if you ask me.
Similar to saying world hunger would be solved in everyone who could donated $100 to those in need. Sure it may be true but foot stomping about why isn’t it organically happening is missing the point.
The problem itself does not matter. If the nature of any solution relies on willful global compliance, it’s not going to be very effective at whatever it’s trying to solve.
Every society fundamentally relies on willful compliance. Sure, you can force a few outsiders, but once a sizeable population stops being compliant, it just breaks down.
I'm saying telling people they shouldn't do something is about as useless as farting in the wind. Blaming someone does nothing, it's the force of action behind the words that mean something.
But, because this person could be anyone, anywhere your words are hollow and empty. You are impotent to stop their future actions.
And may be or could be part of the solution by them taking upon themself to implement it in an ethical manner by having the bots and scrapers respect robots.txt; unless the spec specifically says not to in that case I agree with finding new clients or another job
Yeah it reminds me of the typical copyright notice found in a book, even decades ago:
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
Returning some text files from the server fs is _wayyy_ lighter than letting the bot go through the search, middleware, and db while crawling your stuff.
I'm not convinced that it's possible to poison AI. Real data all correlates, fake data is all unique in its own way. I suspect feeding AI fake data is a waste of time.
Angel banning is better, 100% agreed on giving it false data that the scraper builders can't easily detect. Give it seemingly correct and useful but totally wrong answers
I would assume it's exactly what they described, banning something by pretending to still helpfully assist them while silently undermining them instead.
I can not rely on Opus 5.5+ to "fix" whatever crops up. Claude is great but should it find a security related bug one too many times, it gets the Anthropic Shock Collar [cyber]! and I have to pick Opus 4.8 or older to continue and they try to up-sell Mythos which I am not permitted to use without going through their verification program that they will not offer to me. Thankfully for most of my hobby tasks (thats all I do now) Opus 4.8 is usually mostly good enough. In my opinion their newer models are powerful but not something I can rely on. Maybe I am one of the few running into this.
Same here, Firefox with ublock Origin and NoScript.
On the topic of caching I run my own edge nodes, no regrets. Privacy maintained, finer grained controls, no large scale ddos protection but I have never needed it.
At the moment two large VM's using NGinx on the front-end and HAProxy on the back-end to do the load balancing and health checks. NGinx and HAProxy communicate with each other over a socket thus taking the network out of the picture. NGinx caches files to a tmpfs ram disk. The back-end HAProxy communicates over VPNs to the origin servers. I've automated scaling up edges but I still have work to do calculating the dynamic/static back-end limits per edge but for my tiny hobby sites I do not foresee needing to automate this any time soon. The edges force level 1 zstd which nullifies many of the bots that slip past my anti-bot methods as they just get gibberish. This also breaks old browsers which I do on purpose as nobody should be visiting a chan site (which I use as a replacement for my old blog) without a fully patched system. I keep additional VM's configured as spares or as "side doors" for friends to use in /etc/hosts in the event some day I attract h8 and vitriol. I use NGinx Geo in combination with FireHOL and a few other git repos to determine who should be allowed to post real time vs. be held in a moderator review queue which has saved me multiple times.
I'm not doing Anycast or GSLB as it is just for a hobby site. If it ever grew in popularity (unlikely) then I would probably just do DNS GSLB in PowerDNS with PowerGSLB.
For me it was not a lot of work because it is a hobby. I'm sure others here could automate this in Anisble, Docker, Podman and probably have. My setup is probably childs-play compared to what others here do professionally.
reply