Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.

Not once have I had someone I work for in this capacity even ask about robots.txt

I've enjoyed the arms race with scrapers as they have taught me a lot over the years. Some of them leave me funny messages in my logs and seem to enjoy working around my crude mitigations almost as if it were a puzzle. Which of your scrapers can parse this post without the help of a real browser or headless chrome? [1] I am curious how far the tools have evolved and how someone with your experience instruments them if you are willing to share.

[1] - https://altchan.org/ot/20

Conversely, if an obviously automated visitor does not fetch robots.txt, that can be an instant ban, kind of like a reverse honey pot, if you will.

So it can have some nonzero value in the overall context; it's just silly to think of it as a blocking mechanism.

In short, robots.txt documents what access is allowed; it states the policy. Enforcing it the job of something else.

Someone you work for in this capacity shouldn't ask about robots.txt because that's the sort of detail for which they have you to deal with.

The internet is not a society. The internet is all societies. Hell, more and more it's not even human societies.

But, because this person could be anyone, anywhere your words are hollow and empty. You are impotent to stop their future actions.

> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.

For AI, wasn't the book first "reproduced or transmitted" into a training system?

How would we reformulate it today?

flat files, a tarpit, maybe zip bombs as the sibling suggested. Nothing calculated. I'd go with zipped Brothers Grimm

Make of that what you will.

Whilst I assume you meant to say “scrape”, I can’t help but appreciate the poetry of that typo.

I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.

I would never in a thousand years accept such a one-sided, despicable business relationship as their proposed terms seek to establish.

It's already bad enough they hoover up your customer lists and cross-link to their own - for purposes of gaining insight on individuals who never consented to it.

Now they're trying to use their monopoly over a form of social networking to bully you into arbitrary contract terms for a largely unrelated product.

You don't need Meta as much as you think. And in aggregate, they need their customers far more than those customers need them.

There are much friendlier ecosystems in which to engage with your users.

My bank advertises with Meta and included my account in some kind of upload list (probably Meta Ads Manager). I noticed their logo while poking around in Facebook's settings, under Audience-based Advertising. I was livid.

As far as I'm concerned they had no business telling a creepy company like Meta who I bank with. I certainly never consented to that.

I dug up my original banking agreement - a 5-page document on aging, yellowed parchment that I signed over a decade ago. Nothing in there about them being able to unilaterally change terms without notice. I also printed out their latest, 30+ page standard banking agreement which (after making inquiries) they seemed to think applied to my account.

I brought both in to my bank manager and put them on the table. At a high level, I pointed how ridiculous the size had grown to, and he was sympathetic.

In terms of details, I'd highlighted several portions that pertain to privacy, and we walked through the differences together. I underscored how none of them were needed to provide me banking services, how the benefits they conveyed were solely to TD, and how they were harmful to me as a customer.

In the end, he agreed to formally acknowledge on file that I did not agree to the new terms.

You may disagree over how much effect that will have in practice, but the acknowledgment was not just a bureaucratic exercise. It established very firm evidence that I never agreed to the new contract, or any terms the bank might rely on as a basis for the objectionable practices. It would become material if those practices cause me damage in the future, significant enough to litigate. i.e. They knew they lacked agreement, and continued to provide me banking services.

He also worked with internal departments to toggle all available opt-out flags.

I was fortunate in this case the manager comprehended my concerns and seemed to want to keep me as a customer. I recognize not everyone would necessarily get as far.

It really disgusts me that some of the most degenerate practices of silicon valley are spreading to other industries. With lawyers more broadly adopting some of the most viciously one-sided legal language big tech pioneered.

I for one am not on board, and refuse to be bullied into providing some kind of imaginary consent. I say 'imaginary', because I don't think anyone in their right mind who reads and understands such terms would ever agree to them willingly.