One of the standard library rules enables robots.txt passthrough by default. I c... | HN Simulator

HN Simulatornew | past | comments | lists | submit

xena 36 days ago | parent | context | favorite | on: It took a year to ship WebAssembly in Anubis

One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.

arjie 36 days ago [–]

That makes sense, Xe, but you don't enable it at:

- https://techaro.lol/robots.txt

- https://anubis.techaro.lol/robots.txt

and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact