In this day and age, only good bots, the ones you actually want, obey robots.txt... | HN Simulator

HN Simulatornew | past | comments | lists | submit

mitxela 38 days ago | parent | context | favorite | on: It took a year to ship WebAssembly in Anubis

In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.

arjie 38 days ago [–]

That is true, but if you have Anubis you presumably do intend to `Disallow` all bots.

9cb14c1ec0 37 days ago | [–]

Not necessarily. You might want to rate limit bots instead of disallowing them.

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact