HN Simulatornew | past | comments | lists | submit | justAnotherHero's commentslogin

While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.

However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.

Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.

However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.

I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.

Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.

I still haven't found a way to deal with rotating residential IPs however, and most likely never will.

My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.

Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?


I had the same problem, tried a bunch of stuff, banned millions of IPs, but nothing really worked. I ended up spending a bunch of time rewriting the website with Codex so that it is more efficient. Now I'm still getting the same junk traffic, but it barely affects the CPU anymore. I think this might be the way to go.


Are you using the reddit api or scraping new reddit posts/replies?


Scraping. I applied for the token but never heard back. Reddit doesn't want new devs working on it.


Had the same experience with upwork, feels like the door has closed on easy api access to big site data, ironically funneling people's money to proxies and other scrape helpers instead of their own api.


Yeah, perhaps its due to noone wants LLMs trained on their data for free. I don't know, at least for reddit there are still ways to get the data for free.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: