Hey everyone,
We just had our WordPress server knocked offline for about three minutes due to a massive crawler spike, and I’m trying to figure out if others are seeing this specific attack pattern.
Here is the sequence of events: We initially received a few legitimate indexing requests from verified OpenAI IP ranges using the OAI-SearchBot/1.4; +https://openai.com/searchbot User-Agent.
Immediately after, we got hit with a distributed attack using the exact same OpenAI User-Agent. The requests came from about 1.5K unique IPs scattered all over the world (UK, Netherlands, Switzerland, Brazil, etc.). The access patterns clearly indicate a coordinated crawler targeting our sitemaps and content. To avoid standard rate limiting, they kept the requests per IP very low (fewer than 30 requests per IP), but the aggregate concurrency was large.
My intuition is that this second wave is absolutely not OpenAI. It makes no sense for them to use a global proxy network for standard retrieval. It looks like a third party scraping our site and spoofing the OpenAI UA, possibly hoping that server admins have globally whitelisted AI bots.
Has anyone else noticed this specific one-two punch (legitimate OpenAI IPs followed by a distributed proxy network spoofing the UA)? If so, how are you handling the mitigation at the WAF level without accidentally blackholing actual OpenAI indexing?
Thanks in advance for any insights.