GPTBot Mass Crawling Truncated URLs

Community
1

Not sure if anyone in the community has observed this, but recently GPTBot has been crawling my website pretty heavily, with most requests being truncated URLs that return a 404 response (i.e. if the page’s URL is /example-1/ GPTBot is requesting /examp in Cloudflare logs).

These URLs aren’t linked anywhere else on the site (or on other websites) and I’m curious if anyone else has seen this.

2 Likes
2

Hi and welcome to the community!

There is an official email address that you can send your report to. You can find it here:

1 Like
3

Thank you for the bug report and for the additional information you provided in DM. We have fixed a bug in GPTBot’s link extraction from large html text nodes.

2 Likes