Bots generating errors on my website

Community
1

I suddenly noticed hundreds of errors on the log file for a website I created that are all caused by openai bots as the HTTP_FROM says gptbot(at)openai.com and USER_AGENT is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)

2024-10-28_9-28-35
2024-10-28_9-28-351029Ă—723 4.58 KB

I looked into one example and it seems the path created goes way out of bound. This is the path /Odonto-lieyectoresli-541.aspx/assets/js/plugins/Docs/Productos/assets/js/Docs/Productos/assets/js/assets/js/assets/js/vendor/images2021/Docs/Productos/Docs/Productos/assets/js/vendor/Docs/Productos/Docs/Productos/Docs/Menu/Odonto-Gomas-para-pulido-de-composite-tipo-Enhace-1815.aspx

Before blocking the bot I am writing to find out what’s wrong with my website.

1 Like
2

Same problem here, with very long URLS with infinite number of &amp:
20.171.206.213 - - [29/Oct/2024:10:56:16 +0100] “GET /298-bois-chene-moderne?amp%3Bamp%3Border=product.price.desc&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Classique+Chic&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Classique+Chic&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Classique+Chic&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bamp%3Bq=Style-Design&amp%3Bamp%3Bq=Style-Design&amp%3Border=product.name.desc&order=product.name.desc HTTP/1.1” 200 25215 “-” “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)”

Please FIX THIS or stop crawling.

3

Glad to see I am not the only one.

More than 4000 errors today, I am considering blocking the bot thru robots.txt

4

Same here, I took the time to come here and report the problem, but I fear chatgpt developpers wont read us, there is no “BUG REPORT your AI is broken , please fix this” ticketing system . . . If we have no answer qite fast, I ll also block OpenAI . robots.txt or better . . . firewall dropping the whole IPrange NetRange: 20.160.0.0 - 20.175.255.255
CIDR: 20.160.0.0/12
NetName: MSFT

In the end I had no choice , those gptbot attacks were overloading my dedicated server, so I had to reject all gptbot requests :

ufw insert 1 deny from 20.171.206.0/24

for now. until openAI fix the buggy pseudo AI.

1 Like
5

From my side all errors reported above are not showing up anymore. I don’t know if something was fixed or if the crawler isn’t coming around anymore

1 Like
6

Thanks for coming back to let us know!