My website gets hit a lot by openai gptbot, far to much!!, I’ve got bot blockers on, but i know they are helpful for seo, well I hope! I don’t want to block them completely I guess.
Is there a way to make them only hit at say 2am to 4am???
1 Like
Hi and welcome to the community!
OpenAI has a dedicated mailbox where you can report your issue:
I hope the problem will be resolved soon!
Interesting suggestion! There’s not currently a web standard for this but we can think about whether there might a simple way for sites to express that intent at scale.
In the meantime you can find more information about OpenAI crawlers here. Note that it’s possible to block GPTBot but continue to allow SearchBot, though both are careful to maintain rate limits that are typical of other well behaved crawlers.
2 Likes
I just want them to do it at say 2am when my site is quiet. robots txt doesn’t do it
Seems to always be ending in …https://openai.com/gptbot
I have to keep upping my aws server from t3.medium to t3.xlarge to make it work, so costs me more money!