I tried asking chatgpt to access various Indian websites & it was not able to read them. It always mentioned that its crawler got the error 500.
Sample:
cybage [dot] com
stocksdeveloper [dot] in
I did a lot of investigation to see if the blocking or error is due to server or firewall. But none of it helped.
Conclusion is that we do not even see the requests made by ChatGPT crawler in access.log of the server.
There is no firewall blocking at the hosting or ISP level as far as I can see.
So this means, the issue is deeper into chatgpt or somewhere in between.
I tried accessing using curl chatgpt bot’s user-agent, US server. It worked all the time, but it is not working for gpt’s internal crawler.
So this means, ChatGPT is not able to access latest information.