I built a website and I want to allow chatgpt to scrape and read it. Ideally users can drop a link into chatgpt and it can read/scrape from that link. I own the website, so have full control. My robots.txt allows for all scrapers. My site is proxied through cloudflare, but I disabled scraping rules. The issue is that ChatGPT is saying its getting 403 or a 500 error when reading the page, but when checking my logs I see a 200. Do I need to do anything on my site to allow this?
This forum is not for ChatGPT support, if you need help with your account, payment, etc, please visit https://help.openai.com
Please DO NOT SHARE personal information (email, credit card, etc) or account related details publicly.