OpenAI Tutorial Web Crawler - Please turn Javascript on

Community
, ,
1

Referring to this tutorial (https://platform.openai.com/docs/tutorials/web-qa-embeddings), I followed until the web crawler section, and when I ran the code to crawl openai.com, the .txt files generated all have the following results:

Please turn JavaScript on and reload the page.Please enable Cookies and reload the page.

Does anyone have the same issue? How to resolve?

1 Like
2

It’s a CloudeFlare issue to stop this exact thing. You’ll either need to be more creative with how you access the pages or just scrape some other site.

2 Likes