Lot of things I’m not understanding First of all: the code runs well with the following: (I added spaces so i could post links) Define root domain to crawl domain = “openai . com” full_url = “ht*ps: // openai. com” I get a list of the files its crawling but many say “can’t parse”. Why is that? s…

Welcome to the forum. Do you have a link to the tutorial? Some code? Not all sites will allow you to scrape them…

Thanks Paul! The tutorial was the one on the OpenAi website https://platform.openai.com/docs/tutorials/web-qa-embeddings

I hope this can help you: [image] GitHub - BuilderIO/gpt-crawler: Crawl a site to generate knowledge files to... Crawl a site to generate knowledge files to create your own custom GPT from a URL - GitHub - BuilderIO/gpt-crawler: Crawl a site to generate knowledge files to create your ow…

Web-Crawler Tutorial (OpenAi) 403

Documentation

polepole December 28, 2023, 5:28pm 4

I hope this can help you:

Topic		Replies	Views
OpenAI Tutorial Web Crawler - Please turn Javascript on Community tutorial , openai-documentation , documentation	1	448	June 16, 2024
Web Crawling official documentation Community gpt-4 , api	0	858	May 31, 2023
Web crawler example does not work Documentation	3	1450	August 8, 2023
Crawling Website Tutorial Rate Limit Error API	8	874	February 4, 2024
Create an IA which will crawl the pages and talk about it Community chatgpt	9	2726	January 29, 2024

Web-Crawler Tutorial (OpenAi) 403

Related topics