I am trying to convert a website into chatbot version with FAQs. I want to extract text from all the site links and then auto-categorize and feed into GPT. Anyone out there who has done this?

Yes, there are tools available to scrape website data and feed it into GPT. Some popular options include BeautifulSoup, Scrapy, and Selenium. These tools allow you to extract text from websites and organize it into a format that can be easily fed into GPT for training or use in a chatbot. You may al…

Right. Thanks for sharing. Are there no APIs we can use that do all the scraping? I think there is a limit to what we can feed into GPT so that would just make it harder if we’re trying to feed it a website with 20+ pages/site links

Yes, you cannot simply “give a web site” to a language model. The amount of custom input AI can accept is limited. You must use techniques for providing parts of the knowledge that would be relevant to the current user input. Your keyword search is “embeddings vector database retrieval augmented gen…

@fak500 , you have a few options. As the previous users mentioned, you can try out these tools to get started Playwright Beautiful Soup Selenium However, for any modestly complex application, you’ll soon realize that these tools are not sufficient. If you scrape a considerable portion of a websit…

[image] peternjenga: you’ll soon realize that these tools are not sufficient. If you scrape a considerable portion of a website, you’ll get blocked if the website is protected by DDOS/anti-scrapers like Cloudflare Which should be enough to indicate that these people do not want their websites…

Echoing @anon10827405 said, you are putting yourself in an adversarial relationship with the sites you are scraping. Even proxy providers like Brightdata will block your access if they notice you are scrapping portions of sites that the creators intended to keep private and/or don’t respect robots…

Just to add to my question - we will be asking users for their consent before scraping their data. If their provider blocks us then that’s a completely different story. There is a chat gpt plug in called browserOP that does it really well and wonder how they did it.

[image] fak500: There is a chat gpt plug in called browserOP I am going to build one “scraper GPT” myself this weekend. Mind you : I am scraping my own sites :slight_smile: so that’s allowed. Workflow So I will create something like this; Conversation starter button with GoFetch! Value a…

You can use some free tools which allow website scraping to paste to a GPT

Any tools out there to pull/scrape complete website data and feed it into GPT?

API

lachie1 October 11, 2023, 12:47am 2

I have used Python tools like Beautiful Soup and Selenium before

Topic		Replies	Views
Turn any website into an API with GPT-4 Community gpt-4 , api	11	11838	May 26, 2023
GPT for scraping (extracting from) unstructured web pages API	0	2250	December 18, 2023
Unstable output from GPT: Refuses to regenerate previous success API	3	842	December 14, 2023
How is ChatGPT able to extract webpages so quickly? API chatgpt	3	3716	July 9, 2024
Create an IA which will crawl the pages and talk about it Community chatgpt	9	2833	January 29, 2024

Any tools out there to pull/scrape complete website data and feed it into GPT?

Related topics