I want to browse the web to find out about product descriptions. I have been doing some tests with ChatGPT 4o and it works fine. I have also been reading posts about this subject (browsing the web through API) and didn’t find a relevant answer.
My questions:
Does the API have new capabilities that allow users to browse for a specific subject (like a product name)?
If this is false, what is the best way to browse for information? Is there any built-in function in the API or Assistants?
Can anyone give me a link that shows a code doing this kind of work?
I intend to ask the AI to browse for a product, and then choose (among the 5 first results) the most complete description
If you have been working on a project like this I would appreciate if you could share your experience (even if you failed )
Hi @Lalo ! You may also be interested in checking out this cookbook that explains how to integrate Bing API with OpenAI ecosystem and create a browsing experience. Bing API in particular gives you a certain amount of requests for free each month. There are other services like Brave API as well.
The cookbook gives you the basic structure on how to browse the web, but I found in practice I got pretty bad results from this. Web scraping the page content is difficult as most sites don’t like to be scraped. Using the requests library will result in many 403 forbidden responses, because servers can easily detect that this is not a real browser making the request. Using Selenium or similar is better, but also is quite time-consuming, and you will still get many sites detecting an automated browser.
I am really curious of how OpenAI gets around this problem with their own web browsing, and how they can be so quick. Does anyone know what they are doing differently?