How do I replicate ChatGPT's web search feature?

Community
, , ,
1

Hey,

I have been trying to replicate the web search feature as I am pretty sure there is no API access to it.

At the moment my application accepts queries from the user and uses Bing Search (from Azure) to get the results and summarize the “snippets” from those results using GPT-4o and then present it to the user.

However my search results never seem to be deep or as detailed as the one from ChatGPT.

Is it possible to do it without web crawling?

I simply cant find a definitive answer online anywhere.

Thanks

1 Like
2

You can use exa.ai search api for this

3

  1. Parse search results or use an API like brave for searching.

  2. Scrape all the results or let the AI choose the pages to scrape based on the search snippets. Then, using HTTP requests or something like playwright or puppeteer to scrape the pages in parallel.

  3. (optional, but recommended) Clean data, either programmatically or with ai. Model I would use - GPT-4.1 nano

  4. Summarize in parallel - Model I would use - GPT-4.1

  5. Aggregate all summaries and return. Model I would use - o3-mini (medium)