I’m looking at ways to optimize the search functionality for ChatGPT and other models that perform online searches.
When SearchGPT looks up data on our website, it downloads a lot of HTML, which is a pity.
We’re looking to see if there’s a better way to offer an API access direct to the content for the search functionality, so it can read and render faster.
Alternatively, we could do a user-agent detection and serve different content to the bot.
How do you handle this?