The new web search in GPT-4o is extremely helpful. Will this feature also be available via the Assistant API in the foreseeable future?
5 Likes
I agree it would be awesome if it became available through the API. But you can easily replicate this functionality through function calling. You’ll need two functions. One for Google Search (Google API), and you’ll need a function for website-fetching which can visit and fetch the actual websitecontent. I recommend puppeteer for this.
You can also use Langchain, which already has build in functions for Google-search and website-fetching.
2 Likes
Thanks for the tip on how to solve this yourself. If it is never implemented, I would implement it exactly as you described. The cool thing if it would be integrated is that it would probably be much faster than scraping it myself. I noticed with GPT-4o that it was extremely fast to load the web pages. With previous GPT versions it always took quite a long time for the web page to load. I suspect that OpenAI has probably already saved them.
1 Like
When web search is available through the API, this will be a game change. Can’t wait! 
2 Likes
I solved it for me by sending the following message to GPT-4o
If you need to search the web to get required information, reply with the message starting with the word Search followed by the search query
You will be provided with the Bing Web search API result in JSON format
To get the content of specific URL for analysis, send the message containing only the URL, starting with http(s)
You can ask for multiple web searches and multiple URLs one by one
So then it tells what it wants to search, my app runs the search using the Bing API and feeds the results to GPT, then it asks for certain websites from the search results and my app just downloads and sends the content back. Works pretty well
3 Likes
Can you share the logic of your code? Thank you.
1 Like
+1 to the concept from @asovych and the follow up from @james28 (it’s not working for me and I assume not for you either since you asked for the logic).
Either (1) I get hallucinated/fake links or (2) I don’t get actual URLs.
@asovych can you clarify the logic you’re using and what your input and outputs are?
Hi guys,
Here is my PHP code. As I mentioned, it works well for me. At the beginning sometimes it gave the “Request too large” error, but after some time OpenAI promoted my account to the next tier with the increased limits and the problem was gone
<?php
function getCompanyProfile($company_name) {
$bingEendpoint = 'https://api.bing.microsoft.com/v7.0/search';
//array of predefined topics, like "An overview of what the company does", "Locations where they operate", etc.
$topics = Utils::getList('chatgpt_request');
$chat_messages = [];
$query = "You will be asked to provide the information about a company. The response should include the following:\n";
foreach($topics as $t) {
$query.= $t . "\n";
}
$chat_messages[] = [
"role" => "system",
"content" => $query
];
$query = "If you need to search the web to get required information, reply with the message starting with the word Search followed by the search query\n";
$query.= "You will be provided with the Bing Web search API result in JSON format\n";
$query.= "To get the content of specific URL for analysis, send the message containing only the URL, starting with http(s)\n";
$query.= "You can ask for multiple web searches and multiple URLs one by one\n";
$query.= "Once you are ready, send the company info to the user in one message";
$chat_messages[] = [
"role" => "system",
"content" => $query
];
$chat_messages[] = [
"role" => "user",
"content" => "Please provide the information about the company ".$company_name
];
$run=true;
$i=0;
while($run) {
//Send the request to OpenAI
$data = array(
'model' => 'gpt-4o',
'messages' => $chat_messages,
'n' => 1,
);
$error = "";
// Set up curl request
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.openai.com/v1/chat/completions",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => json_encode($data),
CURLOPT_HTTPHEADER => array(
"content-type: application/json",
"authorization: Bearer " . CHATGPT_KEY
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
$error = "HTTP Error #:" . $err;
} else {
$result = json_decode($response, true);
if(!empty($result['choices'][0]['message'])) {
$chat_messages[] = $result['choices'][0]['message'];
//Split response by paragraphs, as often it sends some reply, like "OK, let's search" and then the actual search request
$lines = explode("\n\n",$result['choices'][0]['message']['content']);
$run = false;
foreach($lines as $message) {
//remove formatting chars that may me included in the response
$message = str_replace(['"','*','#'],'',$message);
if(strtolower(substr($message,0,6))=='search') {
//If the message starts with the word "Search", run a Bing search and send the result as Json to the OpenAI
$qry = str_replace(':','',substr($message,7));
$search_results = Utils::BingWebSearch($bingEendpoint,BING_API_KEY,$qry);
if(!empty($search_results)) {
$chat_messages[] = [
"role" => "user",
"content" => $search_results
];
//Continue querying the API
$run = true;
}
} elseif (strtolower(substr($message,0,4))=='http') {
//If the message starts with 'http', just download the web page content and send it to the OpenAI
$page = file_get_contents($message);
if(!empty($page)) {
$chat_messages[] = [
"role" => "user",
"content" => $page
];
//Continue querying the API
$run = true;
}
}
}
} elseif(!empty($result['error']['message'])) {
$error = $result['error']['message'];
} else {
$error = "Unknown OpenAI API error";
}
}
$i++;
if($i>50) {
//Just in case if it asks for too many searches
$error = "Too many messages";
}
if(!empty($error)) {
//If something went wrong we just show the error message as chat response and stop querying the API
$chat_messages[] = [
"role" => "system",
"content" => $error
];
$run = false;
}
} // end while
//If the loop is over, this means the last message contains the requested info (or the error), so we just return it's content
$last_msg = array_pop($chat_messages);
return $last_msg['content'];
}
1 Like