Can anyone let me know if we can actually use the LinkReader Plugin of OpenAI in the OpenAI API ? I want to be able to send a request to OpenAI API saying that “Crawl through google for this article and give me the response in a html format”. I was able to do that using the OpenAI Web application but no luck with the API.
I get that both of them are different and the plugin functionality is not actually integrated with the API yet but are there are any workarounds for this? I checked out Langchain but I dont exactly see how i can do it.
So, If I am essentially am able to use LangChain Google Search. Will Chatgpt be able to provide me the response in html format? I am actually confused a little over this. I am going to use Langchain google search instead of OpenAI API , so how will chatgpt give me the response?
I tried it out already. I want this to be purely automated, so I want a crawler which will go through Google or Bing and search for the articles I needed and only scrape the text i want and convert it to HTML. Writing that code will surely be large and I will also need to check out ratelimits and captcha defenses and more.
Yeah if you aren’t using AI to summarize or process the data, and are just looking for a tool to extract HTML from pages in a Google Search result, then you just need a web scraper.
ChatGPT could help you write one if you are familiar with any programming/scripting languages. Or there’s a bunch of different Chrome Extensions and other web scraping software out there.
Its not only about HTML actually. I want Chatgpt also read some articles and generate YAML code depending upon the articles found from crawling. If its only simple HTML, I would have surely used some scraping with awk , sed and wget.
Can you share more about your overall goal? Sounds like a multi-step process may be what you need, generic web scraper to get the HTML, then use OpenAI API’s to read/process the text.
So, I am a security guy and I am trying to play around with Chatgpt to essentially find any new exploits/CVE’s released as the first step. Once it finds the newly released CVE’s , I want it to use google, github and more sources to find the exploit code and build a YAML Template from the exploit code which can then be sent over to some scanners in my testing. But the issue here is, OpenAI API does not have Plugins functionality and I am unable to get the YAML back I need.
I am successful in making chatgpt retrieve the CVE IDs I need. But using the API to perform the crawling is not possible and hence the failure. I played around with scrapers to get the exploit codes but I feel its not an efficient process in anyway.
Can you share link to an example chat?
It all seems doable with the API, but you’ll need to cobble together several pieces. Generally URL reading plugins just fetch the page content and include it in prompt.
I feel like when you crawl docs for let’s say programing libs then embeddings would work very well too. Especially since data of gpt-3/4 is becoming outdated.
Haha. I want to receive this YAML Code from the API so that I can use it in my python script. But LinkReader does not work with the API and I receive some gibberish instead of the YAML Code.
I dont exactly get how Chatgpt created the YAML too because I get a different YAML every single time i generate the response and it seems like it uses different link for it then. I am doubtful if LinkReader is the correct one for this task but it seems to work in some cases but not all.
