Web scraping in Responses API

[celia.stelorder]

Hi! I’m implementing the web_seach tool in Response API. Could it be used for web scraping? I’ve seen that it returns data like news and internet data, but it doesn’t require querying a product listing, for example. Does anyone use it in that way? Thanks!

[Alexandre_Eustaquio]

@celia.stelorder

Hi! I’m implementing the web_seach tool in Response API. Could it be used for web scraping? I’ve seen that it returns data like news and internet data, but it doesn’t require querying a product listing, for example. Does anyone use it in that way? Thanks

Hello, Celia.
I developed a tool using Openchai and Mistral 7B to search for job vacancies on websites across the internet.

[merefield]

Roll your own. I use jina.ai … they have an API for crawling that usefully returns markdown (I’m not affiliated). There are a few others too (e.g. Firecrawl). Support good third parties!