I am currently working on a plugin designed to scrape web pages via an API and display the content within our application. However, we are encountering an issue where the prices listed on the pages being scraped are not showing up in our application.
Details:
Functionality: The plugin is supposed to scrape and display content, including prices, from various web pages.
Issue: The plugin is not displaying the prices listed on the scraped pages fills in generic ai.
Questions:
Has anyone encountered a similar issue where specific elements like prices are not being scraped or displayed correctly?
Are there any common pitfalls or challenges in scraping prices that we should be aware of?
Could this be an issue with the way the plugin interacts with the API, or could it be related to the scraping logic itself?
Are there any recommended libraries or frameworks that are known for effectively scraping prices from web pages?
Any insights, suggestions, or recommendations would be highly appreciated. If more details are needed, please let me know, and I will provide them.
Are you making ChatGPT plugins, or features for your own API chatbot? What would you mean by “display the content within our application” then?
You report that an AI doesn’t know prices that should be in the page it retrieves?
When unsure where a problem is cropping up on code that is running without errors, the diagnostics would involve gathering information about the state of variables at the various steps of processing. I would log each state, from page retrieval, processing, sending to the AIs, the data being sent by your API to OpenAI.
You might find some place where a $ means “format text” or “is a variable” in your code.
Thank you so much for your response and apologies for any confusion. To clarify:
Type of Plugin: It’s a standalone WordPress plugin we are developing. Its primary function is to answer questions about our products and services, along with their pricing, to provide quick and accurate information to our users.
Content Feeding: We are feeding the plugin with content from our website, which includes detailed information about all our services and their respective pricing.
Issue with Pricing Information: The plugin performs well in providing information about our services; however, when it comes to questions related to pricing, it seems to revert to generic AI responses instead of providing the specific pricing information available on our website.
We are currently reviewing the data flow and examining the state of variables at each step, as you suggested, to identify any discrepancies or points of failure that might be causing this issue. We are particularly focusing on how pricing information is processed and relayed by the plugin.
If you or anyone else has encountered similar issues or has insights into potential solutions or areas we should investigate, we would greatly appreciate your advice and feedback.
Rather than “feeding” by whatever method you are using (Wordpress keyword search?), I would put a function for use by the function-calling method of the AI models that will retrieve the specific page contents or your own document for pricing information.
metacode:
\\ function description: Must be used for retrieving and reporting on product and services pricing. AI has no pretraining on prices. function: get prices
\\ property description: specify page to retrieve, 0 is index, 1-9 are lists. price_page: number
We are using the api.openai/v1/chat/completions endpoint for our chat completions. We provide prompts that contain URLs and ask the system to crawl these URLs. We then request answers to questions based on the data from the crawled URLs. Below is the prompt:
“You are a helpful assistant, Crawl these urls: www.test/page1, www.test/page2 and give answer to below question according to the content of the link crawled”
“How Much Is A Price Of Warrant?”
Whereas www.test/page1 contains simple text-based data in its content, with prices of different things like “The price of a warrant per share is $11.50.”
In the response, we are often getting “the price of the warrant is not mentioned in the link”, or sometimes it directs us to “visit the URL to see the price”.
So, the question is, why are we not getting the exact price here?