Hi GPT buddies
After launch of GPT store I had expected to see some good work on integration of Latest World Knowledge with existing ChatGPT functionality. I saw few trending GPTs based merely on web search that I am personally finding inefficient (searching multiple resources is not quick) and often ineffective (some GPTs do one search at time providing very limited response context to query).
I’ve not found original OpenAI web search feature very helpful either so since summer I’ve been working on tool using internal and third party web crawlers to create integrated and searchable vectorized knowledge base for GPT or AI agent to refer to when responding… I’ve wrapped the current version into GPT interface and wanted to share with community to see what you guys think - it’s called “Latest World Knowledge” and you can search it in GPT store or ask me to send you directly (can’t share link in post for some reason)
It aims to overcome single-source problem of web search with refering to number of most relevant (filtered) web references. It also does not need consequent web search calls making it pretty quick in my opinion… There are few areas that need work incl. certain topics like coding that can be handled in a bit more robust way but it’s a first release and we’ll work more improvements if there’s interest.
Also if you think you can use this via API to reference lastest knowledge in your custom GPT let me know, public endpoint is almost finished.
Hope to hear from all!
Alexey
Nice to try if you can provide me the endpoint via PM if not intended to be public.
1 Like
Hey Fabrizio, thank you for first message and reaction on this… I’ve connected via LinkedIn and sent you a separate message with link to our free chatbot (ChatGPT wrapper around knowledge base) to have a sense of it. I am finalizing API endpoint and will be in touch shortly. Feel free to share thoughts and preferences on how you would see our interaction with your apps/“GPTs” and if any special requirement on API response we should take in account…
–Alexey
1 Like
How big is the vectorised web scape? As in, is this an entire internet snapshot?
1 Like
@Foxalabs hi, thank you for your message. We have internal knowledge base that is focused on particular subject and is more in-depth. We are also using external partner APIs with more extensive coverage but probably less depth. It is not 100% but we are at active beta and will aim to improve / invest into features if community sees value…
To mention, it already does quite well against most of generic and specialised queries at the moment. Although we are working on interaction of existing LLM knowledge with the one accessed via RAG pipeline for it to be more reliable, less hallucinating and well addressing some specific queries like coding requests etc
By the way, I believe I cannot reference link to our GPT here because of my “new user” status… Although I’ve managed to put it into my profile description. So please feel free to check it out from there
One thing I would question is what use a generalised vector store would be. The model itself IS a generalised information store. Vector databases only really make sense when giving the model information it does not currently have.
2 Likes
@Foxalabs thank you for comment. The existing model knowledge as I know is not up to date and constrained by training data (which is still in 2023 I believe?). One solution to that was doing web search feature and, in fact, there’s number of solutions like WebChat and alike that extend this idea to update the model response with latest facts… Typical problem with this dynamic web scrawling is that it is limits itself to 1 or few web sources thus limiting information. So for example, if you’d ask for “latest AGI research”, tools relying on single source of data might fail to give you the comprehensive overview you’ve requested… Another solution is to dynamically scroll more web sources based on user prompt but we find it massively time expensive and crushing UX… Our idea is to have constantly refreshing knowledge of pre-scrapped web connected to model. It requires more investment into infrastructure and back-end but response enhancement seem to be quite considerable… Thank you for your thought again @Foxalabs ! Glad to keep communication open
And as I’ve mentioned initially, we do work on providing endpoints for GPT builders so their models are boosted with latest knowledge rather then being based merely on GPT-4 training set… would be glad to hear community thoughts on this!
Maybe a further enhancement can be to save scraped data in a specific, shrunken way and put a sort of MITM fast model trained to get such content and enrich it for the main model?
Just a thought and of course I can be wrong.
1 Like
@fabrizio.salmi @Foxalabs Hi all, I’ve been given a “trust level” (thanks God) so now can share our chatbot link here - ChatGPT - Latest World Knowledge
As I said it is a ChatGPT wrapper around the knowledge base API so you can have a sense of response and quality, pros and cons… We are keep testing it and comparing to other GPTs on top of store (like WebChat) and it does have competitive advantage in my opinion to those web-search based ones…