How to feed data for completions, instead of using prompt/answer fine-tuning format?

Finally some insight, thanks :+1:

Basically, if i understand, we currently need a database on our side

  1. Generate embeddings from pdfs, and whatnot, and save to a local database
  2. Generate embedding from question, and do a query against local database, and return original content
  3. Use original content as input to pre-feed prompt & add original question
  4. get result, and pay mucho tokens ^^

I hope OpenAI integrates something like this on their server backend. We need a way to simply add our content to be indexed by GTP.

Once companies can simply have a marketing or documentation team feed the GTP database, and have another team build a frontend that is used internally as a AI Assistant, then this will go to the moon :slight_smile:

Imagine simply feeding it your whole website archive, and then have a search assistant on your site. This would rock on sites that rely heavily on reading materials. QA, marketing, education, …


Here is a simple use-case for the company i work for

They do a lot of collaboration with Airbus industries, much to do with safety regulations, electrical standards, assembly/supply chain management., and training courses/consultancies in these matters

So we have domain experts & instructors (airbus has Service Now as an internal tool)

Now if i could just feed GTP all kinds of technical docs, standards, etc, that would make for a great interactive wikipedia

3 Likes