Hi everyone and apologies for the long post. Just trying to give as much info as possible.
A little background on what I’m trying to do: I would like to generate completions based on the context of a specific project the company is working on. For example, say the company is working on multiple software development projects. Each project has its own set of artifacts (e.g., requirements, project schedule, documentation, user-generated content, etc.). If a developer on Project XYZ asks, for example, when the next release is scheduled for or what the description of a feature is, I would like the completions to be based on the context of Project XYZ’s artifacts—i.e., Q&A-style completions. I would also like to generate creative, open-ended completions based on Project XYZ’s context.
Following the “Question Answering using Embeddings” sample cookbook, I was able to get excellent results for Q&A-style questions. However, I have a couple of concerns with this approach:
Cost can become a factor since I would need to get embeddings for every possible context, and use up a lot of tokens for every completion by dynamically adding context to each prompt
While providing concise context for Q&A-style completions—when the answer is more deterministic—is feasible, it is more challenging for creative, open-ended questions since they require more context, further adding cost concerns and sometimes running up against the prompt size limit
I was hoping I could achieve reasonably strong contextualization using fine-tuning to avoid having to provide explicit project context in the prompt—and playing with temperature for Q&A-style vs creative completions. I tried a bunch of different fine-tuning data sets (e.g., open-ended no-prompt completions, tagged text, synthetic variations on the same text for reinforcement) but unfortunately couldn’t get good results—the model either broke (e.g., kept repeating the same word/phrase) or would provide non-contextualized completions.
The fine-tuning data set isn’t exactly large, so that might be the issue, but strong contextualization with fine-tuning alone might just be unfeasible. I’m curious what others in the community think on the subject of effective and efficient completion contextualization.