This is a request for help on how best to create fine tuning data to help with prompts returning code using a new API (not described on the web anywhere yet). It is for a new Python library. I have read the fine tuning docs, but I have not found any examples how to teach GPT a new API to use.
I have a private Python library that I would like to get example code generated for. I know Codex cannot be trained yet, but someone on Discord suggested to try training Curie instead and see how it goes. I cannot do it in the prompt directly because the full API is quite large.
I have tried supplying 10 sample Python scripts using the library with lots of embedded comments (I have to write them by hand). Trying it in Playground did not work (it generated output with lots of bizarre Python comments and no actual code).
Before I go off and write “hundreds” of code samples, is there a “good” way to provide sample prompts and completions that demonstrates Python code so that GPT has the best chance to “learn” the API of this new library? Or instead of code samples, should I create API documentation and load that up instead? List all the function prototypes and their purpose.
Then how to specify a prompt to trigger usage of the library? Use a weird unique name as the library name so I can reference that name in the prompt to encourage GPT to use the library I want?
My ultimate goal is to have a “scene” description in English and get it to generate Python code to assemble the scene in a 3D rendering engine. E.g. have a prompt like “Using the XYZZY library, create a program to assemble the following scene: Sam is sitting at his desk in class looking down at his hands. He is very sad.” The result should be Python code to add “Sam” to the scene, then set his position to the location of his desk, set the pose to “sitting”, and change “look target = hands”, facial expression set to sad, etc. (A similar project has been done for Blender, but Blender has lots of public documentation that GPT has already learnt from. I wanted to try for my own API.)
Dont use fine-tuning, use embeddings and a good prompt.
Thanks for the reply!
Do you have any information on how embeddings will help? E.g. links to blog posts on similar work?
My understanding is embeddings are useful to create vectors to power experiences like classification and semantic search (similar vectors = similar concepts). That will not generate new code fragments in unique ways using APIs will it?
E.g. for Blender, they had voice control like “Place 12 trees and random positions in the scene, rotated by a random amount.” It generated Python code to do this with loops and variables (which was then immediately executed). I don’t understand how embeddings would make that possible. (I don’t want to find similar code fragments, I want it to use my APIs like it is already doing for other libraries it has found on the web.)
I saw an example prompt (which I lost the link to!!!) which included an API spec in the prompt, which the generated code then used. My problem is the API is too big given the prompt length restrictions. Hence I wanted to train in advance somehow.
If embeddings can do this, then great! But I have not seen anything indicating it can do what I need. Is there something I am missing?
Thanks!!
you use embeddings to search and find the relevant API calls that are needed. Then include those as part of the prompt context, along with whatever else you need.
1 Like
Ahhhh! Okay that is making more sense. So include the function prototypes in the prompt, using embeddings (similarity search) to pick the most relevant function prototypes to include to keep the prompt length within legal bounds.
Assume the following types and function prototypes already exist.
def get_timeline_duration(timeline: Timeline) -> int:
def get_timeline_name(timeline: Timeline) -> string:
def create_timeline(name: string) -> Timeline:
Use only the needed functions. Only return the script body.
# Python 3
# Create a Timeline and and print its name on the screen.
Using the above prompt in the playground correctly returned:
timeline = create_timeline("My Timeline")
print(get_timeline_name(timeline))
I was hoping to pretrain the model somehow. What I really want to do is have prompts which are human descriptions of scenes (“Sam is sitting at his desk…”) and convert that to Python code to assemble a scene. I assume I need to train the model so it can understand that relationship better.
But I have a first step working - thanks for the help!