I am trying to create a model that can understand a text description and create pseudo code.
The code will be a sequence of functions to achieve a goal. There is a set of pre existing functions to choose from. The model can also suggest custom functions that can be added to the function library. Existing functions have a higher priority.
Once the code is generated I want a human to review and provide feedback, even be able to suggest changes.
Once the human accepts the solution, the model ingests it and evolves.
There is a lack of training data.
Are you asking to build a model…from scratch? Or fine-tune a model?
Considering you’re asking this on an OpenAI dev forum, I would recommend you fine-tune something like GPT-3.5 for this task. However, even then, there’s a lot of limitations, like, you’re not gonna be able to “evolve” a model with each interaction like that. There’s a laundry list of reasons why not, and this is something that sounds intuitive and easy to do, but once you start understanding ML, you realize how difficult that problem actually is, especially with RLHF (what you’re suggesting). It’s a problem that looks easy, but in practice is completely different. So, the best you’re gonna be able to do is fine-tune a model to handle responding to you in the specific way you want.
Regardless, you gotta make that training data yourself. Whether you’re fine tuning or building from scratch, if you don’t have enough training data, you have to make it yourself.
I also cannot stress enough: if you think you can tackle making your own LLM from scratch, don’t. Seriously, building a model from scratch with NLU good enough to understand what you described, let alone produce detailed output with suggestions, is an endeavor that takes a startup with hundreds of people and billions of dollars to build and train.
Start small, start looking into how to fine tune a model that already exists, and go from there. Understand the limitations present in the field, and you’ll be able to come up with something that could give you something that would satisfy your needs.