PromptOptimization: Genetic Algorithms for Prompt Optimization
We’ve been focused on making the interaction with chatbots more reliable, efficient, and adaptive. With the GPT-3.5 Turbo model providing us with vast capabilities, we set our sights on enhancing the dynamism and responsiveness of conversational agents. The
PromptOptimization class is designed to make your chatbot conversations smarter by leveraging genetic algorithms, test cases, and evaluation metrics.
Crafting the perfect system prompt to get a desirable response from the chatbot often requires trial and error, intuition, or a deep understanding of how GPT models work. A prompt that works well for one task may not work as well for another. We aimed to address this variability by introducing a new optimization layer:
The main approach is inspired by genetic algorithms where mutations and recombinations are used to evolve a better “species” (or in our case, a more effective prompt). The code is split into four significant areas:
- Test Cases: Defines the input message and the expected response.
- RunGeneticGenerations: Orchestrates the genetic evolution of system prompts.
- Evaluate: Provides a scoring mechanism based on how closely the output matches expectations.
- GeneticApi: Interface for mutation and recombination, implemented using a proxy for ChatGPT.
Import the Library:
Define the Test Cases:
import com.simiacryptus.openai.OpenAIClient import com.simiacryptus.openai.opt.Expectation import com.simiacryptus.openai.opt.Expectation.* import com.simiacryptus.openai.opt.PromptOptimization import com.simiacryptus.openai.opt.PromptOptimization.* val api = OpenAIClient() val promptOpt = PromptOptimization(api) val testCases = listOf( TestCase( turns = listOf( Turn("tell me a joke", listOf(ContainsMatch(Regex("Why did.*")))) ) ) ) val systemPrompts = listOf("Make the user laugh", "Tell a joke") val topPrompts = promptOpt.runGeneticGenerations(systemPrompts, testCases)
- User Message: The input from the user.
- Expectations: A list of conditions the response should meet.
VectorMatch: Utilizes vector embeddings to compare the semantic similarity between an example string and the model’s response.
VectorMatch("tell me a joke").score(api, response)
ContainsMatch: Searches for a regex pattern in the chatbot’s response.
ContainsMatch(Regex("Why did.*")).matches(api, response)
Mutation and Recombination: These are essential aspects of genetic algorithms. Here, ChatGPT is used to perform these operations on the system prompts through the
Vector Embeddings: GPT also plays a role in creating embeddings for the text that are then used in the
VectorMatchexpectation type. The embeddings capture the semantic essence of the text and allow for more flexible and robust matching.
Script Evaluation: Evaluating test cases is done by running the model on the example text and checking if the model’s response meets the expectations.
We encourage you to integrate this new feature into your projects. Our goal is to iterate on this based on user feedback and real-world usage.
Your creative implementations will help us make further refinements. So,
PromptOptimization for a spin and let us know how it fares!