PromptOptimization: Genetic Algorithms for Prompt Optimization

Prompting
,
1

PromptOptimization: Genetic Algorithms for Prompt Optimization

Background

We’ve been focused on making the interaction with chatbots more reliable, efficient, and adaptive. With the GPT-3.5 Turbo model providing us with vast capabilities, we set our sights on enhancing the dynamism and responsiveness of conversational agents. The PromptOptimization class is designed to make your chatbot conversations smarter by leveraging genetic algorithms, test cases, and evaluation metrics.

Problem Statement

Crafting the perfect system prompt to get a desirable response from the chatbot often requires trial and error, intuition, or a deep understanding of how GPT models work. A prompt that works well for one task may not work as well for another. We aimed to address this variability by introducing a new optimization layer: PromptOptimization.

Strategy

The main approach is inspired by genetic algorithms where mutations and recombinations are used to evolve a better “species” (or in our case, a more effective prompt). The code is split into four significant areas:

  1. Test Cases: Defines the input message and the expected response.
  2. RunGeneticGenerations: Orchestrates the genetic evolution of system prompts.
  3. Evaluate: Provides a scoring mechanism based on how closely the output matches expectations.
  4. GeneticApi: Interface for mutation and recombination, implemented using a proxy for ChatGPT.

Example Usage

Import the Library:

implementation 'com.simiacryptus:joe-penai:1.0.22'

Define the Test Cases:

import com.simiacryptus.openai.OpenAIClient
import com.simiacryptus.openai.opt.Expectation
import com.simiacryptus.openai.opt.Expectation.*
import com.simiacryptus.openai.opt.PromptOptimization
import com.simiacryptus.openai.opt.PromptOptimization.*

val api = OpenAIClient()
val promptOpt = PromptOptimization(api)

val testCases = listOf(
    TestCase(
        turns = listOf(
            Turn("tell me a joke", listOf(ContainsMatch(Regex("Why did.*"))))
        )
    )
)

val systemPrompts = listOf("Make the user laugh", "Tell a joke")
val topPrompts = promptOpt.runGeneticGenerations(systemPrompts, testCases)

Turn/Expectation Walkthrough

  1. User Message: The input from the user.
  2. Expectations: A list of conditions the response should meet.

Expectation Types:

  • VectorMatch: Utilizes vector embeddings to compare the semantic similarity between an example string and the model’s response.

    VectorMatch("tell me a joke").score(api, response)

  • ContainsMatch: Searches for a regex pattern in the chatbot’s response.

    ContainsMatch(Regex("Why did.*")).matches(api, response)

How GPT is involved?

  • Mutation and Recombination: These are essential aspects of genetic algorithms. Here, ChatGPT is used to perform these operations on the system prompts through the GeneticApi interface.

  • Vector Embeddings: GPT also plays a role in creating embeddings for the text that are then used in the VectorMatch expectation type. The embeddings capture the semantic essence of the text and allow for more flexible and robust matching.

  • Script Evaluation: Evaluating test cases is done by running the model on the example text and checking if the model’s response meets the expectations.

What’s Next?

We encourage you to integrate this new feature into your projects. Our goal is to iterate on this based on user feedback and real-world usage.

Your creative implementations will help us make further refinements. So,
take PromptOptimization for a spin and let us know how it fares!