How to get Chat API to respond only with one of the given choices?

Prompting
,
1

I pass in the system prompt with a list of specific addresses like below:

You are given a text message. Determine whether the text message mentions one of the specified addresses below: 

- 123 NW 56TH ST, Seattle
- 789 60TH ST, San Francisco
[20 more addresses]

If the text message mentions one of the addresses above, you will output the exact address from the list above. You must not output an address that is not in the above list.
If the text message doesn't refer to any of the addresses, you will output "Not mention any address"

You will output only plain text. You will not use Markdown.

Then, the user prompt is an unstructured text message like: “I’m at 123 doing stuff” and “123 NW 64TH ST right now to look at windows”.

Interestingly, “I’m at 123 doing stuff” gets the Chat API to return the exact address in the list but “123 NW 64TH ST right now to look at windows” yields “123 NW 64TH ST”, which is missing “, Seattle”

I’m still new to prompting. Would love to learn more tricks and tips to get LLM / Chat API to return the exact wording of one of the given choices in the system prompt.

Thank you.

2

Which model are you using? My guess would be 3.5… Give it a try with GPT4

3

Hi @ninat ,
prompt engineering is about trying, iterating, and improving your prompts and approaches. You can try:

  1. different models,
  2. splitting your task into two parts:
    2.1 extract the address from a given text;
    2.2. go through the given list (of 20 addresses) and determine it contains this (extracted) address.
  3. Split your task into actually two LLM functions, means - two models, each with it’s own prompt - one would extract the address, another model would be tasked to go through your list of addresses and answer with the address if the list contains it’s exact match or output “Not mention any address” if there’s no exact match.
  4. Read about Routing - LangChain supports it. You can use routing by semantic similarity or custom functions.

And actually, if you really need an exact match, why don’t you just use custom function to route between different outputs?

Example:

def route(info):
    if "anthropic" in info["topic"].lower():
        return anthropic_chain
    elif "langchain" in info["topic"].lower():
        return langchain_chain
    else:
        return general_chain