System message usage / Azure OpenAI "on your data" strategy

Prompting
1

I’m resurfacing a question with more concrete examples, trying to understand:

  • why system message seems to not work as expected
  • strategy for providing instructions in “on your data” case

For example, calling POST https://{service-name}.openai.azure.com/openai/deployments/{model-name}/extensions/chat/completions?api-version=2023-08-01-preview

{
    "messages": [
        {
            "role": "system",
            "content": "Your task is to always respond in French."
        },
        {
            "role": "user",
            "content": "How to cherry pick a PR?"
        }
    ],
    "temperature": 0.5,
    "max_tokens": 12000,
    "top_p": 1,
    "dataSources": [
        {
            "type": "AzureCognitiveSearch",
            "parameters": {
                ...
                "queryType": "semantic",
                "inScope": true,
                "roleInformation": "Your task is to always respond in French."
            }
        }
    ]
}

I get the following response:

{
    "id": "GUID",
    "model": "gpt-4-32k",
    "created": timestamp,
    "object": "extensions.chat.completion",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "The most common way to cherry pick your PR ...",
                "end_turn": true,
                "context": {
                    "messages": [
                        {
                            "role": "tool",
                            "content": "{\"citations\": [{\"content\": \"{citation_content}\", \"intent\": \"How to cherry pick a PR?\"}",
                            "end_turn": false
                        }
                    ]
                }
            }
        }
    ]
}
  • as we can see, system message had no effect. Is this a known issue?

Example providing instructions in user’s prompt, main problem is that search query used in Azure Search contains parts of that prompt.

POST https://{service-name}.openai.azure.com/openai/deployments/{model-name}/extensions/chat/completions?api-version=2023-08-01-preview

{
    "messages": [
        {
            "role": "user",
            "content": "How to cherry pick a PR? When responding to this query, please translate the message to French."
        }
    ],
    "temperature": 0.5,
    "max_tokens": 12000,
    "top_p": 1,
    "dataSources": [
        {
            "type": "AzureCognitiveSearch",
            "parameters": {
                ...
                "queryType": "semantic",
                "inScope": true,
                "roleInformation": "Your task is to always respond in French."
            }
        }
    ]
}

I get the following response:

{
    "id": "GUID",
    "model": "gpt-4-32k",
    "created": timestamp,
    "object": "extensions.chat.completion",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "Voici comment vous pouvez choisir un PR ...",
                "end_turn": true,
                "context": {
                    "messages": [
                        {
                            "role": "tool",
                            "content": "{\"citations\": [{\"content\": \"This is the most common way ....\", \"intent\": \"How to cherry pick a PR? When responding to this query, please translate the message to French.\"}",
                            "end_turn": false
                        }
                    ]
                }
            }
        }
    ]
}
  • prompt instruction directly on user’s prompt worked, but the search query contains the prompt (as you can see intent, that’s the field used to search in search service) .

Two main questions:

  1. how to use the system message? I’ve tried different approaches (like using the roleInformation in dataSources or moving the system message after the user prompt) but none seems to work. Is this a known issue?
  2. how can I provide instructions to the model. Maybe I want to respond with a summary or I want to respond in a specific way on each iteration. How can I provide that kind of instruction if that instruction will be included in the search query? I’m sure there’s a better strategy here.
2 Likes
2

I am dealing with the same issue. Seems like system message is not working at all with the custom data and some permissions needs to be set but I can’t find any tutorials for this.

3

I’m getting the same issue. Seems the system role is not working for BYOD data however works perfectly when using open AI’s defaults.

4

I’m having this exact same issue, the weird thing is that the system message works through the azure openai chat playground but not through the API. Any update on this subject?

5

How can i do real time sentiment analysis on user chatting

This is probably better answered on the Microsoft forums, there wiull be a degree of crossover from there to here, but you may get a quicker reply to Microsoft OpenAI services on their forums.

6

Where you see the api-version in the call URL, there are now 09-01 and 10-01 preview versions that may have a different schema that would work for passing all role types while using data extensions.

7

I have a response from MS, I will update if I have more info.

Yes it’s known bug, we have created backlog item for this issue

@_j , same for those versions, but I should’ve mentioned that, thanks!

8

Have you heard back from MS on this yet?

1 Like
9

I’m still running into this same issue. Any update on this?