Gpt-4-1106-preview Doesn't listen to instructions!

I’m running into a really annoying bug where I ask chatgpt the following

combined_prompt = (
                        "\nHere are the names and contents of all scripts in the framework: \n" + all_scripts_content +
                        "\nHere is the original prompt: \n" + original_prompt +
                        "\nHere is the name of the script that you are modifying: \n" + the_current_script_being_edited +
                        "Output a SINGLE script based on the following debug output: " + sanitized_debug_output +
                        "JUST OUTPUT CODE, do not include ```python or similar! It is going to be run directly and will cause an error!"+
                        "\n IT DOES NOT MATTER IF THE SCRIPT REQUIRES MULTIPLE FILES TO RUN. JUST OUTPUT A SINGLE SCRIPT BASED ON THE NAME OF WHAT YOU ARE MODIFYING!!!"+
                        "\n The reason why is because this framework goes one script at a time, so it is not necessary to include all scripts at once. Just output the script that you are modifying."
                    )

No matter how much I try the output clearly contradicts this in multiple ways!

```python
# Import necessary Django and other libraries
import os
import openai
from django.core.wsgi import get_wsgi_application
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.generic import TemplateView
from django.conf import settings
from django.urls import path
from django.apps import AppConfig

# Environment setup for OpenAI API Key
# SET THIS ENVIRONMENT VARIABLE IN YOUR SERVER SETTINGS, not here in the code.
# Example: export OPENAI_API_KEY='your-api-key'
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

# Django settings
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
settings.configure(
    DEBUG=True,
    SECRET_KEY='replace-this-with-a-real-secret-key',
    ROOT_URLCONF=__name__,
    MIDDLEWARE=[
        'django.middleware.security.SecurityMiddleware',
        'django.middleware.common.CommonMiddleware',
        'django.middleware.csrf.CsrfViewMiddleware',
        'django.middleware.clickjacking.XFrameOptionsMiddleware',
    ],
    TEMPLATES=[
        {
            'BACKEND': 'django.template.backends.django.DjangoTemplates',
            'DIRS': [os.path.join(BASE_DIR, 'templates')],
        },
    ],
    STATIC_URL='/static/',
)

# Django application setup
class ChatbotAppConfig(AppConfig):
    name = 'chatbot_app'
    verbose_name = "Chatbot Application"

# Django template view for the chatbot webpage
class ChatbotView(TemplateView):
    template_name = "chatbot.html"
    
# Function to interact with the GPT-3.5-turbo API endpoint
def ask_gpt3(prompt):
    response = openai.Completion.create(
        engine="gpt-3.5-turbo",
        prompt=prompt,
        max_tokens=150,
        temperature=0.7,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0
    )
    return response.choices[0].text.strip()

# Django view to handle POST requests with the user input
@csrf_exempt
def chat_with_gpt3(request):
    if request.method == 'POST' and request.POST.get('message'):
        user_input = request.POST.get('message')
        gpt3_response = ask_gpt3(user_input)
        return JsonResponse({'response': gpt3_response})
    else:
        return JsonResponse({'error': 'Invalid request'}, status=400)

# URL Patterns for the Django app
urlpatterns = [
    path('', ChatbotView.as_view(), name='chatbot'),
    path('chat/', chat_with_gpt3, name='chat_with_gpt3'),
]

# Standard Django WSGI application setup
application = get_wsgi_application()

# Main entry point
if __name__ == "__main__":
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)

And the accompanying HTML (templates/chatbot.html) for the above Django app:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Chatbot powered by GPT-3.5</title>
    <style>
        body { font-family: Arial, sans-serif; padding: 20px; }
        #chatbox { width: 100%; height: 400px; border: 1px solid #999; overflow: auto; padding: 10px; }
        #userInput { width: 100%; padding: 10px; margin-top: 10px; }
        .userMsg { color: blue; }
        .botMsg { color: green; }
    </style>
    <script>
        // Function to send POST request to backend
        function sendMessage(message) {
            const xhr = new XMLHttpRequest();
            xhr.open("POST", "/chat/", true);
            xhr.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
            xhr.onload = function () {
                if (xhr.status == 200) {
                    const resp = JSON.parse(xhr.responseText);
                    displayMessage(resp.response, "botMsg");
                }
            };
            xhr.send("message=" + encodeURIComponent(message));
        }

        // Function to display message in chatbox
        function displayMessage(message, className) {
            const chatbox = document.getElementById("chatbox");
            const newMsg = document.createElement("div");
            newMsg.innerHTML = message;
            newMsg.className = className;
            chatbox.appendChild(newMsg);
            chatbox.scrollTop = chatbox.scrollHeight; // Auto scroll to the bottom
        }

        // Function to handle user input
        function handleUserInput(e) {
            if (e.keyCode === 13) { // Enter key
                const userInput = document.getElementById("userInput");
                const userMessage = userInput.value;
                userInput.value = '';
                if(userMessage.trim() !== '') {
                    displayMessage(userMessage, "userMsg");
                    sendMessage(userMessage);
                }
            }
        }
    </script>
</head>
<body>
    <h1>GPT-3.5 powered chatbot</h1>
    <div id="chatbox"></div>
    <input type="text" id="userInput" placeholder="Ask me something..." onkeydown="handleUserInput(event)">
</body>
</html>

Remember to set your OpenAI API key as an environment variable before running your Django application.

For security reasons, please ensure you generate a new Django secret key for your deployment. Also, consider adding user authentication and production-grade settings for a real-world deployment.

This is really frustrating. I just want it to output CODE. I'm sure many others have encountered this issue but I can't seem to make it stop! I've reformatted the prompt many times over.

It doesn’t show the format well but all of what came after what I said and before the small block at the bottom is what it output. This contains ```python, summaries, and multiple scripts. It is failing 3 ways.

:thinking:

what do you want the output to actually look like? do you want it to include the html code as a string in the python script?

How is the model supposed to accomodate what you want it to do?

If you gave the prompt to a human, what would they do? Sometimes the model not doing what you’d expect it to do is the equivalent of a human throwing their hands up in the air and telling you that they give up…

1 Like

My previous version of the script worked, it is supposed to take the scripts as input with a debug file and formulate an output script. It should only focus on one file at a time as I have a handler which provides the name and contents of the singular file to modify.

This is the working prompt from Gen 2.

combined_prompt = (
                        "Modify the script based on the following debug output: " + sanitized_debug_output +
                        "\nHere is the latest script: \n" + last_script_content +
                        "\nHere is the original prompt: \n" + original_prompt
                    )

The big difference now is that it has to take in multiple scripts as context and evidently it forgets its instructions in the process. It needs to retain that information somehow.

Ah, so the actual issue is that this line

"\nHere is the name of the script that you are modifying: \n" + the_current_script_being_edited +

isn’t getting respected as much as it should? It’s kind of buried in there, ngl.

because my question was, how is the model supposed to know which file it should work on?

1 Like

I’ll increase the forum comprehension:

Here are the names and contents of all scripts in the framework:
[contents of all_scripts_content]

Here is the original prompt:
[contents of original_prompt]

Here is the name of the script that you are modifying:
[contents of the_current_script_being_edited]

Output a SINGLE script based on the following debug output: [contents of sanitized_debug_output]

JUST OUTPUT CODE, do not include `python or similar! It is going to be run directly and will cause an error!

IT DOES NOT MATTER IF THE SCRIPT REQUIRES MULTIPLE FILES TO RUN. JUST OUTPUT A SINGLE SCRIPT BASED ON THE NAME OF WHAT YOU ARE MODIFYING!!!

The reason why is because this framework goes one script at a time, so it is not necessary to include all scripts at once. Just output the script that you are modifying.

However, I don’t see combined_prompt employed anywhere in code…

Nor can this possibly work, nor does it include the title’s model:

response = openai.Completion.create(
    engine="gpt-3.5-turbo",
    prompt=prompt,

So I’m at a loss as to what you are actually sending and what you expected and what you got back that was unsatisfactory.

This is part of a larger framework. I have not included the complete script. I am also not using gpt-3.5-turbo.

The code I showed is the output of the prompt I gave it, with the exception of the combined_prompt I generated.

It is also necessary for it to take in all scripts as context with the use case I have. The gpt 4 turbo preview endpoint can fit entire frameworks into it’s context window.

this might be so, but just because it can theoretically hold 128 kilotokens doesn’t mean that it can pay attention to every single concept at the same time.

the model has a limited attention budget, and that’s what you need to work with :confused:

you can’t just throw the kitchen sink at it and hope to get a gourmet meal out of it - it requires some effort :frowning:

some call it prompt engineering, some call it data science :grimacing:


the most immediate thing I would advise is to put the instruction to pay attention to a single file at the end of the prompt. That’s a place that the model pays a lot of attention to.

And I would actually allow the model to wrap the code in backtick blocks - you can just strip them later. you can also add a stop sequence for ```\n or something if you want.

3 Likes

I’ll try your suggestion of parsing the backticks and let you know how it goes. It may end up simpler in the end.

Thanks.

1 Like

Good luck! Let us know how it went :smiley:

1 Like

Another thing you might want to try is re-formatting your prompt using markup.

My applications send large amounts of information in their LLM prompts, and while gpt-4-1106-preview (I’ve not seen the issues you’re having) reads them fairly well, gpt-3.5-turbo-16k consistently failed to comprehend. I improved it’s comprehension by sending it the prompts formatted in XML. API Prompt for gpt-3.5-turbo-16k - #12 by SomebodySysop

1 Like