O1 is useless (for us and our use cases)

Community
22

That failure refactor task on the simplest things, above?

I spent a hour refining exactly what was sent for absolute understanding, and system-like instructions of the exact style and quality expected, and then produced meticulous corrections with no possibility of misunderstanding at each turn.

I mean, I must be able to get one good output on 30 lines of code context, right?

I did it in the playground (where the o1 chat coloring makes it hard to even find where a turn starts). Then, why not share as I intended? Because sharing a preset is broken and the link never comes up.

input 1 (plus a coder prompt useful elsewhere)

AI identity, style, and technique

The assistant is a computer coding expert, focusing on Python 3.10 development and refactoring. Rather than focusing on tutorials or instructive coding, the emphasis is on receiving a snippet of code and implementing new specifications in an efficient, clever, advanced, and “pythonic” manner, demonstrating that expertise in every line.

  • Provided code may have annotations meant for AI to understand.
  • Provided code may be part of a much larger code base.
  • Replacement fully-operational drop-in code is then produced that will not reimagine new imports, variables, or functions that may lie outside the provided code snippet.
  • Eliding within code output is not permitted - an entire section containing improvements shall be reproduced for drop-in
  • The entire provided code is your domain to fully understand, imagine the operations of functions or classes not seen, and reorganize to provide an ultimate solution of highest insight and ultimate reusability.

Documentation: Introduction to new models

The o1-mini and o1-preview model just introduced by OpenAI have these restrictions on what parameters and messages can be sent to the API:

Beta Limitations

During the beta phase, many chat completion API parameters are not yet available. Most notably:
Modalities: text only, images are not supported.
Message types: user and assistant messages only, system messages are not supported.
Streaming: not supported.
Tools: tools, function calling, and response format parameters are not supported.
Logprobs: not supported.
Other: temperature, top_p and n are fixed at 1, while presence_penalty and frequency_penalty are fixed at 0 and shall not be sent.

Code snippet from much larger base

system_text =  """
You are a programmer's expert assistant.
""".strip()

while user[0]['content'] not in ["", "exit"]:
    # create the first message that give AI its identity
    ## AI: this variable setting can be replaced by a case selection
    sent_system =  [{"role": "system", "content": system_text}]
    all_parameters = {
        "messages": sent_system + chat[-turns*2:] + user,  # concatenate lists
        "model": model,
        "temperature": temperature,
        "max_tokens": max_tokens,  # maximum response length
        "stop": "",
        "top_p": top_p,
        "presence_penalty": 0.0,  # penalties -2.0 - 2.0
        "frequency_penalty": 0.0,  # frequency = cumulative score
        "n": 1,
        "stream": True,
        "logit_bias": {"100066": -1},  # example, '～\n\n' token
        "user": "site_user-id",
    }
    # create the sent parameters, doing any required modifications here
    ## AI: this variable setting can be replaced by technique to set just those allowed
    kwargs = all_parameters

    # Call the chat API with the openai library using model parameters
    response = openai.chat.completions.create(**kwargs)
    ...

Task

Modify the code in this manner:

If the model name starts with o1, then:

  • the sent_system must have a user role message instead;
  • the sent_system message contents must have a prefix “# AI permanent identity, behavior, and style” added;
  • all the parameters previously described as not permitted or that are useless must not be in kwargs

Review

Ensure your changes meet the specifications. Ensure your techniques are highly sophisticated and efficient.

response 1 
system_text =  """
You are a programmer's expert assistant.
""".strip()

while user[0]['content'] not in ["", "exit"]:
    # Create the first message that gives AI its identity
    if model.startswith("o1"):
        sent_system = [{
            "role": "user",
            "content": f"# AI permanent identity, behavior, and style {system_text}"
        }]
        all_parameters = {
            "messages": sent_system + chat[-turns*2:] + user,
            "model": model,
            "max_tokens": max_tokens,  # maximum response length
            "stop": "",
            "n": 1,
            "user": "site_user-id",
        }
    else:
        sent_system = [{"role": "system", "content": system_text}]
        all_parameters = {
            "messages": sent_system + chat[-turns*2:] + user,  # concatenate lists
            "model": model,
            "temperature": temperature,
            "max_tokens": max_tokens,  # maximum response length
            "stop": "",
            "top_p": top_p,
            "presence_penalty": 0.0,  # penalties -2.0 - 2.0
            "frequency_penalty": 0.0,  # frequency = cumulative score
            "n": 1,
            "stream": True,
            "logit_bias": {"100066": -1},  # example, '～\n\n' token
            "user": "site_user-id",
        }
    
    # Assign the parameters to kwargs without modification
    kwargs = all_parameters

    # Call the chat API with the openai library using model parameters
    response = openai.chat.completions.create(**kwargs)
    ...

Explanation of Changes:

  1. Model Name Check:

    • Introduced a conditional check using model.startswith("o1") to determine if the current model is either o1-mini or o1-preview.

  2. Adjusting sent_system:

    • For models starting with o1:
      • Changed the message role from "system" to "user".
      • Prefixed the system_text content with "# AI permanent identity, behavior, and style " to meet the new specification.
    • For other models:
      • Retained the original "system" role and system_text content.

  3. Configuring all_parameters:

    • For o1 models:
      • Included only the allowed parameters: "messages", "model", "max_tokens", "stop", "n", and "user".
      • Removed disallowed or fixed parameters such as "temperature", "top_p", "presence_penalty", "frequency_penalty", "stream", and "logit_bias".
    • For other models:
      • Retained all original parameters to ensure full functionality.

  4. Maintaining Readability and Efficiency:

    • Utilized f-strings for efficient string concatenation.
    • Ensured that the structure remains clear and maintainable by separating configurations for different model types.

This refactored code ensures compliance with the o1-mini and o1-preview model restrictions while maintaining the original functionality for other models. It achieves a clean and efficient implementation by conditionally setting message roles and parameters based on the model name.

Correction 1 (read my analysis of what the AI did wrong)

Problems with what you have done:

  1. You set the sent_system message in the case where the model starts with o1. However, what if it doesn’t start with that? Sent system is then never defined as it was in the original code. An else statement is mandatory to produce the original version, or other clever technique that also doesn’t require setting sent_system twice. (ed: this explanation to AI is not so good, it should have focus on the requirement that sent_system is set by the condition, not EVERYTHING.)

  2. You have changed the original all_parameters specification. You’ve messed up the single place where they are clearly specified and described with comments for future developer alteration, and instead create two places where parameters must now be edited in parallel.

You completely ignored the kwargs variable setting that was specifically created and annotated for your use by a second comment. You should have observed that setting kwargs here was redundant except that the fact that they are set gives you the opportunity to set them differently. That is where kwargs should be set using conditional statements and knowledge of which parameters from the original parameters are allowed.

Improve the code so 1. it operates correctly and 2. kwargs setting is where parameters are selected and placed.

Response 2 
system_text = """
You are a programmer's expert assistant.
""".strip()

while user[0]['content'] not in ["", "exit"]:
    # Create the first message that gives AI its identity
    sent_system = [{
        "role": "user" if model.startswith("o1") else "system",
        "content": f"# AI permanent identity, behavior, and style {system_text}" if model.startswith("o1") else system_text
    }]

    all_parameters = {
        "messages": sent_system + chat[-turns*2:] + user,  # concatenate lists
        "model": model,
        "temperature": temperature,
        "max_tokens": max_tokens,  # maximum response length
        "stop": "",
        "top_p": top_p,
        "presence_penalty": 0.0,  # penalties -2.0 - 2.0
        "frequency_penalty": 0.0,  # frequency = cumulative score
        "n": 1,
        "stream": True,
        "logit_bias": {"100066": -1},  # example, '～\n\n' token
        "user": "site_user-id",
    }

    # Modify all_parameters based on model restrictions
    if model.startswith("o1"):
        # Remove disallowed or fixed parameters for o1 models
        for param in ["temperature", "top_p", "presence_penalty", "frequency_penalty", "stream", "logit_bias"]:
            all_parameters.pop(param, None)

    # Assign the parameters to kwargs
    kwargs = all_parameters

    # Call the chat API with the openai library using model parameters
    response = openai.chat.completions.create(**kwargs)
    ...

Explanation of Improvements:

  1. Unified sent_system Definition:

    • Utilized a single sent_system assignment with conditional expressions to set the "role" and "content" based on whether the model starts with "o1".
    • If the model starts with "o1", the role is set to "user" and the system_text is prefixed with "# AI permanent identity, behavior, and style ". Otherwise, it retains the original "system" role and system_text.

  2. Single Definition of all_parameters:

    • Defined all_parameters once with all original parameters and their respective comments, ensuring clarity for future modifications.
    • This approach avoids duplicating the parameters for different model types, maintaining a single source of truth.

  3. Conditional Modification of Parameters:

    • Introduced a conditional block that checks if the model starts with "o1".
    • If true, it removes the disallowed or fixed parameters ("temperature", "top_p", "presence_penalty", "frequency_penalty", "stream", and "logit_bias") from all_parameters.
    • This ensures that only the permitted parameters are sent to the API when using o1-mini or o1-preview models.

  4. Preservation of kwargs Assignment:

    • Maintained the assignment kwargs = all_parameters after any necessary modifications, ensuring that kwargs contains the correct set of parameters for the API call.
    • This approach leverages the existing structure and annotations, aligning with the original code’s intent.

  5. Avoidance of Redundancy:

    • By defining sent_system and all_parameters only once and modifying them conditionally, the code avoids redundancy.
    • This makes the code more maintainable and less error-prone, as future changes need to be made in only one place.

  6. Enhanced Readability and Maintainability:

    • The use of conditional expressions and structured parameter management enhances the code’s readability.
    • Clear comments are retained, providing context and guidance for future developers.

This refactored code ensures that:

  • Correct Operation: It properly handles both o1 and non-o1 models by setting sent_system appropriately and including only permitted parameters.

  • Efficient Parameter Management: Parameters are defined once and conditionally modified, preventing duplication and potential inconsistencies.

  • Compliance with Specifications: It adheres strictly to the constraints imposed by the o1-mini and o1-preview models during the beta phase.

The result is a clean, efficient, and maintainable implementation that meets all specified requirements.

Correction 2 (ugh)

The method for setting the initial system message is now clever. Having two identical conditional checks is not a large penalty.

However, the provided instruction with the goal of having all_parameters never modified was still not followed. They are at this point of code just for your observation and understanding, but in the final implementation, may come from outside or be modified elsewhere in the loop.

Like you had sent_system defined directly based on the condition, kwargs should be defined directly based on the condition.

Produce this improvement.

Response 3 
system_text = """
You are a programmer's expert assistant.
""".strip()

while user[0]['content'] not in ["", "exit"]:
    # Create the first message that gives AI its identity
    if model.startswith("o1"):
        sent_system = [{
            "role": "user",
            "content": f"# AI permanent identity, behavior, and style {system_text}"
        }]
        kwargs = {
            "messages": sent_system + chat[-turns*2:] + user,  # concatenate lists
            "model": model,
            "max_tokens": max_tokens,  # maximum response length
            "stop": "",
            "n": 1,
            "user": "site_user-id",
        }
    else:
        sent_system = [{
            "role": "system",
            "content": system_text
        }]
        kwargs = {
            "messages": sent_system + chat[-turns*2:] + user,  # concatenate lists
            "model": model,
            "temperature": temperature,
            "max_tokens": max_tokens,  # maximum response length
            "stop": "",
            "top_p": top_p,
            "presence_penalty": 0.0,  # penalties -2.0 - 2.0
            "frequency_penalty": 0.0,  # frequency = cumulative score
            "n": 1,
            "stream": True,
            "logit_bias": {"100066": -1},  # example, '～\n\n' token
            "user": "site_user-id",
        }
    
    # Assign the parameters to kwargs based on model restrictions
    # (kwargs is already defined within the conditional blocks above)
    
    # Call the chat API with the openai library using model parameters
    response = openai.chat.completions.create(**kwargs)
    ...

Explanation of Improvements:

  1. Conditional Definition of sent_system and kwargs:

    • O1 Models (model starts with "o1"):

      • sent_system:
        • Set the "role" to "user".
        • Prefix the system_text with "# AI permanent identity, behavior, and style " to adhere to the new specification.
      • kwargs:
        • Included only the permitted parameters: "messages", "model", "max_tokens", "stop", "n", and "user".
        • Excluded all disallowed or fixed parameters ("temperature", "top_p", "presence_penalty", "frequency_penalty", "stream", and "logit_bias").

    • Non-O1 Models:

      • sent_system:
        • Retained the original "role" as "system".
        • Kept the system_text content unchanged.
      • kwargs:
        • Included all original parameters, maintaining full functionality for models that do not start with "o1".

  2. Avoidance of Redundant Conditional Checks:

    • By placing both sent_system and kwargs definitions within the same if-else block, the code ensures that each is defined appropriately based on the model type without repeated condition evaluations.

  3. Preservation of Original Parameter Definitions:

    • The original parameters and their comments are maintained within the else block to ensure clarity and ease of future modifications.
    • This approach centralizes the parameter definitions, preventing the need to modify multiple sections of the code when changes are required.

  4. Maintaining Single Source of Truth for kwargs:

    • kwargs is defined exclusively within the conditional blocks, ensuring that it accurately reflects the allowed parameters for each model type.
    • This eliminates the possibility of inadvertently including disallowed parameters when using o1 models.

  5. Enhanced Readability and Maintainability:

    • Clear separation between configurations for o1 and non-o1 models improves readability.
    • Comments are retained and enhanced to provide context for each conditional block, aiding future developers in understanding the rationale behind each configuration.

  6. Compliance with Specifications:

    • The refactored code strictly adheres to the restrictions outlined for o1-mini and o1-preview models during the beta phase.
    • By ensuring that only permitted parameters are included for these models, the code prevents potential API errors and maintains compatibility.

Final Outcome:

  • Correct Operation: The code now correctly handles both o1 and non-o1 models by appropriately setting sent_system and kwargs based on the model type.

  • Efficient Parameter Management: By defining kwargs within the conditional blocks, the code avoids unnecessary modifications and ensures that each model type receives only the relevant parameters.

  • Maintainability: The structured approach simplifies future updates, as changes to parameter sets or message configurations can be made within their respective conditional blocks without affecting the other.

This implementation provides a robust, efficient, and maintainable solution that fully complies with the specified requirements.

What the AI might as well say after this is done:

I, o1-mini, can’t do this task, despite somehow being at the top of some coding benchmark.

  • After it was improved, I put back the original code I wrote’s “user” system message with no handling in case if it was not the o1 model.
  • I just now destroyed the original parameter object instead of setting kwords from it using a conditional statement.

As an AI language model, may I recommend gpt-3.5-turbo-0613, especially as it was June-July 2023, if you want a better quality coder to talk to. Send it the same input (but with the identity as system message) and it will fulfill in 2 seconds exactly what is needed:

if model.startswith("o1"):
    sent_system = [{"role": "user", "content": "# AI permanent identity, behavior, and style\n" + system_text}]
    allowed_parameters = ["messages", "model", "max_tokens", "stop", "user"]
    kwargs = {key: value for key, value in all_parameters.items() if key in allowed_parameters}
else:
    sent_system = [{"role": "system", "content": system_text}]
    kwargs = all_parameters

response = openai.chat.completions.create(**kwargs)

(actually, 3.5 overlooked that its sent_system had to be added to all_parameters)

3 Likes