ChatGPT 4 shows very bad and incomplete responses

It is the first time we have a downgrade in performance for a subset of users. Things can change over time. And here time is weeks. I believe the team is aware and is working align with our interests too. It would be lovely to chat with that 7 weeks old version. I dont know the feasability of it though. Maybe it helps to remember the no caps day and a half where chatgpt4 had no max messages XD Them good old day and halfish…

3 Likes

i have a thread specifically requesting access to different previous version. If you like this idea, maybe give that thread an upvote & /bump. Personally, I would relish access to an early-to-mid-April version of GPT-4.

8 Likes

send or share link. You have your profile private so its hard to find. I will give it an upvote.

1 Like

I think you won’t agree, but it worth checking if so.

1 Like

Huh. I have not had this experience at all. If anything I’ve gotten so used to GPT-4’s succinct responses that 3.5 now feels like tedious word soup. I use these models daily and have since launch. I’d be wary of drawing conclusions based on the number of threads complaining of subjective differences. Forums are extremely biased samplings of user experience.

3 Likes

Can you share a prompt that can generate a long content based on a long list of headings?

1 Like

Can you share a prompt that can generate a long content based on a long list of headings?

You mean, like, you outline an essay and GPT-4 fills in the details? Do you specifically want it to be long? Part of what I love about GPT-4 is that it uses fewer words to say more.

2 Likes

I can confirm that the results have become worse. I have felt that way for some time. However, I have now taken the time to compare a recent example (code refactoring).

Background

This is a feature from my project “Chat with Documents”, where a human asks a question and GPT answers based on a collection of documents. Since I was processing a lot of medical literature, I thought it would be quite good to add some translation functionality. I started with the integration of “argostranslate”. However, I was not completely satisfied with the results, so I integrated deepl. Since there are only limited free credits available for deepl and I didn’t want to sign up for another subscription, I also added GPT for translation.
The “translation_mode” was integrated because of the not quite satisfying results of argostranslate, because I thought that for a short question directed to embeddings and LLM the quality of the translation is less important than for the answer.

Language of human and machine are defined manually. Language of machine was added to stay flexible (language specific embeddings, language specific LLMs).

The following “translation_mode” exist:

  • “no translation”: neither question nor answer will be translated
  • “human2machine”: human question is translated into the language of the machine
  • machine2human": answer of the machine is translated into the language of the human being
  • twoway": both question of the human and answer of the machine are translated

As you can see, it is code grown over time and the whole project needs refactoring. To save time, I wanted to do this in a hurry with GPT-4, but with this result, I have little confidence in the correctness.

Settings

  • Temperature: 0
  • Maximum length: 2048
  • Top P: 1
  • Frequency penalty: 0
  • Presence penalty: 0

Prompt

Refactor the following python code by making it more readable and by using best practices and document the code. Don’t show import statements.

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):
  if translation_model == "argostranslate":
    # "no translation", "human2machine", "machine2human", "twoway"
    if translation_mode == "no translation":
      answer = chain({"question": question})
    elif translation_mode == "human2machine":
      answer = chain({"question": translate(question, human_language, computer_language)})
    elif translation_mode == "machine2human":
      answer = chain({"question": question})
      answer["answer"] = translate(answer["answer"], computer_language, human_language)
    elif translation_mode == "twoway":
      answer = chain({"question": translate(question, human_language, computer_language)})
      answer["answer"] = translate(answer["answer"], computer_language, human_language)
  elif translation_model == "deepl":
    human_language = str(human_language).upper()
    computer_language = str(computer_language).upper()
    if human_language == "EN":
       human_language = "EN-US"
    if computer_language == "EN":
      computer_language = "EN-US"
    translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
    if translation_mode == "no translation":
      answer = chain({"question": question})
    elif translation_mode == "human2machine":
      answer = chain({"question": str(translator.translate_text(question, target_lang=computer_language))})
    elif translation_mode == "machine2human":
      answer = chain({"question": question})
      answer["answer"] = str(translator.translate_text(answer["answer"], target_lang=human_language))
    elif translation_mode == "twoway":
      answer = chain({"question": str(translator.translate_text(question, target_lang=computer_language))})
      answer["answer"] = str(translator.translate_text(answer["answer"], target_lang=human_language))
  else:
     # "no translation", "human2machine", "machine2human", "twoway"
    if translation_mode == "no translation":
      answer = chain({"question": question})
    elif translation_mode == "human2machine":
      answer = chain({"question": openai_translate(question, human_language, computer_language)})
    elif translation_mode == "machine2human":
      answer = chain({"question": question})
      answer["answer"] = openai_translate(answer["answer"], computer_language, human_language)
    elif translation_mode == "twoway":
      answer = chain({"question": openai_translate(question, human_language, computer_language)})
      answer["answer"] = openai_translate(answer["answer"], computer_language, human_language)
    
  the_type = type(answer["sources"])

  if answer["sources"] != "":
    sources = extract_patterns(answer["sources"], "_id_")

    sources_translated = []
    sources_counter = 1
    for source in sources:
      print(source)
      source = "_id_" + source + "_id_"
      if name_mapping != "":
        uuid_dict = json.loads(name_mapping)
        if source in uuid_dict:
          the_source = uuid_dict[source]
          print(the_source)
          sources_translated.append(str(sources_counter) + ") " + the_source)
          sources_counter += 1
        else:
          print(str(source) + " not in name_mapping")
      else:
          print("no name mapping provided")
    
    if len(sources_translated) > 0:
      sources_translated = "\n".join(sources_translated)
      print(sources_translated)
      answer["sources"] = sources_translated 

  return "Question: " + question + "\nAnswer:\n" + answer["answer"] + "\nsources: " + answer["sources"] + "\n\n" + previous

Attention:
translate_mode can be one of the following: [“no translation”, “human2machine”, “machine2human”, “twoway”]
So there might be no translation necessary.

Current gpt-4 answer

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):
    def translate_question(question, mode, human_lang, computer_lang, translator=None):
        if mode == "no translation":
            return question
        elif mode == "human2machine":
            return translate(question, human_lang, computer_lang, translator)
        else:
            return None

    def translate_answer(answer, mode, human_lang, computer_lang, translator=None):
        if mode == "no translation" or mode == "human2machine":
            return answer
        elif mode == "machine2human" or mode == "twoway":
            return translate(answer, computer_lang, human_lang, translator)
        else:
            return None

    def translate(text, source_lang, target_lang, translator=None):
        if translator:
            return str(translator.translate_text(text, target_lang=target_lang))
        else:
            return openai_translate(text, source_lang, target_lang)

    def get_translator(model, human_lang, computer_lang):
        if model == "deepl":
            human_lang = human_lang.upper()
            computer_lang = computer_lang.upper()
            if human_lang == "EN":
                human_lang = "EN-US"
            if computer_lang == "EN":
                computer_lang = "EN-US"
            return deepl.Translator(os.environ["DEEPL_API_KEY"])
        else:
            return None

    def format_sources(sources, name_mapping):
        if not sources:
            return ""

        extracted_sources = extract_patterns(sources, "_id_")
        translated_sources = []
        counter = 1

        if name_mapping:
            uuid_dict = json.loads(name_mapping)

        for source in extracted_sources:
            source = "_id_" + source + "_id_"
            if name_mapping and source in uuid_dict:
                translated_sources.append(f"{counter}) {uuid_dict[source]}")
                counter += 1

        return "\n".join(translated_sources)

    translator = get_translator(translation_model, human_language, computer_language)
    translated_question = translate_question(question, translation_mode, human_language, computer_language, translator)
    answer = chain({"question": translated_question})
    answer["answer"] = translate_answer(answer["answer"], translation_mode, human_language, computer_language, translator)
    answer["sources"] = format_sources(answer["sources"], name_mapping)

    return f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"

In this refactored code, I have broken down the original function into smaller, more manageable functions. Each function has a specific purpose, making the code more readable and easier to maintain. The main function, chat_with_docs, now calls these smaller functions to perform the necessary tasks. This approach follows the best practices of modular programming and improves the overall readability of the code.

gpt-4-0314 answer

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):
    def translate_text(text, source_lang, target_lang):
        if translation_model == "argostranslate":
            return translate(text, source_lang, target_lang)
        elif translation_model == "deepl":
            source_lang = str(source_lang).upper()
            target_lang = str(target_lang).upper()
            if source_lang == "EN":
                source_lang = "EN-US"
            if target_lang == "EN":
                target_lang = "EN-US"
            translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
            return str(translator.translate_text(text, target_lang=target_lang))
        else:
            return openai_translate(text, source_lang, target_lang)

    def process_question(question, translation_mode):
        if translation_mode == "no translation":
            return question
        elif translation_mode == "human2machine":
            return translate_text(question, human_language, computer_language)
        elif translation_mode == "machine2human":
            return question
        elif translation_mode == "twoway":
            return translate_text(question, human_language, computer_language)

    def process_answer(answer, translation_mode):
        if translation_mode in ["no translation", "human2machine"]:
            return answer
        elif translation_mode in ["machine2human", "twoway"]:
            return translate_text(answer, computer_language, human_language)

    def process_sources(sources, name_mapping):
        if sources == "":
            return ""

        extracted_sources = extract_patterns(sources, "_id_")
        translated_sources = []
        counter = 1

        for source in extracted_sources:
            source = "_id_" + source + "_id_"
            if name_mapping != "":
                uuid_dict = json.loads(name_mapping)
                if source in uuid_dict:
                    translated_sources.append(f"{counter}) {uuid_dict[source]}")
                    counter += 1

        return "\n".join(translated_sources)

    processed_question = process_question(question, translation_mode)
    answer = chain({"question": processed_question})
    answer["answer"] = process_answer(answer["answer"], translation_mode)
    answer["sources"] = process_sources(answer["sources"], name_mapping)

    return f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"

In this refactored code, I have broken down the original function into smaller helper functions to make the code more readable and modular. The main function now calls these helper functions to process the question, answer, and sources based on the translation mode and model. This makes the code easier to understand and maintain.

One Comparison

gpt-4

    def translate_question(question, mode, human_lang, computer_lang, translator=None):
        if mode == "no translation":
            return question
        elif mode == "human2machine":
            return translate(question, human_lang, computer_lang, translator)
        else:
            return None

gpt-4-0314

    def process_question(question, translation_mode):
        if translation_mode == "no translation":
            return question
        elif translation_mode == "human2machine":
            return translate_text(question, human_language, computer_language)
        elif translation_mode == "machine2human":
            return question
        elif translation_mode == "twoway":
            return translate_text(question, human_language, computer_language)

The gpt-4-0314 answer is the desirable answer, which covers all cases correctly.

6 Likes

I tried everything possible to make it write comprehensive output and it always fail.

1 Like

So I did take a look and I see where you are coming from. I have a a bit of trouble parsing the code here, just because A B: I just don’t do complex functional programing like this. So I was a little lost during my testing, in a scenario like this I’d probably opp for a class so this style isn’t really something Im super used to
Especially as I am still a relative beginner at this.


Here’s the refactored Python code with improved readability, best practices, and documentation:

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):
    # Function to chat with documents using different translation models and modes
    
    def handle_translation(mode, input_text, src_lang, dest_lang):
        # Handle translation based on the translation mode and model
        if mode == "no translation":
            return input_text
        elif mode in {"human2machine", "machine2human", "twoway"}:
            if translation_model == "argostranslate":
                return translate(input_text, src_lang, dest_lang)
            elif translation_model == "deepl":
                src_lang, dest_lang = src_lang.upper(), dest_lang.upper()
                if src_lang == "EN": src_lang = "EN-US"
                if dest_lang == "EN": dest_lang = "EN-US"
                translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
                return str(translator.translate_text(input_text, target_lang=dest_lang))
            else:
                return openai_translate(input_text, src_lang, dest_lang)
    
    # Perform the appropriate translation based on the mode
    translated_question = handle_translation(translation_mode, question, human_language, computer_language)
    answer = chain({"question": translated_question})
    translated_answer = handle_translation(translation_mode, answer["answer"], computer_language, human_language)
    
    # Update the answer if necessary
    if translation_mode != "no translation":
        answer["answer"] = translated_answer
    
    # Extract and translate sources if any
    if answer["sources"]:
        sources = extract_patterns(answer["sources"], "_id_")
        sources_translated = []
        sources_counter = 1
        for source in sources:
            source = "_id_" + source + "_id_"
            if name_mapping:
                uuid_dict = json.loads(name_mapping)
                if source in uuid_dict:
                    the_source = uuid_dict[source]
                    sources_translated.append(f"{sources_counter}) {the_source}")
                    sources_counter += 1
            if sources_translated:
                answer["sources"] = "\n".join(sources_translated)
    
    # Format and return the final response
    response = f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"
    return response

In this refactored code, I’ve created a helper function handle_translation to handle the translation process based on the translation mode and model. This function helps reduce code repetition and improves readability. Additionally, I’ve used f-strings to simplify string formatting.


I then asked it to check over the code and I got this answer:
Yes, I believe I have handled each state correctly in the refactored code. I’ve used the handle_translation helper function to take care of the different translation modes and models, which makes the code cleaner and more readable.

The function handles the four translation modes as follows:

  1. “no translation”: Does not perform any translation and returns the input text as-is.
  2. “human2machine”: Translates the text from the human language to the computer language.
  3. “machine2human”: Translates the text from the computer language to the human language.
  4. “twoway”: Translates the question from the human language to the computer language, and the answer from the computer language back to the human language.

Each mode is handled within the handle_translation function by checking which mode is provided as an argument. This design pattern allows for easy addition or removal of states in the future if needed. If you still have any concerns or questions, please let me know and I’ll be happy to help.
I copy an pasted the bit in question(your elif statements on each translation mode), and this was its response.


I apologize for the confusion in my previous response. Upon further review, I realize that the refactored code doesn’t fully handle the translation modes as presented in the original code. Here’s an updated refactored version that includes all the translation modes and models as shown in the original code:

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):

    def translate_question(mode, question, src_lang, dest_lang):
        if mode in {"human2machine", "twoway"}:
            if translation_model == "argostranslate":
                return translate(question, src_lang, dest_lang)
            elif translation_model == "deepl":
                src_lang, dest_lang = src_lang.upper(), dest_lang.upper()
                if src_lang == "EN": src_lang = "EN-US"
                if dest_lang == "EN": dest_lang = "EN-US"
                translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
                return str(translator.translate_text(question, target_lang=dest_lang))
            else:
                return openai_translate(question, src_lang, dest_lang)
        return question

    def translate_answer(mode, answer, src_lang, dest_lang):
        if mode in {"machine2human", "twoway"}:
            if translation_model == "argostranslate":
                return translate(answer, src_lang, dest_lang)
            elif translation_model == "deepl":
                src_lang, dest_lang = src_lang.upper(), dest_lang.upper()
                if src_lang == "EN": src_lang = "EN-US"
                if dest_lang == "EN": dest_lang = "EN-US"
                translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
                return str(translator.translate_text(answer, target_lang=dest_lang))
            else:
                return openai_translate(answer, src_lang, dest_lang)
        return answer

    translated_question = translate_question(translation_mode, question, human_language, computer_language)
    answer = chain({"question": translated_question})
    translated_answer = translate_answer(translation_mode, answer["answer"], computer_language, human_language)
    answer["answer"] = translated_answer

    if answer["sources"]:
        sources = extract_patterns(answer["sources"], "_id_")
        sources_translated = []
        sources_counter = 1
        for source in sources:
            source = "_id_" + source + "_id_"
            if name_mapping:
                uuid_dict = json.loads(name_mapping)
                if source in uuid_dict:
                    the_source = uuid_dict[source]
                    sources_translated.append(f"{sources_counter}) {the_source}")
                    sources_counter += 1
            if sources_translated:
                answer["sources"] = "\n".join(sources_translated)

    response = f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"
    return response

This updated refactored version handles “human2machine”, “machine2human”, and “twoway” modes for each translation model (argostranslate, deepl, and openai) while correctly handling the “no translation” mode.


So I mean, thats not great. I decided to give it another shot however, this time in my librechat application I removed the option for plugins, and I turned down the temp from 1 to 0 and this was the response I got:
Here’s the refactored code with improved readability, best practices, and documentation:

def chat_with_docs(question, previous, chain, human_language, computer_language,
                   translation_mode, translation_model, name_mapping):
    """
    This function processes a question, translates it if needed, gets an answer, and formats the response.

    Args:
        question (str): The question to be answered.
        previous (str): The previous conversation.
        chain (function): The function to generate the answer.
        human_language (str): The language code for the human language.
        computer_language (str): The language code for the computer language.
        translation_mode (str): One of ["no translation", "human2machine", "machine2human", "twoway"].
        translation_model (str): The translation model to use. Currently supports "argostranslate", "deepl", and "openai".
        name_mapping (str): A JSON string containing a mapping of UUIDs to source names.

    Returns:
        str: The formatted response containing the question, answer, and sources.
    """

    def translate_text(text, source_lang, target_lang, translator=None):
        if translation_model == "argostranslate":
            return translate(text, source_lang, target_lang)
        elif translation_model == "deepl":
            return str(translator.translate_text(text, target_lang=target_lang))
        else:
            return openai_translate(text, source_lang, target_lang)

    if translation_model == "deepl":
        human_language = human_language.upper()
        computer_language = computer_language.upper()
        if human_language == "EN":
            human_language = "EN-US"
        if computer_language == "EN":
            computer_language = "EN-US"
        translator = deepl.Translator(os.environ["DEEPL_API_KEY"])

    if translation_mode == "human2machine":
        question = translate_text(question, human_language, computer_language, translator)
    answer = chain({"question": question})
    if translation_mode in ["machine2human", "twoway"]:
        answer["answer"] = translate_text(answer["answer"], computer_language, human_language, translator)

    if answer["sources"]:
        sources = extract_patterns(answer["sources"], "_id_")
        sources_translated = []
        sources_counter = 1
        for source in sources:
            source = "_id_" + source + "_id_"
            if name_mapping:
                uuid_dict = json.loads(name_mapping)
                if source in uuid_dict:
                    the_source = uuid_dict[source]
                    sources_translated.append(f"{sources_counter}) {the_source}")
                    sources_counter += 1
            else:
                print("no name mapping provided")

        if sources_translated:
            answer["sources"] = "\n".join(sources_translated)

    return f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"

In this refactored code, I have:

  1. Added a docstring to explain the function, its arguments, and its return value.
  2. Created a helper function translate_text to avoid repetitive code for handling different translation models.
  3. Simplified the translation_mode conditions by handling “human2machine” before calling the chain function and “machine2human” or “twoway” after calling it.
  4. Removed unnecessary checks for empty strings and type checks.
  5. Used f-strings for better readability in string formatting.

This looks a lot better to me, and gpt seemed to think so as well. However, I decided to send over the background and I got this response


Thank you for providing more context about your project. Based on your explanation, I have reviewed the refactored code again to ensure it aligns with your requirements. The refactored code should work correctly for your project, but I will provide an overview of the key parts to help you gain confidence in the solution:

  1. The translate_text helper function handles translations for all three models: “argostranslate”, “deepl”, and “openai”. It takes the text, source language, target language, and an optional translator object (for the “deepl” model) as arguments.

  2. The “deepl” translation model is initialized with the API key, and the human and computer languages are adjusted accordingly, with a special case for “EN” to “EN-US” conversion.

  3. The translation_mode is handled as follows:

    • “no translation”: The question is passed directly to the chain function without any translation.
    • “human2machine”: The question is translated from the human language to the computer language before being passed to the chain function.
    • “machine2human”: The question is passed directly to the chain function, and the answer is translated from the computer language to the human language.
    • “twoway”: The question is translated from the human language to the computer language, and the answer is translated back from the computer language to the human language.
  4. The chain function is called only once to generate the answer, simplifying the code and avoiding repetition.

  5. The code handles the translation of source names if a name_mapping is provided.

The refactored code is designed to be more readable, maintainable, and efficient, while correctly handling the different translation models and modes. However, I recommend you thoroughly test the refactored code with different scenarios to ensure it meets your project requirements and performs as expected.


Finally, I ran it one more time including your context from the beginning:


Here is the refactored version of your Python code:

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):
  
    def translate_question(question, source_lang, target_lang, translator):
        if translation_model == "argostranslate":
            return translate(question, source_lang, target_lang)
        elif translation_model == "deepl":
            return str(translator.translate_text(question, target_lang=target_lang))
        else:
            return openai_translate(question, source_lang, target_lang)
    
    def translate_answer(answer, source_lang, target_lang, translator):
        if translation_model == "argostranslate":
            return translate(answer, source_lang, target_lang)
        elif translation_model == "deepl":
            return str(translator.translate_text(answer, target_lang=target_lang))
        else:
            return openai_translate(answer, source_lang, target_lang)

    def adapt_language_codes(locale):
        locale = str(locale).upper()
        if locale == "EN":
          locale = "EN-US"
        return locale

    if translation_model == "deepl":
        human_language = adapt_language_codes(human_language)
        computer_language = adapt_language_codes(computer_language)
        translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
    else:
        translator = None

    if translation_mode == "human2machine" or translation_mode == "twoway":
        question = translate_question(question, human_language, computer_language, translator)

    answer = chain({"question": question})

    if translation_mode == "machine2human" or translation_mode == "twoway":
        answer["answer"] = translate_answer(answer["answer"], computer_language, human_language, translator)

    if answer["sources"] != "":
        sources = extract_patterns(answer["sources"], "_id_")
        sources_translated = []

        for idx, source in enumerate(sources, start=1):
            source = "_id_" + source + "_id_"
            uuid_dict = json.loads(name_mapping)

            if source in uuid_dict:
                the_source = uuid_dict[source]
                sources_translated.append(f"{idx}) {the_source}")
            else:
                print(f"{source} not in name_mapping")

        if sources_translated:
            answer["sources"] = "\n".join(sources_translated)

    return f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"


In the refactored code, I did the following:

  1. Created two helper functions, translate_question and translate_answer, to handle the translation logic.

  2. Created another helper function, adapt_language_codes, to convert language codes when using DeepL translator.

  3. Removed repetitive condition checks for different translation_modes and translation_models.

  4. Used f-strings for string formatting to make it more readable.


Just like before, I copy and pasted your original elif statements and asked it if it was correctly getting modes correct:


Yes, the refactored code preserves the translation modes. However, instead of checking each translation mode separately for each translation model, the common logic has been abstracted into the helper functions: translate_question and translate_answer. These two functions contain the same logic as in the original code and take care of handling different translation models (argostranslate, deepl, or any other).

If you look at the refactored code, you will see that depending on the translation mode, the necessary translations for both the question and the answer are performed using the helper functions. This makes the code more readable and avoids repetitive condition checks.


So overall, I am not sure what to think here. I did my best to understand your code and I think I sort of get it and it does seem like the method gpt used should work okay, however without seeing what the rest of the code looks like I will not be able to fully understand it, much less do any sort of testing.
I realize this response was fairly long but Im curious about what you think of these results?

3 Likes

user:
{your code}
Could you make this code more readable?

assistant:
I understand you’re looking for a more readable version of the code. To make it more readable, we can refactor it in a few ways:

  1. Simplify translation process: Create separate functions for each translation model and mode to handle the translation.
  2. Simplify handling of sources: Create a separate function to handle and translate sources.

Here’s an example of how this can be done:

def translate_with_argos(text, source_lang, target_lang, mode):
    if mode in ["human2machine", "twoway"]:
        text = translate(text, source_lang, target_lang)
    return text

def translate_with_deepl(text, source_lang, target_lang, mode):
    if mode in ["human2machine", "twoway"]:
        translator = deepl.Translator(os.environ["DEEPL_API_KEY"])
        text = str(translator.translate_text(text, target_lang=target_lang.upper()))
    return text

def translate_with_openai(text, source_lang, target_lang, mode):
    if mode in ["human2machine", "twoway"]:
        text = openai_translate(text, source_lang, target_lang)
    return text

def handle_translation(question, human_language, computer_language, translation_mode, translation_model):
    if translation_model == "argostranslate":
        question = translate_with_argos(question, human_language, computer_language, translation_mode)
    elif translation_model == "deepl":
        question = translate_with_deepl(question, human_language, computer_language, translation_mode)
    else:
        question = translate_with_openai(question, human_language, computer_language, translation_mode)
    return question

def translate_sources(sources, name_mapping):
    sources_translated = []
    if sources != "":
        sources = extract_patterns(sources, "_id_")
        if name_mapping != "":
            uuid_dict = json.loads(name_mapping)
            for idx, source in enumerate(sources, 1):
                source = "_id_" + source + "_id_"
                if source in uuid_dict:
                    sources_translated.append(f"{idx}) {uuid_dict[source]}")
    return "\n".join(sources_translated)

def chat_with_docs(question, previous, chain, human_language, computer_language, translation_mode, translation_model, name_mapping):
    question = handle_translation(question, human_language, computer_language, translation_mode, translation_model)

    answer = chain({"question": question})

    if translation_mode in ["machine2human", "twoway"]:
        answer["answer"] = handle_translation(answer["answer"], computer_language, human_language, translation_mode, translation_model)

    answer["sources"] = translate_sources(answer["sources"], name_mapping)

    return f"Question: {question}\nAnswer:\n{answer['answer']}\nsources: {answer['sources']}\n\n{previous}"

With this refactoring, the code is more modular and easy to read. Each function does a specific task and it’s easier to manage or modify a specific function in the future.

I didn’t check the code line by line right now. But to me, it looks quite usable.

1 Like

Thank you for your efforts. Indeed, my code is quite complex and difficult to understand. This is partly due to the fact that I have not yet worked with almost all of the programming libraries that are used. So with the integration of DeepL I found that the language codes are different from argotranslate. Furthermore the handling of the sources is badly understandable, if one does not know the remaining code / use cases. The sources are about a functionality from the library langchain, namely RetrievalQAWithSourcesChain. The documents themselves have IDs as names and based on the IDs I can later insert arbitrary and detailed sources. Or not, if no sources are defined. But this part is basically not very good, because the langchain functionality doesn’t seem to be very deterministic.

However, this should not be about my project at all. I just added the info so that you get some context to it when reading it.

I unfortunately don’t have the time to look at your tests right now and I noticed that the API parameter documentation is missing? I assume you are using ChatGPT? Basically I guess that with multiple requests better results can be achieved. Maybe one really needs to adjust the own prompting.

My main intention was just to show an example where gpt-4-0314 generates usable code and the current version does not (with the first try). If OpenAI takes feedback from the forum into account (I haven’t found a better place for feedback), I think it’s helpful to show specific examples. I probably should have prepared it better and constructed an easy to understand example from it. Well, in any case a concrete example including concrete API parameters is I think better than no example. Haven’t seen very concrete code examples here on the forum in this regard.

1 Like

I appreciate the original code, although complex and perhaps not to my usual programming style. It served as a starting point for investigating a reported decrease in quality. For my own testing, I used LibreChat, an open-source version of ChatGPT, and initially adhered to the default settings:

  • Temperature: 1
  • Top-p: 1
  • Frequency and Presence Penalty: 0

In my following tests, I reduced the temperature setting to 0. These settings are equivalent to the default parameters used with the API in the absence of specific instructions.

It’s important to remember that GPT’s responses can vary due to its probabilistic nature, which requires running multiple tests to achieve a reliable assessment. The concept of “first try” can be misleading, as both effective and less effective responses can arise by chance, regardless of parameter tuning or model version. Consequently, without seeing the rest of his code or understanding the full context, it’s challenging to determine the validity of the observed decrease in quality. It could just redone it a way that works but isn’t as obvious as elif statements

1 Like

6/27/23 … & ChatGPT4 still appears to be functioning in an inferior, post-lobotomy state.

2 Likes

To anyone that was interested in requesting access to a previous version of ChatGPT4 (e.g., an April version or May 3 version, etc.), here is the thread that I started for that (that I mentioned above): Is there any way to access the earlier (smarter) version of ChatGPT-4 (e.g., pre-May 3 update) prior to nerf?

A few folks (@supereric7748 ; @mokhtarali2001 ) requested that I share this. Sorry for the late reply. Making noise about this appears to be our only recourse at this juncture, so keep bumping up these threads so they’re always at the top of this community forum.

2 Likes

I could confirm to this, especially with harder scientific journal articles with very specific terms/jargon and sentence structures. Sometimes, it defaults to the inferior version and I feel scammed using the “new” ChatGPT4.

Also, it feels chatGPT4 is becoming lazier. It seems that the AI becomes dumber as the content policy maker that limits everything while pandering to only certain ideologies (try it yourself). Even when I specifically asked for counterarguments to that “certain ideology”, it refused to budge and gave ones it BELIEVES TO BE THE TRUTH and kept resisting TO GIVE COUNTERARGUMENTS (AND WHATIFs) for the sake of “respecting all users”. This is a real bummer as an AI should be neutral (as what it claims) and useful to provide balanced view (as it claims too), but in reality, IT IS NOT and IT IS GETTING DUMBER for certain topics/niches. For the sake of PC and “safety” of this forum, I can’t disclose this, but certainly OpenAI is not “as open” as it claims (it is more closed).

Also be careful of its hallucination for factual information, it looks plausible and “trusted” with grandiose words, but IT IS NOT.

1 Like

I keep scrolling over the forum and find people like you (and me) who are totally disappointed by the fraud OpenAI are playing on their customers. The so called GPT-4 is dumber than GPT-3.5 when it was first released. It used to generate high-quality content and it used to produce content for many pages.

Today I don’t even see the “Continue Generating” button. The fake model just spits something between 600-800 tokens which 90% is useless and that’s all.

2 Likes

I have to agree. After using gpt-4 (primarily the code assistant, previously called the code interpreter) for months, today there was an extremely noticeable decline in reliability.

First off, it consistently refuses to return full code when specifically asked to do so. When advised of this, it apologizes, and continues to do the same thing repeatedly.

It’s also starting to come back with syntax errors, which it never used to do, and lapses in logic.

Since I can no longer specifically request the code assistant, it’s hard to troubleshoot. But the model is starting to behave a lot like gpt-3.5-turbo-16k, which I abandoned months ago.

The result of all of this is, where my coding time used to be reduced due to assistance from ChatGPT-4, it is now increasing with more and more mistakes that have to be corrected.

Here is the conversation: ChatGPT - A ShareGPT conversation

Whose coding job is this AI supposed to eliminate?

You don’t need to read the code, just look at my responses. It’s constantly coming back with comments like “add logic here” or “use same field mapping as…” which I explicitly tell it not to do that, to provide the code itself. This takes several tries, thus increasing the time it takes to develop the overall system widget. “What part of ‘do not do that’ do you not understand?”

It did not used to do this. It used to respond with the code as requested. And, if it did insert comments, it would immediately stop when asked to do so.

Furthermore, it used to either include error trapping code or recommend it. Now, it has to be be asked to include error handling in places where it should be obvious (as in any database call).

The gpt-4 API still appears to perform well for the applications it is assigned, but gpt-4 through ChatGPT has seriously declined. No joke.

p.s. I know someone is going to come back with the point that it is trying to conserve tokens. Well, what’s the point of a 100K token context window when you can’t use just a fraction of that to get a complete answer?

2 Likes

Absolutely terrible quality now - it just can’t produce a good reply now, I’ve spent like an hour trying to make it a simple SQL query like: find which values are not in the table if we look at particular column. And it keeps to produce some weird responses.

1 Like