Feature Request: Enhanced Diff Format Support in ChatGPT for Streamlined Code Integration

Dear OpenAI Team and Community,

I am excited to share a proposal for a feature enhancement in ChatGPT that could significantly streamline the application of AI-generated code modifications: the integration of support for generating output in the Diff format.

This idea was inspired by the new JSON format output.

1. Current Output Format:
Currently, when I request code modifications from ChatGPT, the response includes initial explanations, a modified code block, and final explanations. For instance, I recently asked for a modification in a Python script. The response, though helpful, was not in a Diff-compatible format:

Sample of current output from gpt-4:


Certainly! To ensure that the statistics are logged to the file in a single line and printed to stdout as before, we need to slightly adjust the logging and print statements at the end of the script.

Here’s the updated script:

#!/usr/bin/env python3
# ... [rest of the script] ...
def main(name, token=None):
    # ... [other parts of the function] ...
    # Prepare statistics for logging and printing
    stats_log = (f"Statistics - Total Directories: {stats['total']}, Total Size: {stats['total_size']} KB, "
                 f"Copied: {stats['copied']}, Updated: {stats['updated']}, "
                 f"Already copied: {stats['already_copied']}, Failed: {stats['failed']}")
    stats_print = (f"\nStatistics:\n- Total Directories: {stats['total']}\n- Total Size: {stats['total_size']} KB\n"
                   f"- Copied: {stats['copied']}\n- Updated: {stats['updated']}\n"
                   f"- Already copied: {stats['already_copied']}\n- Failed: {stats['failed']}")
    # Log and print statistics
    logger.info(stats_log)
    print(stats_print)
# ... [rest of the script] ...

I’ve updated your script with the changes you requested. Let me know if I can help you with anything else!


This approach, while informative, requires manual application of the changes, which can be time-consuming and error-prone.

2. Proposed New Diff Format Support:

I propose that ChatGPT should be able to generate responses in the Diff format. This would make the output more concise and directly applicable using tools like patch. Here’s an example of how the same modification would look like:

Proposed Diff format:


--- original_script.py       2023-12-02
+++ modified_script.py       2023-12-02
@@ -88,7 +88,15 @@
-    logger.info(f"\nStatistics:\n- Total Directories: {stats['total']}\n- Total Size: {stats['total_size']} KB\n- Copied: {stats['copied']}\n- Updated: {stats['updated']}\n- Already copied: {stats['already_copied']}\n- Failed: {stats['failed']}")
+    # Prepare statistics for logging and printing
+    stats_log = (f"Statistics - Total Directories: {stats['total']}, Total Size: {stats['total_size']} KB, "
+                 f"Copied: {stats['copied']}, Updated: {stats['updated']}, "
+                 f"Already copied: {stats['already_copied']}, Failed: {stats['failed']}")
+    stats_print = (f"\nStatistics:\n- Total Directories: {stats['total']}\n- Total Size: {stats['total_size']} KB\n"
+                   f"- Copied: {stats['copied']}\n- Updated: {stats['updated']}\n"
+                   f"- Already copied: {stats['already_copied']}\n- Failed: {stats['failed']}")
+    # Log and print statistics
+    logger.info(stats_log)
+    print(stats_print)

This format is instantly usable for updating the code base, saving time, and reducing the likelihood of manual errors.

Additional Context and Benefits:

3. ChatGPT’s Explanations as Comments and Commit Messages:
The model’s explanations could be included as comments in the code or formatted as Git commit messages, enhancing clarity and documentation.

4. Previous Attempts:
I’ve attempted to “force” gpt-4 (with failed “Prompt Engineering” attempts) to output a Diff file within a code block but found it not precise enough for tools like patch.

5. Efficiency and Resource Savings:
This feature could save significant time and tokens, particularly when AI-generated changes are substantial, avoiding manual integration or requesting the entire script for immediate testing.

6. Beyond the patch Utility - Pythonic Integration Methods:
The idea can evolve beyond traditional tools like patch, employing Python-based methods for applying diffs, such as the difflib module, python-patch library, or GitPython for repositories.

7. Future Vision: GitHub/GitLab Integration with AutoGen:
I envision a future where, upon defining a GitHub issue, a swarm of agents powered by multi-modal LLMs could automatically resolve the issue and create a Pull Request with a tested solution, building upon this Diff format support.

This enhancement would make AI-assisted coding more efficient and pave the way for more advanced integrations, revolutionizing the software development process. I look forward to your thoughts, feedback, and the potential of such a feature in AI-driven development.

Thank you for considering this proposal. I look forward to any feedback and the possibility of seeing this feature in future iterations of ChatGPT.

Cheers!
Thiago

5 Likes

Hi Thiago

First of all, great idea!

To the point.

Though you mention you tried prompt engineering, but unsuccessfully. I still think it is possible,

Plus this could be solve using model fine-tuning.

1 Like

Hello Tony,

Thank you for your insightful reply and the suggestion to use model fine-tuning. While fine-tuning might be a viable path for specific applications, I believe that incorporating the Diff format as a native feature in ChatGPT, similar to the recent JSON output capability, would be highly beneficial for software developers. This approach would streamline the process of integrating AI-generated code changes, allowing the option to turn on or off output in the Diff format based on the user’s needs.

Your suggestion made me wonder why the JSON format output was considered necessary as a native feature and not something to be addressed through fine-tuning. This line of thinking reinforces my belief that having a native Diff format output could greatly enhance the utility of ChatGPT for coding applications, just as the JSON feature has done in its respective domain.

Thanks again for engaging in this discussion!

2 Likes

This would be a very welcome feature! I too struggle with this.

Hi Thiago

I agree diff would be a very useful feature.

At the same time I would argue that not all programmers need diff as badly as json (as otherwise it is extremely difficult to build complex ai flows with statuses) and OpenAI is a pretty small team for what they do and they have to choose.

There are many more of such features that might be useful.

The question is: will companies like OpenAI manage to implement them all considering that the technology is increasingly self-enhancing or we’ll go to good old protocol paradigm, when a company makes the core which consists of relatively small amount of such features, and the features are build by the dev community?

I’m struggling with this problem as well :slight_smile:
I think it’s a good idea. However, the diff format involves numbers and some logic to them. From my experience, the LLM is not very good at stating exact line numbers and this produces invalid diffs. It’d be interesting to think of a solution to that.

1 Like

I’m a bit out of my game here, I just started delving into AI development.

I also am interested in solving this problem for my first AI-related project, and found this model on the Hugging Face (“CarperAI/diff-codegen-6b-v2”, I cannot include links yet in my posts) which has been trained on GitHub public data to specifically improve the diff generation. My two cents :slight_smile:

2 Likes

Pretty sure you can hack this in GPTs using an api, just prompt it to either write the code then diff it and return the both diff and code, or you can just tell it to generate the code inside the diff api call and just return the diff. Unfortunately without diff support in the code viewer it won’t be an amazing experience but at least you can get it.

Hi @nathan.r !

Did you mean how to make ChatGPT generate diffs to print in a chat? That’s not quite what this thread was about, as I understood it. The problem is as follows. ChatGPT is wonderful to generate text snippets or program fragments, or even write small documents as a whole, but at some point you will want it to automatically edit files. It seems for that problem many are still looking for a viable solution.

For instance with my Co-Developer GPT engine you can make a GPT that has the capability to read and write text files in a local directory on your computer - e.g. source code. If a file is long and the GPT needs to change it then you don’t want to have it regenerate the whole file with 3 sentences changed or a method added, because that costs time and tokens and might even hit the token limit. That’s the point where it would be wonderful if ChatGPT could reliably print diffs in some format - not to print them to the user, but to write these diffs into an API that’s called from a GPT, or as a function call.

I tried various approaches how to structure the corresponding replaceInFile action :

  • regular expression replacement
  • unified diffs or diffs with context lines
  • sed script (doesn’t work because ChatGPT isn’t aware of line numbers and adding them in the text confuses it)
  • some special format where it has to give the start line and the end line of the replacement as text

but nothing worked reliably, so for now I settled to a simple search and replace where several literal search strings + their replacements are given, and if any of the search strings is not found or is not unique the engine just refuses to make the change and tells the GPT in the error message to re-read the file and reconsider that search string. That mostly works, but doesn’t feel like a really good solution.

As it is I guess the only reliable solution would be to have the GPT just describe changes whatever way it wants, and to train a lower grade fast local LLM to re-print files with those changes executed, but I don’t have the time to pull something like that off.

So it would be nice if OpenAI would consider that problem of having GPTs edit small portions of documents in an action and incorporate the solution into their training data - be that the unified diff format proposed in the beginning of this thread, or something else.

Best regards,
Hans-Peter

This is precisely what the project does behind the scenes: GitHub - paul-gauthier/aider: aider is AI pair programming in your terminal.

Moreover, I’ve never encountered any errors from the AI…

I haven’t reviewed the code to understand its exact workings. However, the AI assistant does modify specific parts of your codebase.