Does anyone know of any Open Source AI tools which can make arbitrary changes to software projects? For example, I’d like to be able to just do prompts like, this:
“Add a new date property to my class named ‘abc.py’, and update my SQL file named ‘abc.sql’ to reflect the change.”
That prompt should result in actual updates to my project files, to implement what I asked for.
I’m aware of Devon, Pythagora (GPT Pilot), and MetaGPT, but I have a feeling those are going to analyze the entire software project and consume massive amounts of OpenAI tokens as a result. I want something that’s in Python and uses Langchain, that I can call directly from my own Python. I don’t want anything that relies on RAG or Vector Databases. There should be a way to do this without RAG.
Indeed, you can achieve the desired functionality without relying on RAG. If your codebase is under 128k, utilizing GPT-4 with its 128k capacity becomes straightforward, requiring minimal calls to process the entire codebase efficiently.
However, for larger codebases, a more targeted approach is necessary. You could implement a system that identifies specific line locations or markers within your code. Python scripts could then extract the relevant sections, make modifications, and seamlessly reintegrate them. This concept simplifies the process, ensuring precise changes without the need for exhaustive analysis.
In essence, whether your codebase is small enough to be processed in its entirety or requires a more granular approach, leveraging GPT-4 and Python scripting offers versatile solutions tailored to your project’s needs.
Ps if smaller code 16k you could do it with GPT 3.5 which is fraction of the costs.
Good thoughts, and ideas. I agree with all that. My code-base is 500K lines, and it just seems like a bad idea to send ALL of any codebase to an LLM, even if it was small enough. It seems like it could get expensive…and even sending that much over a network for every call is kind of bad practice too if you ask me. You’re right for tiny projects it’s reasonable, tho.
One test I ran today however is that you can (even in Chat GPT-3.5) for example send a source file, and ask for some kind of edit to be done to the file, and request that the new updated file be sent back in Unified Diff format!! It works!! That’s such a cool way to get LLMs to save space and send back the smallest packets of info to perform code edits directly in existing files.
And your idea of putting markers into the code is something I had though of too, and am experimenting with.
EDIT 1: Added image. This image was my first test but it worked just as well when I asked it to refactor some Java code. It worked even though I accidentally said Univsal instead of Unified.
EDIT 2: Of course difflib.patch() is all you need to get the full final file.
EDIT 3: So you could actually probably send an entire project, by specifying all the filenames and file contents as a big text block in a prompt, and request that the AI make some project-wide refactoring to your entire project and send back only the DIFF for each file it needed to modify, but again as I said, I’m trying to avoid sending a full project. I just wanted to note this as a possibility. And although I’ve never personally tried “function calling” AI feature, you would probably do a “fileSave(fileName, content)” as another way to let it make mods to your project.
yeah, there are a lot of ways to do it using various methods. the difflib is indeed a faster way to get there indeed. where I played with kruel.ai my system is differences between code tool using assistant, and gpt-4 sending text, and GPT 3.5. if you can do it with GPT 3.5 with understanding than it’s always the cheapest option imo. may not have the context length but you can fit a lot in 16k and you can also build yourself a document chunker and system that summaries each part for understanding to build the full response with understanding well more steps if the costs are fractional compared to a full cent per transaction even with more steps long as it works and outcome is what you expect it works really well.
Well it definitely seems to me like we need a good Python library (probably part of Langchain) that’s capable of doing refactorings on codebases, in a super simple way unlike Devon, Pythagora, and the other existing Coding Assistants, because it seems to me like it can be done with about 500 lines of Python, and no RAG or Vector Databases, or complicated things. Just a way to embed parts of code into prompts, and get the results back and integrated automatically into the project files directly. Maybe I’ll write that 500 lines tonight. The night is young.
Actually “Github Copilot Workspace” (different from the original Copilot) was just released on the day after I asked this question!
I had written and published my own tool in Python (over the past two days) that can refactor code too, but I’m glad Microsoft is now providing this capability (code refactoring)!
I guess my tool will still be valuable because it’s using Langchain so it won’t be limited in what AI LLM it can connect to and it’s open source, and under 1000 loc.
I actually did that! I wasn’t bluffing. I wrote my own agent over the past few days. It’s on Github as “QuantaAgent”. It’s Python + Langchain so it will be able to run any Cloud AI API (or even Local LLMs too), but currently it’s just got OpenAI hard-coded.
It can answer questions about a codebase, refactor, implement new features, etc.
I wrote my own agent over the past few days. It’s on Github as “QuantaAgent”. It’s Python + Langchain so it will be able to run any Cloud AI API (or even Local LLMs too), but currently it’s just got OpenAI hard-coded.