While playing with the text-gpt-003 model, I figured I’d throw some code at it, and I noticed that I could ask it lots of questions about code and it is even better at understanding, changing and conversing about code than the codex models.
So I thought, what if it can lend its code wisdom to the QA of a merge/pull request, saving developers time when reviewing changes of their peers, or getting advice on their own submitted code etc.
Created a quick POC within a couple of hours (written in a single file that was 90% generated by chatGPT, btw), just as a little demo to my colleagues, nothing more, but I thought I’d share the use case:
Logic:
- Whenever a new MR gets created in Gitlab, a webhook kicks off my bot.
- The bot fetches the code changes of the MR and extracts the git diff strings for all the changes.
- Prepared a pre-prompt paragraph that asks it to act as a senior developer who reviews the changes and answers 10 code review questions about those changes.
- Add the questions and the code diffs to the prompt and throw them at the GPT-3 model via OpenAI’s REST API, as-is (and I mean really as-is, without any parsing or additional explanation of what a git diff looks like! GPT fully understood the format out of the box)
- Also made part of the prompt some instructions about the formatting of the response, so it renders nicely in Gitlab.
- Post the response back to the MR in Gitlab as a comment.
I pass the questions in from an environment variable, tweaked it a bit for more useful and pleasant output, and it now gives me these types of answers:
I am also thinking of using the chat API now, to use Gitlab’s comment thread system as a chat interface, so that the developer can ask follow-up questions about the review and the code changes.
Lots of fun ideas in this space, and I’m sure there will be tools for this within months, if not weeks.