Merge Request Reviewer bot

While playing with the text-gpt-003 model, I figured I’d throw some code at it, and I noticed that I could ask it lots of questions about code and it is even better at understanding, changing and conversing about code than the codex models.

So I thought, what if it can lend its code wisdom to the QA of a merge/pull request, saving developers time when reviewing changes of their peers, or getting advice on their own submitted code etc.

Created a quick POC within a couple of hours (written in a single file that was 90% generated by chatGPT, btw), just as a little demo to my colleagues, nothing more, but I thought I’d share the use case:

Logic:

  1. Whenever a new MR gets created in Gitlab, a webhook kicks off my bot.
  2. The bot fetches the code changes of the MR and extracts the git diff strings for all the changes.
  3. Prepared a pre-prompt paragraph that asks it to act as a senior developer who reviews the changes and answers 10 code review questions about those changes.
  4. Add the questions and the code diffs to the prompt and throw them at the GPT-3 model via OpenAI’s REST API, as-is (and I mean really as-is, without any parsing or additional explanation of what a git diff looks like! GPT fully understood the format out of the box)
  5. Also made part of the prompt some instructions about the formatting of the response, so it renders nicely in Gitlab.
  6. Post the response back to the MR in Gitlab as a comment.

I pass the questions in from an environment variable, tweaked it a bit for more useful and pleasant output, and it now gives me these types of answers:

I am also thinking of using the chat API now, to use Gitlab’s comment thread system as a chat interface, so that the developer can ask follow-up questions about the review and the code changes.

Lots of fun ideas in this space, and I’m sure there will be tools for this within months, if not weeks.

2 Likes