Codex for Robotics

Hello, my name is Abhi Patil. I’m the lead programmer for FIRST Tech Challenge Robotics Team 7172 ‘Technical Difficulties’. We’re a high school robotics team based in Dallas, Texas, USA, and I wanted to use OpenAI Codex to help rookie teams program their robots. I saw that Codex is trained on github repositories and was wondering how I could train codex further on the following repo and its forks: GitHub - FIRST-Tech-Challenge/FtcRobotController



Hello @pabhiram05 :wave:,

As a former FIRST Tech Challenge competitor, I am stoked that you want to be able to utilize Codex in order to assist fellow members in programming a robot!

You can use A.I. to help program the robot:

Teams can use advanced artificial intelligence (AI), augmented reality (AR), and machine learning (ML) technologies to improve the performance of their competition robots during play.

Source: FIRST Tech Challenge Post

Now, I’d like to provide insight into what exactly you’re hoping to get out of this request. Unfortunately, fine-tuning the Codex model is not available publically, but I’d like to chime in and say that training a model requires a lot of carefully constructed training data to train a model for a specific task with the desired result of it performing better than it’s parent model, so I personally wouldn’t advise trying to fine-tune Codex within this use-case. However, it’s not the end of the world! I did take a look at the GitHub repository that you linked, and it appears that y’all are having to develop an Android app that communicates with the robot to accomplish tasks. Programming an Android mobile app requires programming in primarily Java, and considering that Codex was trained on most of the data on GitHub, it should be able to help write code to successfully build Android apps.

I’d like to point out what Codex will be good at and what it won’t be good at within the context of helping teach rookie teams how to program their robots. Codex, if you review and practice good prompt design, will produce code that will help design Android apps generically, but since it doesn’t know the specifics of the FIRST Tech Challenge Android SDK, you will need to design your prompt with any information that you think Codex will need that’ll improve the completion it gives you. For example, you can ask Codex to implement a button that shows up on the Android app, but the action you’d like to bind to the button when pressing it may pertain to a function that is unique to the FIRST Tech Challenge when communicating with your robot, so Codex may not have enough information to provide useful completions in that regards.

There are quite a few things you can do with Codex. You could designate some of the tokens in your prompt towards providing Codex with a list of Java functions inside the repo as well as the document comments counterparts provided by FIRST Tech Challenge to build a system that provides the best function(s) that may accomplish what you want, and possibly get Codex to provide a rough implementation of those picked function(s) within the scope of the script!

It’s really about finding the balance of how to best utilize Codex within this specific use-case for these teams. There will definitely be trial and error with Codex if you do decide to work on this project you have (and giving students access to a very smart A.I. system may cause some distractions along the way :wink:.)

Overall, I do think letting these teams at least experience Codex would be nice, as that will surely inspire them since Codex is truly amazing. Start small, develop some small projects that produce positive results, and adjust as you find what works and doesn’t work with Codex so that the end result is positive for the teams!

If you have additional questions, certainly reach out and we’ll try our best to provide further insight! Hope this was a little helpful! Cheers! :robot: