OpenAI’s spinningup is an exemplary educational resource produced by OpenAI that makes it easier to learn about deep reinforcement learning (deep RL). It’s very thorough and comprehensive; it has well written explanations of major concepts & algorithms and links to all the important papers. But there is one major practical problem, which is that the accompanying code was written in 2018 and has not been updated, which can make it difficult to get running with old and mutually incompatible versions of Python, tensorflow, etc. This status is according to plan, it was designed with a limited lifespan in mind, no doubt for excellent business reasons. But this leaves the curious researcher with issues, which are reflected in the github repository for spinningup. The bottom line appears to be that some people are successfully working through the issues but others are stuck. So this raises two points.
-
Are any plans for updating spinningup? If not, what are some other good, more current resources for hands-on learning deep RL? I liked some of the things I found at Kaggle.
-
I also want to share the relatively clean path I finally found to setting up spinningup, in hopes that this will spare others pain. If you look on github, you’ll see that there are several forks of spinningup available. I haven’t tried all of them, but I had success using the docker repository for. VS Code written by cj-l at ISOLABs. Installing it is straightforward if you are familiar with vs code extensions and the newly built container successfully runs the spinningup installation test right out of the box. There was one snag, which is that running X windows on Big Sur is more complicated than it needs to be; the key point is that you need to be sure your Docker container has permission to connect to the port that Docker uses for the X display, which is host.docker.internal:0. Good details are found here.
The broader issue beyond all this is what can and should OpenAI do to help people in its community who want to get up to the state of the art with deep RL and have business needs that go beyond the current API. Maybe this forum can be a good place to start!
Spinningup in action on a Docker container inside VS Code.