Thanks for all of these excellent comments. It is not so much a question of the value of determinism, but instead the attempt to characterize the uncertainty given that we have a stochastic process. That the system is non-deterministic is fine. So, what does the uncertainty look like exactly? Bringing it home, if I were not so lazy, I would enter a prompt the ended with “Draw 10 circles”, run it 50 times, and try to understand the variance. Some of the variance may be due to Python library chosen for drawing, or indeed whether different programming languages are used. Which are used more or less often? I’ve also seen that some % of completions are invalid and will not compile/interpret.
Some recent browsing includes slides from Nvidia on GPU non-determinism: https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9911-determinism-in-deep-learning.pdf Look at Slide 51 (Takeaways). There is also something very recent on uncertainty in neural networks here: https://arxiv.org/pdf/2107.03342.pdf
All of this makes the Codex more interesting to study.