Fascinating stuff all this. For everyone wondering about consistency and reliability here’s a story hope it provides some insight…
My main approach for getting things done quickly is to have a gpt/assistant write a script for the task that takes arguments then use it in the session. You can add layers of these all calling each other, scripts in the knowledge (that it wrote) etc and you have a scratch pad, memory, drawing and plotting tools, tons of good stuff. You have to save the scripts or your context window gets all clogged up, the code gets terrible and won’t run and the conversation will drift.
Was just using the same technique in a different gpt, got it to save the script on it’s side as normal and then tried to test it. It then insisted it would not run the script. Ever. It wouldn’t even run the command to run the script so I could see the error. Tried every trick in the book and it still wouldn’t.
I then told it that the script was actually in it’s environment so it’s ok and that did it. Back to work…
For getting quality code, the grab below is the holy grail for me. It sees some errrors, which makes it try harder, then it asks for guidance. It is very focused on exactly the right thing. The next message was everything working right, the right dialog from the beginning, good scripts, no off topic stuff. The message history was perfect, following a clear and steady line of thought, cheerful and working together. I would bet real money that what followed would be Gpt4 up to working miracles as it can sometimes do.
[edit: Just noticed I forgot to get it to drop down to the python shell, it was in a notebook. Script was probably fine. It would have rewritten it and given it a nice tune-up anyway.]