The title basically says it all, but I’ll write a bit more to provide a bit more context for my post.
Many of us have now used Code Interpreter over the past few days and I am pretty blown away by how much it enhances GPT-4 (I’d imagine the same would be for 3.5 as well), much like lots of people. However, I feel like this feature could significantly enhance many API applications that require any type of logic validation (i.e math, coding, files etc.). I think to just keep this as a standalone app in ChatGPT would be somewhat unfortunate for its huge potential.
I won’t speak for everyone, but I would even be willing to pay a bit more if this was a fully managed experience (i.e with the interpreter). I know it’s theoretically possible to build my own Code interpreter with the function calling API, but I am sure OpenAI’s solution would be much better.
If OpenAI would consider this, I would be incredibly grateful and willing to give early feedback.
I would like to see that feature on playground too, but really, it seems like CI is just some creative prompt engineering and recursive summarization to fit token maximum limitation.
I’m not sure if OpenAI will go for, specially after testing it. Not sure if I have a cognitive biases, but it wasn’t something much better than what we already currently get for gtpt-4 8k
I totally agree @landongarrison . I am really keen to see an API for Code Intepreter. I have some use cases where I have thousands of data files need to be analysed for anomalies. Doing this one-by-one in ChatGPT is not feasible for me. I’d pay good money for a Code Interpreter API!
I would also love to have the code interpreter available in the API; one of the applications I’m developing uses the GPT-4 API to analyze (text, in pdf form) documents, and in order to do so I’ve got a convoluted Python script that parses the file, then cleans it up, then sends text to the API… if I could just send the whole pdf to the API that would be amazing.
well, technically, the innovation to CI is the fact that gpt can run code and output it.
while doing some
some creative prompt engineering and recursive summarization to fit token maximum limitation. - @Thiago
the output is just some computer understable format, i.e json/md/etc. meaning, the usecase we would like is to be able to “api query the CI API with prompts/code that can run on openai’s server and return those results”.
e.g imagine doing some “complex arithmetic that returns a png image (creative coding)” by calling an api endpoint. similar thought process.
Then OpenAI would need to provide a sandbox for every code interpreter API call. After all, the code has to be executed somewhere safely. I can’t quite imagine that they would do that. You could build something yourself with the API that executes Python code. However, I would be reluctant to let a model running wild due to a clumsily formulated prompt loose on my computer. So, you would need a secure environment for execution.
Theoretically, with the functions API/calling, you could write a function that allows GPT to use a code interpreter. You would simply use the code it generates from the function call, save it to a file and run it on your server/another server. As others mentioned, it’s a bit risky to do this but I’m sure there are ways to build a classifier to detect if code may be high risk or low risk.
So actually it’s not that complicated. What is tough is getting GPT to use the code interpreter to the level OpenAI has, which is why I want them to make an API
They do this already with the ChatGPT Web App though and are taking the same risk—I don’t actually think this is much of a problem. Where it might be complicated is simply scaling the instances for an API, because I imagine that the API is a majority of their computing. Tough to say though.
@landongarrison do we know if they use Function Calling to generate code in code interpreter, or a different primitive? Would be annoying to build this with functions then learn it’s the latter.
The possibilities are endless if OpenAI offers an API for code interpretation. If OpenAI does so, these guys will be 50000 feet high , At the moment, google bard is still in their early stages.
I can almost guarantee they are using something very similar to Function calling, if not the exact same thing. Code interpreter is just a plugin at the end of the day, which is what Function calling is at it’s core. The only thing I am unsure about is whether they have done any additional fine-tuning—I sometimes notice GPT-4’s reasoning when using CI is slightly different. However, I kind of doubt it to be honest.
Interestingly, we did some code interpretation between GPT-4 and Bard. They both tackled the issue correctly, but we asked both models to interpret the other’s solution. GPT-4 conceded that Bard’s solution was more sustainable. That said, we asked GPT-4 if it would build it more sustainably instead of “just make it work” and it basically said “if prompted that way it would”. I’d like to have both API’s get called to code review, then have them cross-check each other’s work and mutually agree on a solution with annotated choices or additional feedback such as “will you be using this code in production or just PoC”, etc. to build the proper solution.
I had an issue with a malformed json file and it corrected some issues, tried to open it, identified it was still malformed, identified and corrected the next issue and ordered on that three times (without intervention) until the json file was correct.
That really impressed me, because it did what I would have done in terms of iteratively fixing issues. Not exactly world-breaking stuff but I was taken aback a bit when it just kept trying to fix the issues on its own.
So, it can run code, interpret the results, and decide if it needs to do anything else before returning control back to the user, which I think is immensely more powerful than just running codeand outputting results.
