I just wanted to see if anyone had advice on keeping Codex on track. I’m able to get it started well enough, but sometimes it seems to go off the deep end, and often I end up needing to change things higher up than the last generation.
I’m only using the Playground to investigate what direction I want to go because I have three solid ideas for use cases/products, but as stated, I’m looking for any advice.
I am trying to figure out how to insert episodic memory so it can understand entire projects.
After getting familiar with Codex, I’ve found quite a few techniques that may help you out!
I’m usually able to keep Codex on track during bigger completions by doing a few things. I’ll usually start a multi-line comment area written in the programming language I want Codex to output. Then within that area, I’ll state the overall task that I want Codex to accomplish including specific details when desired.
After I write the overall task out, I then provide Codex with the overall task again, but this time broken down into subtasks so that each subtask is basically instructions to create a class or function.
Once I’ve written out the subtasks, I’ll start the completion with the first subtask commented above where I want Codex to start writing the class or function. If it’s a more advanced topic, I’ll try to go into more detail and provide examples along the way, but I myself am still fine-tuning my prompt design process.
By starting the completion off with the first subtask as a comment above the code, Codex will write out the code, then upon completing that subtask, will continue the pattern and comment in the second subtask before writing out the code for that subtask. I’ve found that it sort of “reminds” Codex of the overall task while also adding in enough detail in each subtask to get fairly good code throughout the completion.
Let me provide an example of what I mean.
Python:
"""
You are a programmer tasked with writing a Python script that uses the 'Selenium' web automation module and the 'chromedriver_autoinstaller' module to scrape webpages. Here are some objects you will need to use in the script:
# Link that goes to OpenAI's API documentation:
openai_documentation_link = "https://beta.openai.com/docs/introduction"
# Xpath for any 'a' element under the 'side-nav-section' div element:
documentation_nav_section = "//div[contains(@class,'side-nav-section')]//a"
Tasks:
1. Import the necessary modules we'll need for web automation.
2. Initialize the variables referenced above.
3. Use the 'chromedriver_autoinstaller' module to install the correct chromedriver we'll need for Selenium.
4. Open a Chrome browser with Selenium and navigate the URL that 'openai_documentation_link' references.
5. Get all elements by XPath 'documentation_nav_section' and save them in an array.
6. Create a dictionary before looping through the array of elements. Use the get_attribute() function to get both the attribute 'textContent' and 'href' from each element and then append them to the dictionary where the text content is the dictionary key and the href link is the value of that key.
7. Print the entire dictionary.
"""
"""
1. Import the necessary modules we'll need for web automation.
"""
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import chromedriver_autoinstaller
"""
2. Initialize the variables referenced above.
"""
openai_documentation_link = "https://beta.openai.com/docs/introduction"
documentation_nav_section = "//div[contains(@class,'side-nav-section')]//a"
"""
3. Use the 'chromedriver_autoinstaller' module to install the correct chromedriver we'll need for Selenium.
"""
chromedriver_autoinstaller.install()
"""
4. Open a Chrome browser with Selenium and navigate the URL that 'openai_documentation_link' references.
"""
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(options=chrome_options)
driver.get(openai_documentation_link)
"""
5. Get all elements by XPath 'documentation_nav_section' and save them in an array.
"""
documentation_nav_section_elements = driver.find_elements_by_xpath(documentation_nav_section)
"""
6. Create a dictionary before looping through the array of elements. Use the get_attribute() function to get both the attribute 'textContent' and 'href' from each element and then append them to the dictionary where the text content is the dictionary key and the href link is the value of that key.
"""
documentation_nav_section_dict = {}
for element in documentation_nav_section_elements:
documentation_nav_section_dict[element.get_attribute("textContent")] = element.get_attribute("href")
"""
7. Print the entire dictionary.
"""
print(documentation_nav_section_dict)
Preset: Python Webscraper Demonstration (Fixed as of 10/10/2021)
I started the completion at “”" before starting Task 2.
I wonder if there are better methods, this is simply one of many possible methods to help keep Codex on track.
Hope I helped somewhat!
EDIT 10/8/2021: I fixed my prompt so that Codex was able to get the ‘textContent’ attribute which correctly grabbed the text from each link. The completion above correctly grabs the text and link, puts them in a dictionary, then print the dictionary!
EDIT 10/10/2021: Oh my! I thought I had fixed the prompt to generate correct output. After a couple minutes I managed to fix my prompt and Codex supplied the desired output flawlessly!! Sorry y’all (the Preset link has been updated too)!
I think the real deal is to automate your manual process.
I’m currently experimenting with the other models as a way for them to produce a prompt so there is less manual input from me. However, if you don’t provide enough information in your prompt, then the models have a higher chance of not providing the output that you want. That’s why I tend to provide a lot of detail in my prompts so that Codex and the other models know what output I am looking for.
Hi, I am using a similar technique as you do. But to repeat the steps again is quite new for me. Works quite well. But I am facing a problem that codex frequently does not import all needed libraries if it comes with the solution that needs them.
For example, it does not import the base64 library. I, as a prompt developer probably know that I will need pandas and streamlit. But if codex uses a solution with base64 or something else is something that I do not know beforehand. Do you have any idea how to let codex import all needed libs? :
"""
Create a Python script that uses the 'Streamlit' module and the 'Pandas' module to create a data webapp.
Tasks:
1. Import the necessary modules we'll need for web app and initialize any objects we will reference
2. Load the data with the file uploader.
3. When data are loaded, compute square of the data from the first column.
4. Create a link with the text 'download'.
5. Show the results.
"""
# 1. Import the necessary modules we'll need for web app and initialize any objects we will reference
import pandas as pd
import streamlit as st
import numpy as np
# 2. Load the data with the file uploader.
st.title('Data Explorer')
st.write("Upload a csv file to see its data.")
file = st.file_uploader('', type='csv')
if file is not None:
df = pd.read_csv(file)
st.write(df)
# 3. When data are loaded, compute square of the data from the first column.
st.write("Square of the data from the first column.")
col = st.selectbox('Select a column to square:', df.columns)
df['squared'] = df[col] ** 2
st.write(df)
# 4. Create a link with the text 'download'.
st.write("Download the result as a csv file.")
st.markdown(get_table_download_link(df), unsafe_allow_html=True)
# 5. Show the results.
def get_table_download_link(df):
"""Generates a link allowing the data in a given panda dataframe to be downloaded
in: dataframe
out: href string
"""
csv = df.to_csv(index=False)
b64 = base64.b64encode(csv.encode()).decode() # some strings <-> bytes conversions necessary here
href = f'<a href="data:file/csv;base64,{b64}">Download csv file</a>'
return href
I gotta say I’m happy to see I’m not the only one with the problem of missing imports etc.
Thanks for this suggestion. I look forward to giving it a shot.
Hello @tomas.batrla,
I wondered if there was an easy way for Codex to include the base64 module, but it’s a cheeky workaround where you simply tell it to use the module and why.
- Import the necessary modules we’ll need for web app and initialize any objects we will reference.
From your prompt, I added this next statement and Codex did import the base64 module:
- Import the necessary modules we’ll need for web app and initialize any objects we will reference. Also use the base64 module that we’ll need to download files from Streamlit.
However the completion now takes out the numpy module since Codex seems to be able to complete the code without using the numpy module 
I check it at the moment but I think it’s correct.
Thanks for the suggestions @DutytoDevelop . Getting rid of numpy is fine since it is not used. Explicitly mention base64 is probably the ok solution, but if you find some solution for this in the future, please let me know. Maybe we can use some auto-import postprocessors, but since it is not a codex solution, it is more a hack than a real solution for me.
There is some work to be done with Codex, I agree. I did manage for Codex to import the base64 function without needing to explicitly tell it, however, that came with more problems that I believe are a little out of Codex’s current knowledge base (they were out of mine too until further examination).
When you upload a file to Streamlit, the cursor is at the end of the stream after the first evaluation. A workaround is by doing uploaded_file.seek(0) before reading the CSV file to reset the cursor back to the top of the file in order for Streamlit to write out the data (Source: Issue #2213). Since Streamlit 1.0.0 won’t even work on my computer and 0.89.0 hasn’t fixed this issue, I believe uploaded_file.seek(0) is the primary solution but Codex does not pick this up.
If you’re trying to go with a perfect solution here, I believe Codex will have to know the cursor of the uploaded_file during evaluation which it appears it does not because that would mean extensive knowledge of how Streamlit works (at multiple versions). As stated by FrankKr in Issue #2213, this wasn’t an issue in 0.64.0 and so Codex may not know that this is a problem.
I know what you’re looking for is to write the perfect prompt where Codex will give you the desired working code you’re looking for. However, since we know Codex will make mistakes sometimes, it’s best to ensure that whoever uses it will review the code and “smooth out the wrinkles”, so to speak.
Did manage for Codex to import the base64 function without needing to explicitly tell it
could you please tell me how the prompt looks now? btw. I am looking to create an app for sharing working prompts since it looks like make good prompt is sometimes bigger problem than do the code itself :]
Here’s the exact prompt I used @tomas.batrla:
"""
Create a data web app using the 'Streamlit' and 'Pandas' module that allows you to upload a CSV file, compute the square of the first column, and then generate a download link to that data.
Tasks:
1. Import the necessary modules we'll need for the web app.
2. Display the data from file uploader and compute the square of the first column.
3. Show the new data and then create a link for us to download it.
"""
"""
1. Import the necessary modules we'll need for the web app
"""
The completion from that prompt is:
"""
Create a data web app using the 'Streamlit' and 'Pandas' module that allows you to upload a CSV file, compute the square of the first column, and then generate a download link to that data.
Tasks:
1. Import the necessary modules we'll need for the web app.
2. Display the data from file uploader and compute the square of the first column.
3. Show the new data and then create a link for us to download it.
"""
"""
1. Import the necessary modules we'll need for the web app
"""
import streamlit as st
import pandas as pd
import numpy as np
import base64
"""
2. Display the data from file uploader and compute the square of the first column.
"""
st.title("Data Explorer")
st.write("This app will help you explore a CSV file.")
# Create a file uploader widget
uploaded_file = st.file_uploader("Choose a CSV file", type="csv")
# If the user uploads a file, then display the data
if uploaded_file is not None:
data = pd.read_csv(uploaded_file)
st.write(data)
# Compute the square of the first column
data_sq = data.iloc[:, 0] ** 2
st.write(data_sq)
# Create a link to download the data
csv = data.to_csv(index=False)
b64 = base64.b64encode(csv.encode()).decode()
href = f'<a href="data:file/csv;base64,{b64}">Download CSV File</a> (right-click and save as <some_name>.csv)'
st.markdown(href, unsafe_allow_html=True)
It’s super close to working, however modifying the prompt even slightly results in big differences in the code and I believe it’s because of the confidence of the program when working with the prompt above. The more information you provide Codex, like explicitly telling it to use a module, for example, will yield more accurate completions. I can see how beginners who aren’t familiar with programming or knowledgeable with certain libraries/modules can benefit from Codex to fill in that knowledge gap but may find it harder to troubleshoot lesser known issues.
Anyways, I was trying to get Codex to provide a link to the new data at ‘csv = data.to_csv(index=False)’ where ‘data’ should be ‘data_sq’, and not the original data which is what the completion gives you. I’m sure spending more time on tweaking the prompt will give you the results you’re looking for. If you’re on the Playground, be sure to thumbs up the completion that gives you the output you’re looking for. 
Everything gets easier with practice! I’m certainly still learning how to write prompts but I’m certainly getting the hang of it!
What are the other values you use in the playground or API like ‘Response length,’ ‘Temperature,’ ‘Top P,’ etc.? So far, I’m using their examples to guide my settings here, but I do want to punish the AI a bit for ‘Frequency penalty’ and ‘Presence penalty.’
I load the “Natural language to python” example and tweak it from there. Once I have something viable to report from working with Codex more, I will report back, but so far, the settings from this example have been the most interesting.
I usually use the following settings:
Response length: (Best guess - I usually give the models enough tokens so that completions don’t get cut off if their responses are larger than expected)
Temperature: 0
Top P: 1
Frequency penalty: 0
Presence penalty: 0
Best of: 1
Depending on how much you tweak the Frequency and Presence penalty settings, you may get better or worse results. I would keep those values on the low side if Codex’s completions are referencing variables quite often since I believe those would be the tokens most affected by that setting.
I personally don’t like having penalties, especially with Codex, unless I know the desired output I’m looking for won’t need those same tokens. Here’s a quick example where having really high penalties caused Codex to go off the rails in a few lines (~1.4 for both):
Even having the penalty setting values set to ~.21 yields undesired results. For example, I have Codex write comments before code blocks to help it stay on track, but with the penalty settings being present, it penalizes the ‘#’ character since it’s how you write single-line Python comments, so it knocks itself off track.
Try to find the sweet spot if you really do want to have penalties. May I ask why you want those settings present?
Side note: I just realized that I haven’t been messing with the ‘Best of’ setting and so I am going to start using it more often to see how it changes the output!
Tangentially, related to getting the imports right: Often you realize that imports are missing after the code was already generated, and especially when they make the api non-free it’s annoying to have it start from scratch. Instead I found that just doing something like
# let's do all the necessary imports for the code above:
imp
Works perfectly 99% of the time