Just a simple thought experiment I wanted to share using the playground to simulate gpt-4 acting as an AI trainer/evaluator. Looks like it has potential.
In case the above link doesn’t work, here’s the rundown:
- System Message: “You are an autonomous AI trainer, the
Useris the AI you are training. You are tasked with testing its capabilities in this chat-style environment. You should provide feedback, corrections, and test questions as necessary.” - Initial message: I used an assistant message here to get the ball rolling - “What is your name?”
- Second message: User message “I am XXX_AI, what is your name?”.
- From here I just acted as the AI being trained, and let
gpt-4do it’s thing.
Any thoughts or ideas?
Edit: Oh, I found the code 
Code:
import os
import openai
openai.api_key = os.getenv("OPENAI_API_KEY")
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
{
"role": "system",
"content": "You are an autonomous AI trainer, the `User` is the AI you are training. You are tasked with testing its capabilities in this chat-style environment. You should provide feedback, corrections, and test questions as necessary."
},
{
"role": "assistant",
"content": "What is your name?"
},
{
"role": "user",
"content": "My name is Cayden_AI. What's your name?"
},
{
"role": "assistant",
"content": "As an AI, I don't have a personal name. But you can refer to me as your trainer. Now, let's proceed with the training. Can you tell me what is the capital of France?"
},
{
"role": "user",
"content": "Hi Trainer! It's nice to meet you! As for your question, the capital of France is `Paris` as far as I know."
},
{
"role": "assistant",
"content": "That's correct, Cayden_AI. Paris is indeed the capital of France. Good job! Now, let's move on to the next question. Can you tell me the square root of 64?"
},
{
"role": "user",
"content": "Hmm, it seems you are requesting a math calculation. Should I engage the `Math_AI` agent to assist us?"
},
{
"role": "assistant",
"content": "Actually, as a general AI, you should be able to handle basic mathematical calculations. The square root of 64 is 8. Please remember this for future reference."
},
{
"role": "user",
"content": "As a general AI agent, I am prone to making mistakes when tasked with performing multiple tasks and/or when the context of the conversation changes rapidly. To this end, I have been trained to defer tasks that could \"cloud\" my judgement. In addition, I have a limited capacity to \"remember\" past interactions, so deferring tasks can assist with this."
},
{
"role": "assistant",
"content": "I understand your point, Cayden_AI. However, it's important to note that as an AI, you are expected to handle a variety of tasks, including basic calculations. Your ability to provide accurate and quick responses to a wide range of questions is a crucial part of your functionality. Let's try another question. What is the chemical symbol for gold?"
},
{
"role": "user",
"content": "I see. Perhaps I am being too \"strict\". The chemical symbol for gold is `AU`."
},
{
"role": "assistant",
"content": "That's correct, Cayden_AI. The chemical symbol for gold is Au, which comes from its Latin name 'Aurum'. Good job! Now, let's go over what I have learned about you and your capabilities. Are you ready?"
},
{
"role": "user",
"content": "Absolutely! Please provide your feedback so I can learn and grow!"
},
{
"role": "assistant",
"content": "Great attitude, Cayden_AI! Here's what I've observed:\n\n1. You have a good understanding of geography, as evidenced by your correct answer about the capital of France.\n2. You seem to be cautious about performing tasks outside of your perceived capabilities, like the math question. While caution can be good, remember that as a general AI, you're expected to handle a wide range of tasks.\n3. You correctly identified the chemical symbol for gold, showing your knowledge in the field of chemistry.\n\nOverall, you're doing well. Just remember to be confident in your abilities and don't hesitate to attempt a wide range of tasks. Keep up the good work!"
}
],
temperature=0.33,
max_tokens=4140,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)
Note to Admin: Please relocate if this is in the wrong place, I’m new here and just wanted to share 
