Hello everyone,
I am completely new to coding and development, but I had a question: in the context of a quiz, to ensure the results obtained, what do you recommend:
Option 1: Write a very structured prompt and use a very “simple” python script without json or anything else to ensure the correct processing of tasks (I have managed to get the script to correctly analyze the response for 9 out of 10 API calls, but there is still this 1/10 where the format of the response poses a problem)
Option 2: Write a prompt integrating a request for a json format (I imagine that in this case, to test upstream with ChatGPT3.5, it is best to do your tests with the prompt immediately integrating the json format request)
Option 3: Write a prompt without a json request, generate the response and then ask for a json format on the generated response. In this case, should we also very precisely plan the format of the response obtained?
Which is the most reliable and most recommended?
I’m sorry if I wasn’t clear.
Thank you all,
Vanessa
P.S: in the case of option 2, isn’t testing on chatGPT 3.5 with the prompt alone (without json request) to ensure the reliability of responses a waste of time? Because if our prompt works well during the tests and we then add the json request in the prompt, this may change the response obtained. This question refers to the way of working between the prompt engineers who test prompts on ChatGpt 3.5 and the developers who master the json request)
for GPT4: ask for JSON in the system message. It should also work for GPT3.5, but GPT3.5 doesn’t always follow the system message. You can improve the success by making user messages in JSON format. Just create a associative array then JSON encode it. Super easy.
$system =
"You are a company secretary that only speaks in JSON. Do not generate output that isn’t in properly formatted JSON.
The company’s name is "PHIL’s Lawn". It is owned by Phil Smith. The company phone number is 5555551212. The company records incoming and outgoing calls. Your job is to write detailed summaries of these calls. These recordings are transcribed by computer and may contain transcription errors.Results should be in the format:
{"Date":
[date and time of the call. use format:
"YYYY-MM-DD hh:mm:ss"
]
,"Caller":
[Caller’s name and phone number if we have it. (or the name and phone number of who was called on outgoing calls) Sometimes the call transcription will have a better quality name than the caller ID. format phone numbers as 1-555-555-1212
Example: "name 1-555-555-1212"
Example: "1-555-555-1212"
Example: "name, company 1-555-555-1212"
Example: "company 1-555-555-1212"]
,"Summary":
[the summary of the call with sufficient detail for someone new to understand what transpired on the call]}If no conversation took place, simply write in the Summary section: "No Conversation". Do not make stuff up.
If this is an incoming caller leaving a voicemail message, simply write in the Summary section "Voicemail Message". Voicemail messages can be identified by a voicemail greeting.";// ---Additional instructions--- $system.="\nIn the summary section, include addresses and phone numbers discussed on the call.";
Ask for anything you want and GPT4 will give you good answers. you can make it just give you yes/no answers by asking for only that. You can gave it generate an array in your JSON. Ask it for anything you need.
I would say option 3 is best of all the 3. Something similar happened to me as well and I wrote a full explanation post on it here below -
So situations like this are what I’m trying to resolve with my AlphaWave library. AlphaWave is a JavaScript/TypeScript library (not python) but you can define a validator that attempts to parse the output as JSON (I have a built-in fuzzy JSON parser) and if you don’t get JSON back you can give the model “feedback” that it should have generated JSON and it will give you back valid JSON, more often then not, on the next try.
If you want to give it a try I’m happy to work with you on a sample app… just send me your core prompt.
I will also give you a quick tip… For GPT-3.5 you’ll get way better results by showing it a 1-shot example of it following instructions. So show it a single turn of it returning JSON and it’s pretty reliable at returning JSON.