I am trying to build a prompt that would extract the question from a text along with bunch of actions. ex:
question: Summarize my meeting notes and share it with my team on slack
response: { question: “Summarize my meeting notes”, actions: [{“SHARE_SLACK”: [“team”]}]}
another example:
question: Draft a new MSA for Acme Corp based on their old one and share it with my team, date for today
response: { question: “Draft a new MSA for Acme Corp based on their old one”, actions: [“CREATE_DRIVE_FILE”, {SHARE_DRIVE_FILE: [“team”]}] }
Hi @bads, I’ve had a fair bit of success getting GPT to generate JSON’s by providing the sample output to it in the JSON format, while prompting GPT to generate the output in the same format as the sample output.
For a check, I have used a try and except in python, where it catches a failed json.loads and redoes the generation if the output is not in JSON format.
There’s so much about this thread that hurts my head.
Correct me if I’ve misinterpreted your intentions – you want the human interface to behave like the machine interface? If so, to what end? Is this supposed to help people learn how to construct JSON response payloads? Or do you then copy and paste these responses into another app of some type?
“Fair” is not “good”, and when it comes to JSON used in any context, “fair” is tantamount to failure. An API interchange is a pass/fail proposition. A 60% success rate is largely considered unacceptable. In the world of conveying data using JSON, we have a word for what you’re trying to accomplish - it’s called an API response.
I rarely ask inquirers why, but in this case, I must because this is nonsensical without understanding your motivations.
Thanks for the responses. @udm17 I am getting JSON responses almost all the time as I have provided 4 examples but the problem is my confidence is still low.
Here what I am seeking is better examples and prompt. below is my complete prompt, please let me know how can I make it better. @bill.french Thanks for your response as well There is no harm is trying something, I believe and also we don’t yet know the full capability of the model yet so who knows if it can provide me results 100% of the time with the help of you guys?
You are a human like bot which parses the input and extracts the set of actions from that.
You extract the labels such as below and generate the labels as required based on real services such as google drive, dropbox, asana etc. You return the response strictly as JSON
<labels>
1. DRAFT_MSA
2. CREATE_GOOGLE_DRIVE
3. SHARE_GOOGLE_DRIVE
4. SUMMARIZE
5. SEND_SLACK
6. SEND_EMAIL
9. TIMEFRAME
</labels>
Q: draft a MSA based on acme corp document, create it in google drive and share it with my team
{"question":"draft a MSA based on acme corp document","actions":["DRAFT_MSA","CREATE_GOOGLE_DRIVE",{"SHARE_GOOGLE_DRIVE":["team"]}]}
Q: Draft MSA for acme corp inside google docs and share it with arnold and jenny
{"question":"draft a MSA for acme corp","actions":["DRAFT_MSA","CREATE_GOOGLE_DRIVE",{"SHARE_GOOGLE_DRIVE":["arnold","jenny"]}]}
Q: Draft a new MSA for Acme Corp based on their old one and share it with my team, date for today
{"question":"Draft a new MSA for Acme Corp based on their old one","actions":["DRAFT_MSA","CREATE_GOOGLE_DRIVE",{"SHARE_GOOGLE_DRIVE":["team"]}]}
Q: how many tickets have we received in the past week? share the details on slack
{"question":"how many tickets have we received in the past week?","actions":["FILTER_TICKETS",{"TIMEFRAME":"WEEK"},{"SEND_SLACK": ["me"]}]}
Q: Summarize our investor notes and share it with my team on slack
Indeed, there isn’t. And there’s nothing inherently wrong with asking the LLM to generate JSON (which is ostensibly code).
Just an idea - suppose the goal is to use NLP to make something automated happen (I believe that’s the vector of your solution). In that case, I might suggest you experiment with the transformation from language into a common set of pseudo-instructions and then use those outputs to trigger the automated tasks.
This would move the JSON serialization process away from the responsibility of the LLM and thus, performed inside an app designed to:
Listen for conversational inputs
Use the API to harmonize/interpret the meaning
Serialize the meaning into JSON
Dispatch the automated process(es) with meaningful JSON payloads
I fell that I might have put my response up as a bit of an open-ended statement because as of yesterday, I have a 98% success rate at getting gpt-3.5 turbo to generate JSON’s. When i meant fair bit of success, what i meant getting GPT to undertake different tasks and getting all these responses back as a JSON file.
Currently, I do not know of a software that will allow me to return the output message from the GPT API as a JSON directly, so this is the only way i know. If you have one, please let me know and I’ll be happy to use it if it performs better.
The JSON serialization process you have mentioned here is the go-to, but for my task I had the harmonize/interpret step be dynamic and an API was not able to perform the task as good as GPT was doing and that’s why I am using the method I am using right now.
Understood. I referenced the “API” only in the sense of using OpenAI APIs to perform non-JSON generation and instead use it for finding the entities and actions that are required in your JSON output.
Given this prompt:
draft a MSA based on acme corp document, create it in google drive and share it with my team
I think I would have come at this solution by using the LLM to extract the entities (MSA, Acme Corp, and Team), classify the prompt based on the actions, and generate the JSON from these values. In this approach, JSON is a function of the AI process, and the AI process is a function of the prompt.
I’m uncertain how to perform the classification (haven’t done much with classifiers), but I’ll bet some more experienced GPT developers know how to achieve this.
It’s quite possible your approach is the better choice.
I found a solution and it works 99% of the time. Only faced a couple instances in last two days where it failed. Here is an example:
Given the question fill out the below schema only for the available values and return valid JSON:
{
"label": string,
"summarize": string[],
"summarize/notion": string[],
"summarize/drive": string[],
"summarize/email": {
"filter": string[],
"timeframe": seconds | null
},
"count": {
"notion": {
"filter": string[],
"timeframe": seconds | null
},
"github": {
"filter": string[],
"timeframe": seconds | null
}
},
"draft": {
"content": string[],
"reference": string[]
},
"create/drive": boolean,
"share/drive": string[],
"share/slack": string[]
}
Q: draft a MSA based on acme corp document, create it in google drive and share it with my team
{
"label": "Draft MSA",
"draft": {
"content": ["MSA"],
"reference": ["acme corp document"]
},
"create/drive": true,
"share/drive": ["team"]
}
Q: Draft a new MSA for Acme Corp based on their old one and share it with my team
{
"label": "New MSA draft",
"draft": {
"content": ["MSA"],
"reference": ["acme corp document"]
},
"create/drive": true,
"share/drive": ["team"]
}
Q: how many github tickets have we received in the past week? share the details on slack
{
"label": "Github ticket count",
"count": {
"github": {
"filter": [],
"timeframe": 604800
}
},
"share/slack": []
}
Q: Summarize layup demo and share it with tony and bads on google drive
If I had a nickle … LOL Great to see you are able to dial this in. Lots to learn about your approach.
I get nervous when I see claims like this for any AGI application. I’m sure it’s meeting your satisfaction threshold, but isn’t it more accurately stated:
I found a solution that works 99% of the time when tested with known cases that are well within my ability to generate and test use cases.
As a person charged with testing the inferencing accuracy of something like this, the very nature of prompt testing is an enormous and often complex task. Anecdotal testing is fine but unlikely to represent the actual rate of success.
I would say it something which vary’s from problem to problem right ? While yours is definitely a highly complex situation, hiw is a bit simpler where he can afford to have missed some information which he can generate.
My comment, though, relates to outcome assessment – e.g., how we get to the point where we can state 99% (or whatever) accuracy in a given prompt or trained approach. My advice is to be careful of claims unless you have the data to back it up, and such data is based on a rigid test protocol.
As testers of our own solutions, we bias outcomes in two ways.
Humans tend to “see” what they want to see;
Developers tend to validate what they built.
We know how it’s built; simply possessing that knowledge subconsciously shapes our test protocols.
