My Assistant is not following instructions

API
1

I know there have been multiple complaints about this problem but I really don’t think my case is that hard to follow.
I have this big list of json elements composed of “categories” and “tags” for every category. (Example Below)
I give this list of categories and tags to the model, then I give it a job experience title and description (Example Below).
In the system prompt I tell him to choose the most matching category and max 5 tags from that chosen category.
It never works, really, has not worked once. The only problem I get is that the model picks tags from different categories, no matter how much I stress the fact it can only take tags from the chosen category (I tried to tell it in many different ways).

Example Title and Description:
“title”: “Software Engineer”,
“description”: “Worked on various web projects using Python, Node.js, Kubernetes and Tensorflow.\nDeveloped powerful AI models to release for the world to use freely.”

Example Category and Tag list:
: “category”: “Full-Stack Development”,
“tags”: [
“JavaScript”,
“Node.js”,
“Express.js”,
“React.js”,
“Vue.js”,
“Ruby on Rails”,
“Python”,
“Django”,
“Java”,
“Spring Boot”,
“SQL”,
“NoSQL”,
“MongoDB”,
“PostgreSQL”,
“Git”,
“API Development”,
“Heroku”,
“Docker”,
“AWS”,
“CI/CD”
]
},
{
“category”: “Front-end Development”,
“tags”: [
“HTML5”,
“CSS”,
“SASS”,
“LESS”,
“JavaScript”,
“ES6”,
“TypeScript”,
“React.js”,
“Vue.js”,
“Angular.js”,
“Redux”,
“Vuex”,
“Bootstrap”,
“Material-UI”,
“JQuery”,
“Bulma”,
“Webpack”,
“Babel”,
“NPM”,
“Yarn”,
“GraphQL”,
“Ajax”,
“Web Accessibility”,
“UI/UX Design”,
“Performance Optimization”,
“Responsive Web Design”,
“Cross-Browser Compatibility”
]
},

2

Which model are you using? I assume 3.5?

You say ‘example category list’ - do you have (many) more elements in the ‘real’ list? How big is your total prompt?

gpt-4 models should be ok with this.

3

I tried with both 3.5 and 4 and none work (I was hoping I could make 3.5 work though). The total list is about 7,400 tokens, is that maybe too much? Although I tried reducing the list drastically, to even just not even a thousand tokens probably, and it never gets it right. I tried entering the list in prompt and in system prompt, none work. I also tried to give it the json file and make it read the list from there, and as always…

4

If if it is more than 32k length I think that is where it goes wrong. But if your list is that long - why not let GPT just make up the tags - chances that they are not in the list are very low?

5

It is definetely shorter than 32k length. I really can’t let him make up the tags because I need them to be coherent to my software mechanism. The tags need to be matching.
I mean how can it be this hard for him to have strict rules, he can do the “thinking job” just fine, he messes up in following the rules.

6

It’s the size of your list. It’s simply too long for it to work. I am talking 32k characters not tokens BTW.

7

But it does not work even when I give it just one category and its tags. I will show you. I always get this result, I tried with many different system prompts.

Screenshot 2024-03-26 at 12.19.18
Screenshot 2024-03-26 at 12.19.181462×937 158 KB

Here is the title and description: “title”: “Software Engineer”,
“description”: “Worked on various web projects using Python, Node.js, Kubernetes and Tensorflow.\nDeveloped powerful AI models to release for the world to use freely.”, and as you can see it adds kubernetes, that is not in the tags list of Full-Stack development

8

Can you share the full instructions that you used in the playground? That we can give it a run too.

9

Here are the entire instructions that I used on the assistant in the example given above. "Task: Process job descriptions to categorize and tag accurately using only the provided list.

Rules: - Only use tags within the matched category. Avoid using tags from unrelated categories. - If no exact tag matches, omit the tag rather than borrowing from other categories. - Output format: Only the matched category and applicable tags.
For example, given this category: “{
“category”: “Cybersecurity & Ethical Hacking”,
“tags”: [
“Penetration Testing”,
“Network Security”,
“Application Security”,
“Endpoint Security”,
“Cryptography”,
“Firewall Management”,
“IDS/IPS”,
“Vulnerability Assessment”,
“Malware Analysis”,
“Risk Assessment”,
“Incident Response”,
“Security Compliance”,
“Ethical Hacking”,
“Forensic Analysis”,
“OWASP”,
“Social Engineering”,
“Phishing”
]
},” you may only select the tags inside
the "{}”, you may only select tags inside the parenthesis with the category chosen.

Stick to the provided category and tags list. Do not infer or add new ones."

10

Thanks - do have a few samples to process as well?

11

I don’t have a predefined list even because if it doesn’t work with that prompt it is enough of a problem for me. Although I just checked and you can generate how many you want, just give chat gpt this: ““Generate examples of job experience titles and descriptions just like this but for different roles “” “title”: “Software Engineer”,
“description”: “Worked on various web projects using Python, Node.js, Kubernetes and Tensorflow.\nDeveloped powerful AI models to release for the world to use freely.”””“”. Also, the problem will be you will not have the full list of categories and tags for other types of jobs as well. Here is github gist with the entire list “CategoriesAndTags · GitHub”

12

Btw, thank you man, appreciate the help!

14

Ok - I understand now.
I think it would be better to split this up into two jobs. First you provide it with a list of categories and you let it choose the category
The you let it call you tool function that will provide the tags for that category and you let it choose tag(s) from the list

15

Thank you for this amazing suggestion, I am testing it, apparently it still doesn’t work with gpt-3.5 (strange, it seems a task so easy!), although with gpt-4 I seem to be getting consistent right responses, I think this will be the solution I try, thank you !

1 Like
16

glad that worked. 3.5 is really not up to those types of tasks.