Multi class classification with response only from trained classes

Prompting
1

Hello,

I have to assign intents to sentences. e.g. (Query) 6s conceded by PLAYER against TEAM in 2020 - runs_conceded (Intent)

For ChatGPT, I tried by giving ~130 sentences along with their intents and then asked to classify new sentences. It worked perfectly well, but the problem was I had to give all ~130 sentences for each query, as it has no memory. Not surprisingly, this proved expensive.

So, I started training “ada” and “davinci”, with these 130 sentences along with their intents. But the responses on new sentences are way off. I tried replacing intents with numbers (For token differentiation, as matches_played and matches_won are nearly same), it also had no effect.

Then, I increased the training set to 1200 sentences, with nearly same number in validation set. The responses came out even more horrible for nearly all sentences. Also, both the “ada” and “davinci” are not restricting intent response to pre-trained dataset. They are coming up with intents that are totally new.

Any help as to what I should do and what I am doing wrong. Thanks.

2

How many intent categories are there? I used gpt-3.5-turbo as a classifier with about a dozen different categories. I provided only the category names (which were fairly descriptive, but still short) and a single example (not one per category—one single example), and it has worked without issue. So you might start with fewer examples, or by clearly laying out the categories and descriptions, and rely less on examples.

3

I have like 54 intents, I gave 130 training examples…
Did you fine-tune? For fine tuning, is 3.5 turbo available?

4

I did not fine-tune and I don’t think fine-tuning is available for gpt-3.5-turbo.

5

Could you give some idea of your description and the 1 example? Maybe I am doing it wrong…

6

My prompt looks like this

    """Your task is to determine which of the following tasks the user's prompt falls into.

    user prompt: {user_prompt}

    tasks:
    - execute operation
    - create a new object
    - update an object
    - import object from a URL
    - list metadata for all objects
    - get metadata about a specific object
    - create an API key 
    - revoke all API keys
    - export an object
    - delete an object
    - no task specified

    return only the task name. For example:

    user: Provide information about my objects
    assistant: list metadata for all objects

    user: Load data from this URL 
    assistant: import object from a URL
    """

Not sure how well this would scale to that many options, though.