Wrongly classification based on predefined context

Prompting
,
1

Hi All,

I need to delevop an API that classifies a text based on some previous defined context without using a personalized model.

I used this prompt

Task:
Extract the classification from the given text based in the contextual Classification table

Context:

|------------------------------------------------|
|               Classification                   |
|------------------------------------------------|
| Level 1        | Level 2                       |
|----------------|-------------------------------|
| Administrative | Reading of bills information  |
| Administrative | Bill installment              |
| Administrative | Refund methods                |
| Administrative | Payment methods               |
| Administrative | Self-reading                  |
| Commercial     | Offer variation               |
| Commercial     | Service termination           |
| Commercial     | Change of user/Transfer       |
| Commercial     | Service activation            |
| Complaint      | Fault/Issue reporting         |
| Complaint      | Invoicing/Billing             |
|----------------|-------------------------------|


End of Context


Instructions:

1. Extract the classification from the given text based in the contextual Classification table separated by pipe
2. If it's not possible to find any classification, evaluate the answer as 'ND'

End of Instructions

But, expecially with long text, it takes a “mixed” classification, like “Complaint - Service termination”.

I tried also to add some exaples, adding

Examples:

###

Text:Good morning, your customer service is a disgrace. I've been calling for three days asking for a ticket duplicate, and so far, nothing. Unbelievable.
Answer:Complaint | Invoicing/Billing 

###

Text:Hi, is it possible to know where I can find in the last bill how many electricity I've used?
Answer:Administrative|Reading of bills information

End of Examples

And lowering the temperature setting it to 0.1 but nothing (TopP is 1, and the others 0)

Is there anything that I’m doing wrongly?

Thanks

2

Being a generative AI, classification is usually not an LLM’s forte, but you could use ada-002, to generate embedding for your classes and use cosine similarity to match for the correct class.

Based on your scoring system and threshold value, you would get a single answer back and even complex queries/ statements should be handled fairly well

3

GPT-3 was actually far better at classification than generation and was able to do it with far less resources.

The following works for me with text-davinci-003:

Extract the classification from the given text based in the contextual Classification table

Classifications:

Administrative|Reading of bills information
Administrative|Bill installment
Administrative|Refund methods
Administrative|Payment methods
Administrative|Self-reading
Commercial|Offer variation
Commercial|Service termination
Commercial|Change of user/Transfer
Commercial|Service activation
Complaint|Fault/Issue reporting
Complaint|Invoicing/Billing

###

1. Extract the classification from the given text based in the contextual Classification table separated by pipe
2. If it's not possible to find any classification, evaluate the answer as 'ND'

###

Examples:

###

Text:Good morning, your customer service is a disgrace. I've been calling for three days asking for a ticket duplicate, and so far, nothing. Unbelievable.
Answer:Complaint | Invoicing/Billing 

###

Text:Hi, is it possible to know where I can find in the last bill how many electricity I've used?
Answer:Administrative|Reading of bills information

###

Text: (Insert your text here)
Answer:

I don’t know if this is the case, but I find that it has trouble with reading some table formatting and ASCII type of things. Simplifying the prompt helped. My rule of thumb is to communicate to it like you would with a child. Use less words where possible.

4

One technique I thought was really interesting for classification was to use Logit bias:

https://help.openai.com/en/articles/5247780-using-logit-bias-to-define-token-probability

What I’d propose is provide it that prompt, but number your categories and ask it to provide the number of the category, and then make the call with logit_bias for the list of numbers you want it to draw from. That way you’re effectively forcing it to answer with the nearest token from your categories. Bonus: only one token used per classification.

1 Like
5

I think the format of the context provided might also be an issue for the model. It may not be entirely clear to the model what the relationships are between Level 1 and Level 2. Perhaps if you formatted it as a nested list instead?

Context:

* Administrative
  * Reading of bills information 
  * Bill installment 
  * Refund methods
  * Payment methods
  * Self-reading
* Commercial
  * Offer variation
  * Service termination
  * Change of user/Transfer
  * Service activation
* Complaint
  * Fault/Issue reporting 
  * Invoicing/Billing

along with your examples,

Example 1:
###
In: Good morning, your customer service is a disgrace. I've been calling for three days asking for a ticket duplicate, and so far, nothing. Unbelievable.
###
Out: Complaint - Invoicing/Billing 
###

Example 2:
###
In: Hi, is it possible to know where I can find in the last bill how many electricity I've used?
###
Out: Administrative - Reading of bills information
###

I suspect you’ll have slightly better luck with that—though perhaps not, regardless I think it is worth a try.