What will be prompt for extracting entity extraction , summarization and classification

[msusman97]

Hi,
I have created an expense tracker app that will extract 4 things from voice transactions commands

1. Transaction Summary
2. Amount
3. Organization
4. Category
   with fine tuning ada model with below dataset for classification and summarization.
   {“prompt”:“Summarize the transaction: Purchased a facial cleanser for 25 dollars at Ulta Beauty \n\n###\n\n”,“completion”:" Purchased a facial cleanser\n"}
   {“prompt”:“Transaction classification: Purchased a facial cleanser for 25 dollars at Ulta Beauty. Category: \n\n###\n\n”,“completion”:" Healthcare\n"}

I want the open ai completions api return the response like this
organization: Ulta Beauty
amount: 25$
summary: Purchased a facial cleanser
category: Healthcare

Can you please give me suggestion how can design prompt to get above required info from transaction?
Is there any thing i did wrong for data preparation?
Can we extract the above info without fine tuning model?
Your help will be appreciated.
Thanks

[sergeliatko]

@msusman97

Purchased a facial cleanser for 25 dollars at Ulta Beauty

here is some suggestion for you APP , let me know what you think JSON Structure: Expense Tracking

[sergeliatko]

I would use a separate prompt for each field and maybe fine-tune each model separately to extract just that field (or return default value or “fault” code on failure) so that the task is simple and cheap to get high quality response. The just send the prompts in parralel. If fine-tuning the format may be reduced to :

Example merchant extractor model:

{“prompt”:“Purchased a facial cleanser for 25 dollars at Ulta Beauty<|endoftext|>”,“completion”:" Ultra Beauty<|endoftext|>"}

and failure example

{“prompt”:“Purchased a facial cleanser for 25 dollars<|endoftext|>”,“completion”:" <|null|><|endoftext|>"} so that the app can check if the response is an error or no-data type.

[msusman97]

Thanks Serge for your explanation.

I need to call api throgh OpenAi Rest endpoint
here in attached screenshot i am call completion api for custom model. for classification and summarization which is fine(even though api return duplicate response)

i need to combine them into 1 api call to reduce api cost
when i update the prompt to some thing like this:
Summarize and classify the transaction, and also return amount, currency and organization in json : Purchased a facial cleanser for 25 dollars at Ulta Beauty.
not getting the required response.

expected response should be:
{
“summary”: “Purchased a facial cleanser”,
“amount”: 25,
“currency”: “USD”
“organization”: “Ulta Beauty”,
“category”: “Healthcare”
}

i have attached my training set.Create a new version of paste: - Pastebin.com
and training ‘Ada’
By profession I am android developer new to Openai.

Thank you again for your valuable time

[sergeliatko]

On July 6, 2023, we announced the deprecation of ada, babbage, curie and davinci models. These models, including fine-tuned versions, will be turned off on January 4, 2024. We are actively working on enabling fine-tuning for upgraded base GPT-3 models as well as GPT-3.5 Turbo and GPT-4, we recommend waiting for those new options to be available rather than fine-tuning based off of the soon to be deprecated models.

Basically means using ada fine-tuning (and other older fine-tuned models) will be deprecated by Jan 4, 2024.

I would start directly with gpt-3.5-turbo to get the required result and collect data for further fine-tuning of gpt-3.5-turbo when available.

Quickly going through your training set, some notices:

• the examples look pretty much like those generated by a machine because they are all in a finite set of grammatical structures with poor language variations (have they been generated by an LLM or come out preformatted from a previous stem of your assistant APP ?)
• example number is too short for a decent training of ada (725 unique items) do you have more real-life examples?

How I would personally approach the task (let me know your thoughts):

1. study features of similar apps like the wallet (from budget bakers) and decide if there is a chance you would support one day the features like:

• multiple accounts (cash, cards, bank 1, bank 2 etc)
• multiple currencies
• custom labels (when the app reads various input and reformats it in a standard user-defined label)
• custom transaction categories
• custom transaction tags
• etc.

2. Make a list of all those features to build a roadmap of features and see which of them have to be foreseen (prebuilt) on the launch (not to recode the core engine when you need to add a feature later on)

3. The above will define (will help you define) the transaction object structure to be stored in the APP DB:

{
    "transactionId": "1234567890",  // A unique identifier for the transaction. This can be automatically generated by the backend.
    "description": "Purchased a facial cleanser",  // A brief description of the transaction. This can be directly extracted from the "userInput".
    "amount": {
        "value": 25.0,  // Transaction amount. Extracted from the "userInput".
        "currency": "USD"  // Currency type. This can be inferred based on the region or explicitly provided.
    },
    "merchant": "Ulta Beauty",  // The store or service where the purchase was made. Extracted from the "userInput".
    "category": "Beauty & Personal Care",  // A broader classification of the transaction. This can be automatically inferred using the description and merchant, or selected by the user.
    "suggestedCategory": "Beauty & Cosmetics",  // This can be an AI-generated suggestion or from a predefined list.
    "paymentMethod": "Credit Card",  // Specifies how the transaction was paid for. Taken directly from the input "paymentMethod".
    "timestamp": "2023-08-21T12:00:00Z",  // The time when the transaction occurred. Taken directly from the input "timestamp".
    "location": {
        "latitude": null,  // Latitude of the transaction location. Taken from the input "location".
        "longitude": null,  // Longitude of the transaction location. Taken from the input "location".
        "address": null,  // Physical address of the location (can be null if not provided). Taken from the input "location".
        "isExact": true   // Indicates if the note was made directly at the merchant's location. Taken from the input "location".
    },
    "tags": ["facial", "cleanser"],  // A list of keywords or tags associated with the transaction. This can be generated based on the "description".
    "userNote": "This is a special cleanser for sensitive skin",  // Notes added by the user about the transaction. Taken directly from the input "userNote".
    "userInput": "Purchased a facial cleanser for 25 dollars at Ulta Beauty"  // Original raw input by the user. Taken directly from the input "userInput".
}

Or similar. And also will help you to define the API input object format:

{
    "userId": "abcd1234",     // A unique identifier for the user making the request
    "userInput": "Purchased a facial cleanser for 25 dollars at Ulta Beauty",  // The original raw input string provided by the user
    "timestamp": "2023-08-21T12:00:00Z",  // The time when the user made the input
    "location": {
        "latitude": null,    // Latitude of the location where the user made the input (can be null if not provided)
        "longitude": null,   // Longitude of the location where the user made the input (can be null if not provided)
        "address": null,     // Physical address of the location (can be null if not provided)
        "isExact": true      // Indicates if the note was made directly at the merchant's location
    },
    "paymentMethod": "Credit Card",  // How the user paid for the transaction, e.g., "Credit Card", "Cash", etc.
    "userNote": "This is a special cleanser for sensitive skin"  // Any additional notes or details the user wants to add about the transaction
}

4. then decide on how many steps are needed to get the stored transaction object from the input (example):

• get input and parse to base fields extracted (all fields without the category)
• embed the user input and match it against the vector of existing categories to find the best match
• update the transaction object with found category and user input vector values
• store the transaction in the DB

or :

• get input and parse to base fields extracted (all fields without the category)
• embed the user input and match it against the vector of existing user-defined categories to find the best match
• if nothing fits, match the vector against the existing vectors of standard categories to get the one that fits
• update the transaction object with found category and user input vector values
• store the transaction in the DB

i need to combine them into 1 api call to reduce api cost

Well, the trade is Quality vs Cost, personally, I would take quality in the beginning (if the cost is financially not insane) to start collecting real-life data and see how I can optimize the cost later (using fine-tuned GPT3.5-turbo when available or switch to a different LLM or embedding model), because if quality lacks in the beginning, the chances are this will never take off.

5. for the DB I would use weviate.io as the best fit for my taste, but that is arguable

Here is an example of how the chat GTP3.5-turbo setup would look like: OpenAI Platform

[msusman97]

Hi Serge,

  Thank you  so much for your help. I really appreciate the time and effort you took to guide me through the solution. Your expertise and willingness to help made a big difference, and I'm very grateful for your support.  

Regarding the dataset, Yes most of the example transactions generated by the machines.
And also yes 725 transactions(most are less variants) are not enough for each category?
curranty we have 11 categories to target.
And you are right, we should prefer quality over cost saving at least at beginning.

Thanks again valuable suggestions and support!

[sergeliatko]

@msusman97

And also yes 725 transactions(most are less variants) are not enough for each category?
curranty we have 11 categories to target.

The aproach I’m talking about does not need the the training on categories… You actually need the list of categories and you need to decide if you want your users be able to edit/add/remove items from the list of categories.

Once you have the list, you embed (get vector) for each category separately, store them in your vector DB, when a transaction comes in, you embed the user input text and match the vector of transaction directly to one of the categories vector to get the category directly in the DB. The cost is peanuts to get the category, because you pay the category vectors once, and you pay only once per vectorizing the transaction…