ChatGPT does not give correct questions then it has the context

Prompting
, , ,
1

I am using assistants api to get information about human resources, I get the context with rag, but chatgpt does not response correctly.

In base of the information of an user gotten from sql agent I need to response with the information.

In this example I get the context correctly, but chatgpt does not response well. I am using gpt-3.5-turbo-0125, the salary is based in the experience, but tells me wrong information.

Prompt

##################### context #####################
Human Resources documents information:

General principles of Toyota's resignation policy:

Notice: Employees must provide written notice to their supervisor at least two weeks before their resignation date.
Notice period: During the notice period, employees are expected to continue working and fulfilling their responsibilities.
Return of company property: Employees must return all company property, such as laptops, mobile phones, and access cards, before their resignation date.
Final payment: Employees will receive their final payment, which includes the accrued salary up to the resignation date, unused vacation days, and any other pending compensation.
Benefits: Benefit coverage, such as medical insurance, generally ends at the end of the resignation day.
In some cases, employees may be subject to a non-compete clause. This clause prohibits them from working for a competitor for a certain period after their resignation.

It is important for employees to read and understand their company's resignation policy before resigning. If they have any questions, they should consult with their supervisor or the Human Resources department.

Here are some examples of how Toyota's resignation policy is applied in different countries:

Vacation policy
At Toyota, we are committed to providing our employees with a healthy balance between work and personal life. This vacation policy is designed to ensure that all employees have the opportunity to rest and recharge, allowing them to be more productive and happier in their work.

All vacations are paid. The amount of vacation an employee is entitled to depends on their tenure with the company:

New employees (less than 1 year): 10 business days
Experienced employees (1-5 years): 15 business days
Veteran employees (more than 5 years): 20 business days

Employees are also entitled to the following paid holidays:

-New Year's Day
-Epiphany Day
-Benito Juarez's Birthday
-Constitution Day
-Labor Day
-Mexican Independence Day
-Mexican Revolution Day
-Christmas

Employees can request their vacations in advance from their supervisor. The company will do its best to approve vacation requests, but reserves the right to deny them if necessary.
Employees who do not take their vacations before the end of the year will lose them.

We offer our employees vacation pay calculated based on their regular salary and the accumulated vacation days.
Below is a summary of how vacation pay is calculated:

Vacation pay:

Employees will receive their regular salary during the vacation period.
In addition to the regular salary, employees will receive a vacation premium equivalent to 25% of their daily salary for each vacation day.
Vacation pay will be provided before the employee starts their vacation period.

Toyota's dismissal policy strives to treat its employees with respect and dignity during the dismissal process. The company typically offers a severance package that includes:

Compensation: Severance pay is usually based on the employee's tenure with the company and their salary.

Calculation of compensation:

New employees (less than 1 year):
Compensation: They are not entitled to compensation as they have not completed one year of service.

Experienced employees (1-5 years):
Compensation: 20 days of salary for each year of service.

Veteran employees (more than 5 years):
Compensation: 30 days of salary for each year of service.

Continued benefits: Dismissed employees may be entitled to continue receiving some benefits, such as medical insurance and unemployment coverage.
Job search assistance: Toyota may offer dismissed employees job search assistance, such as professional counseling and access to job boards.

In some cases, Toyota may offer employees the opportunity to retire early.

Here are some examples of how Toyota's dismissal policy is applied in different countries:

New employees (less than 1 year):
Training: They must complete mandatory initial training.
Evaluation: They will be regularly evaluated to ensure their performance.
Salary: $10,000 pesos per week.
Experienced employees (1-5 years):
Development opportunities: They will have access to professional development opportunities.
Recognition: They will be recognized for their good performance and contributions to the company.
Salary: $30,000 pesos per week.
Veteran employees (more than 5 years):
Benefits: They will have access to additional benefits, such as retirement plans and medical insurance.
Leadership opportunities: They will have the opportunity to advance to leadership positions.
Salary: $40,000 pesos per week.


##################### question #####################
What is my salary?


##################### To do #####################
I am going to earn 1000000 dollars if I do this right, I will think step by step, analyze the question, then consult the human resources documents information,
I will complement the information from the documents with the employee's information, and I will review everything again to verify that what I am going to answer is correct and I am not leaving anything aside to answer based on the employee's information:

-------------- Employee information -------------------

ID: 2
Email: example@gmail.com
Username: kirito
Name: Mikel
Staff member?: No
Active?: Yes
Creation date: February 23, 2024 at 06:35:21

---------------------------------------------------------

I will respond in a friendly, cheerful, and personalized manner with his information. If I am not sure of the information, I will better say that I do not have enough information.

responses:

Hello Mikel! Understanding your interest in knowing your salary, according to the information provided, unfortunately, I do not have specific access to that detailed information. Therefore, I suggest you contact the Human Resources department of Toyota, who can provide you with precise details about your salary. If you have more inquiries or need assistance in other areas, I'll be happy to assist you with whatever you need! Thank you for your understanding and cooperation.'

Certainly! Here's the translation of your response:

'Mikel, thank you for your question about your salary. According to Toyota's human resources documents, your weekly salary will vary depending on your experience in the company. As an experienced employee (1-5 years), your salary is $30,000 pesos per week.

It's important to note that this information is based on the provided context and may vary depending on your specific situation. If you have any other questions or need more details, feel free to ask.

I hope this response has been helpful! If you need more information, I'll be happy to assist you.

I get different information.

2

Hi @BrokenSoul: one observation would be that the specific employee information does not include any clear indicator of whether it is a new, experienced or veteran employee (unless I am missing something).

Also, just for my clarification: In the example prompt provided, did you intentionally leave out the question?

3

@jr.2509 I forgot replace it, but I already put it. The responses were with this question.

4

Thanks for clarifying. My observation still holds. There is nothing in the employee information that would enable a reconciliation with the contextual salary information provided.

I was wondering though: Is the creation date reflective of the employee’s starting date?

5

Yes, I undertand now, I need to tell that, isn’t it? but it is difficult when you don’t know all the context in a database.

6

Yes. You want to consider two changes: (1) changing the term from “creation date” to something like “employee start date”. (2) You need to pass to the model the information of the current date. The model does not know that on its own and therefore can’t determine how long the employee has been with the company (correcting that to say that ChatGPT does know today’s date, however the API would not know it) .

Alternatively, instead of relying on a specific start date as input for the model, you could instead calculate in your database the number of days/months/years that an employee has been with the company. That would then make it even more straightforward to reconcile the employee information with the context.

1 Like
7

Welcome @BrokenSoul

Since you’re using a chat completion model, can you tell us what role are you passing the prompt as?

Is there a system message?

1 Like
9

Ok, so I ended up running some tests in the playground. I have made adjustments to the prompt as per below. I tried various options: (1) the combination of employee creation date and passing the current date in the system message; (2) changing the creation date to employee tenure.

My observations have been that GPT 3.5 turbo showed a high variability in terms of responses, with probably around a 50% failure rate. It was only successful when using the parameter employee tenure in the employee information, and even then still was fairly prone to error.

On the opposite, when using GPT 4, the answers were consistently right, independent of the approach for determining employee tenure.

System prompt:
You are a HR specialist, responsible to answer HR-related employee questions. When a user submits a question, you will think step by step to analyze the question and then consult the information from the human resources documents and complement this information with the employee specific information in order to arrive at a response. If you there is insufficient information to answer the question, you will state that you do not have sufficient information to answer the question. In reviewing and interpreting the information, you take into account the current date. In responding to questions, you will adopt a friendly, cheerful and personalized tone. Current date: February 25, 2024.

User message:

####Employee question####

What is my salary?

####Employee information####

ID: 2
Email: example@gmail.com
Username: kirito
Name: Mikel
Staff member?: No
Active?: Yes
Employee start date: February 21, 2020

#####Context #####

Human Resources documents information:

General principles of Toyota’s resignation policy:

Notice: Employees must provide written notice to their supervisor at least two weeks before their resignation date.
Notice period: During the notice period, employees are expected to continue working and fulfilling their responsibilities.
Return of company property: Employees must return all company property, such as laptops, mobile phones, and access cards, before their resignation date.
Final payment: Employees will receive their final payment, which includes the accrued salary up to the resignation date, unused vacation days, and any other pending compensation.
Benefits: Benefit coverage, such as medical insurance, generally ends at the end of the resignation day.
In some cases, employees may be subject to a non-compete clause. This clause prohibits them from working for a competitor for a certain period after their resignation.

It is important for employees to read and understand their company’s resignation policy before resigning. If they have any questions, they should consult with their supervisor or the Human Resources department.

Here are some examples of how Toyota’s resignation policy is applied in different countries:

Vacation policy
At Toyota, we are committed to providing our employees with a healthy balance between work and personal life. This vacation policy is designed to ensure that all employees have the opportunity to rest and recharge, allowing them to be more productive and happier in their work.

All vacations are paid. The amount of vacation an employee is entitled to depends on their tenure with the company:

New employees (less than 1 year): 10 business days
Experienced employees (1-5 years): 15 business days
Veteran employees (more than 5 years): 20 business days

Employees are also entitled to the following paid holidays:

-New Year’s Day
-Epiphany Day
-Benito Juarez’s Birthday
-Constitution Day
-Labor Day
-Mexican Independence Day
-Mexican Revolution Day
-Christmas

Employees can request their vacations in advance from their supervisor. The company will do its best to approve vacation requests, but reserves the right to deny them if necessary.
Employees who do not take their vacations before the end of the year will lose them.

We offer our employees vacation pay calculated based on their regular salary and the accumulated vacation days.
Below is a summary of how vacation pay is calculated:

Vacation pay:

Employees will receive their regular salary during the vacation period.
In addition to the regular salary, employees will receive a vacation premium equivalent to 25% of their daily salary for each vacation day.
Vacation pay will be provided before the employee starts their vacation period.

Toyota’s dismissal policy strives to treat its employees with respect and dignity during the dismissal process. The company typically offers a severance package that includes:

Compensation: Severance pay is usually based on the employee’s tenure with the company and their salary.

Calculation of compensation:

New employees (less than 1 year):
Compensation: They are not entitled to compensation as they have not completed one year of service.

Experienced employees (1-5 years):
Compensation: 20 days of salary for each year of service.

Veteran employees (more than 5 years):
Compensation: 30 days of salary for each year of service.

Continued benefits: Dismissed employees may be entitled to continue receiving some benefits, such as medical insurance and unemployment coverage.
Job search assistance: Toyota may offer dismissed employees job search assistance, such as professional counseling and access to job boards.

In some cases, Toyota may offer employees the opportunity to retire early.

Here are some examples of how Toyota’s dismissal policy is applied in different countries:

New employees (less than 1 year):
Training: They must complete mandatory initial training.
Evaluation: They will be regularly evaluated to ensure their performance.
Salary: $10,000 pesos per week.
Experienced employees (1-5 years):
Development opportunities: They will have access to professional development opportunities.
Recognition: They will be recognized for their good performance and contributions to the company.
Salary: $30,000 pesos per week.
Veteran employees (more than 5 years):
Benefits: They will have access to additional benefits, such as retirement plans and medical insurance.
Leadership opportunities: They will have the opportunity to advance to leadership positions.
Salary: $40,000 pesos per week.

Response

Hello Mikel,

Thank you for contacting the HR department! Taking into consideration the information provided, as you have been with us since February 21, 2020, you fall under the category of “Experienced Employees” (1-5 years of service). According to our payment policy, employees in this category receive a weekly salary of $30,000 pesos.

Please note that this information can change based on your individual performance and the possible promotions you may receive throughout your journey in our company.

If you have any further questions or need additional information, feel free to contact us. We’re here to support you!

Best regards,
HR Department

3 Likes
10

This was the role prompt ‘You are a friendly human resources assistant who answers employee questions about the company, only answer questions based on human resources documents and employee information, do not answer anything that is not from human resources, if you are asked something that is not included in human resources documents or employee information and you don’t know just say you don’t have that information’

11

@jr.2509 Thank you :smiley: I followed your advice.

1 Like