Hi - I am facing a strange situation with a SQL reader application which I have built using gpt-4o-mini. This is a Langgraph based tool and below is the state which is being tracked within the application.
The following is the state in a gpt-4o-mini based LLM application.
{'messages': [HumanMessage(content='How many customers are there?', additional_kwargs={}, response_metadata={}, id='b507375d-edff-4dec-8e6e-f6f227e1153f'), HumanMessage(content='For the following plan:1.Retrieve the total count of customers from the customers table in the database., you are tasked with executing step 1, Retrieve the total count of customers from the customers table in the database.. Respond with the available answer to the replanner node once you have executed the Retrieve the total count of customers from the customers table in the database. you have been asked to execute', additional_kwargs={}, response_metadata={}, id='69a1da3e-0665-446a-a4cf-d2be481b3d64'), AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_dViRE2GmtvwkrbYuwMafh8Nl', 'function': {'arguments': '{"query":"SELECT COUNT(*) AS total_customers FROM customers;"}', 'name': 'sql_db_query'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 24, 'prompt_tokens': 2021, 'total_tokens': 2045, 'completion_tokens_details': {'reasoning_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_f85bea6784', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-b63cf647-9b74-48c9-8e29-d05029ce2f40-0', tool_calls=[{'name': 'sql_db_query', 'args': {'query': 'SELECT COUNT(*) AS total_customers FROM customers;'}, 'id': 'call_dViRE2GmtvwkrbYuwMafh8Nl', 'type': 'tool_call'}], usage_metadata={'input_tokens': 2021, 'output_tokens': 24, 'total_tokens': 2045}), ToolMessage(content='[(13,)]', name='sql_db_query', id='cb5a3101-64eb-4264-a56c-383da102a488', tool_call_id='call_dViRE2GmtvwkrbYuwMafh8Nl'), HumanMessage(content='For the following plan:1.Retrieve the total count of customers from the customers table in the database., you are tasked with executing step 1, Retrieve the total count of customers from the customers table in the database.. Respond with the available answer to the replanner node once you have executed the Retrieve the total count of customers from the customers table in the database. you have been asked to execute', additional_kwargs={}, response_metadata={}, id='0d2c1dd1-4d0d-46e6-9bf1-8d3fd215e0eb'), AIMessage(content='The total count of customers in the database is 3.', additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 13, 'prompt_tokens': 2141, 'total_tokens': 2154, 'completion_tokens_details': {'reasoning_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_f85bea6784', 'finish_reason': 'stop', 'logprobs': None}, id='run-89591011-1d9f-4c3c-97d3-72a6b8488538-0', usage_metadata={'input_tokens': 2141, 'output_tokens': 13, 'total_tokens': 2154})], 'plan': ['Retrieve the total count of customers from the customers table in the database.'], 'input': 'How many customers are there?', 'past_steps': [('Retrieve the total count of customers from the customers table in the database.', 'The total count of customers in the database is 3.')]}
The following is the state in a gpt-4o-mini based LLM application.
{‘messages’: [HumanMessage(content=‘How many customers are there?’, additional_kwargs={}, response_metadata={}, id=‘b507375d-edff-4dec-8e6e-f6f227e1153f’), HumanMessage(content=‘For the following plan:1.Retrieve the total count of customers from the customers table in the database., you are tasked with executing step 1, Retrieve the total count of customers from the customers table in the database… Respond with the available answer to the replanner node once you have executed the Retrieve the total count of customers from the customers table in the database. you have been asked to execute’, additional_kwargs={}, response_metadata={}, id=‘69a1da3e-0665-446a-a4cf-d2be481b3d64’), AIMessage(content=‘’, additional_kwargs={‘tool_calls’: [{‘id’: ‘call_dViRE2GmtvwkrbYuwMafh8Nl’, ‘function’: {‘arguments’: ‘{“query”:"SELECT COUNT() AS total_customers FROM customers;"}', ‘name’: ‘sql_db_query’}, ‘type’: ‘function’}], ‘refusal’: None}, response_metadata={‘token_usage’: {‘completion_tokens’: 24, ‘prompt_tokens’: 2021, ‘total_tokens’: 2045, ‘completion_tokens_details’: {‘reasoning_tokens’: 0}}, ‘model_name’: ‘gpt-4o-mini-2024-07-18’, ‘system_fingerprint’: ‘fp_f85bea6784’, ‘finish_reason’: ‘tool_calls’, ‘logprobs’: None}, id=‘run-b63cf647-9b74-48c9-8e29-d05029ce2f40-0’, tool_calls=[{‘name’: ‘sql_db_query’, ‘args’: {‘query’: 'SELECT COUNT() AS total_customers FROM customers;’}, ‘id’: ‘call_dViRE2GmtvwkrbYuwMafh8Nl’, ‘type’: ‘tool_call’}], usage_metadata={‘input_tokens’: 2021, ‘output_tokens’: 24, ‘total_tokens’: 2045}), ToolMessage(content=‘[(13,)]’, name=‘sql_db_query’, id=‘cb5a3101-64eb-4264-a56c-383da102a488’, tool_call_id=‘call_dViRE2GmtvwkrbYuwMafh8Nl’), HumanMessage(content=‘For the following plan:1.Retrieve the total count of customers from the customers table in the database., you are tasked with executing step 1, Retrieve the total count of customers from the customers table in the database… Respond with the available answer to the replanner node once you have executed the Retrieve the total count of customers from the customers table in the database. you have been asked to execute’, additional_kwargs={}, response_metadata={}, id=‘0d2c1dd1-4d0d-46e6-9bf1-8d3fd215e0eb’), AIMessage(content=‘The total count of customers in the database is 3.’, additional_kwargs={‘refusal’: None}, response_metadata={‘token_usage’: {‘completion_tokens’: 13, ‘prompt_tokens’: 2141, ‘total_tokens’: 2154, ‘completion_tokens_details’: {‘reasoning_tokens’: 0}}, ‘model_name’: ‘gpt-4o-mini-2024-07-18’, ‘system_fingerprint’: ‘fp_f85bea6784’, ‘finish_reason’: ‘stop’, ‘logprobs’: None}, id=‘run-89591011-1d9f-4c3c-97d3-72a6b8488538-0’, usage_metadata={‘input_tokens’: 2141, ‘output_tokens’: 13, ‘total_tokens’: 2154})], ‘plan’: [‘Retrieve the total count of customers from the customers table in the database.’], ‘input’: ‘How many customers are there?’, ‘past_steps’: [(‘Retrieve the total count of customers from the customers table in the database.’, ‘The total count of customers in the database is 3.’)]}
The tool clearly states that, the count is 13. Why does the model state that, the count is 3? Is there any special parsing to be done when using gpt-4o-mini. Please note that, gpt-4o does provide the correct answer(13)