Accuracy of retrieved data (large CSV)

GPT builders
1

I’m testing a GPT that has a single 3MB (20K row) CSV file. It is a catalogue of historical objects.

I have written an instruction prompt that urges the GPT to be accurate when presenting the data, and not to change any text from “Description” field when answering questions.

However, ChatGPT will always change the Description. For example, I can ask “Show me the record for object ID123”. It analyses the CSV and presents a believable-looking record, outputting each field in a list. It can do this for any of the 20K records, and it is finding relevant records. The searching works. But…

When looking closely, it makes things up. Asking it about a portrait, it adds information to the Description that just isn’t there, such as the artist name (always a name taken from somewhere else in the data), which way they’re sitting, etc.

Real data from CSV: “A portrait of a man lit by candlelight dated 1780. Artist and subject unknown”

ChatGPT output: “A portrait of a man lit by candlelight dated 1780. He is sitting on a chair facing left. The signature has been identified as well-known local artist John Smith.”

The location of the object, recorded in the “Location” field such as “Room 1, Wall A” might appear as “Room 1, Wall A, hangs above the door”. It has just made up “hangs above the door”.

I have tried some crafty prompting in the GPT instructions. I have told it in many ways to only present the actual data. I’ve asked it to write Python code to only extract the values of the field without amendment, summarisation and all the synonyms you can think of to try and get it to give me the real unadulterated records. But it never works.

I tried splitting the CSV into 10 smaller files, and this had no effect whatsoever, even starting a new GPT. Trying with just one smaller file (2K rows) and the same errors and hallucinations occurred.

Can anyone recommend a different approach? Is the kind of output that I want even possible using a Custom GPT? All tips welcomed.

Thanks in advance.

2

Have you tried threatening it? I had an issue where it was adding information and then I added to the prompt “if you don’t return just “x” you have failed.” That helped clear things up for me.

1 Like
3

LLMs are pretty bad with tabular data unless you give very detailed information about each process while verifying the output. I struggled a lot in the past when figuring out a way to pull this off. The optimal solution is to follow a prompting method explained here, namely “chain-of-table”:
https://research.google/blog/chain-of-table-evolving-tables-in-the-reasoning-chain-for-table-understanding/

By restructuring my prompts with chain-of-table with multiple LLMs through an agentic pipeline, I got things properly working. Please do not underestimate the power of prompt engineering for foundational models.

4

@serknslmz It does seem that way. Annoyingly, I thought that the Advanced Data Analysis features of ChatGPT Plus would be able to handle this kind of simple find operation, but it seems not. Occasionally I have been able to get it to write and execute a Python script using the pandas data analysis library and it has worked, but then later, it stops being accurate again. It’s quite maddening!

I’ll take a look at the “chain-of-table” prompting. I’ve done the Vanderbilt Prompt Engineering course but this seems to be only published recently, so thanks for pointing it out.

I have tried it with a rival LLM’s free tier and it worked perfectly every time, but they don’t have the equivalent of a Custom GPT for Teams, or I’d switch.

@cdunn Yes I have threatened it. After several hours of frustration, and after telling it to try again more that 10 times (“I apologise for the error, I did alter the text in the Description. This was an error. Let me try again, this time making sure to extract the exact values from that field.”) I beat it with words, and it did it. But only that time. And that was from the general prompt, not the Custom GPT instruction prompt.

I’ve even made sure that the instructions I use are positive. Rather than “don’t change the values in the CSV” I way “ensure that you are as accurate as possible, and output the values exactly as they are in the CSV”.

Time to create a third GPT and try some new persuasive prompt engineering.

2 Likes
5

@tom36 Another alternative is to use it in JSON mode. Although not 100% consistent, it can increase the consistency.

First you transform your CSV to JSON, meaning that tabular data is presented in JSON format using objects, keys and arrays. Then you implement the logic behind the chain-of-table in your prompts. And finally use the LLM in JSON mode to boost the consistency.

JSON mode: https://platform.openai.com/docs/guides/text-generation/json-mode

6

Have you tried to use Assistants and functions? Those can give you a closer control of your data. Besides, you can attach your data directly to your assistant, or create a function that fetchs exactly the content you want