Inconsistent Results in Basic Task Automation: How Can ChatGPT Be Trusted for Complex Data Analysis?

ChatGPT
1

I’ve been using ChatGPT, along with the new ‘4o with canvas’ feature, to streamline many tasks in my marketing business, and while it has been a huge time-saver, I’ve encountered some frustrating inconsistencies that raise questions about its reliability for more complex tasks, such as deep-dive analytics and data manipulation.

For example, when I ask ChatGPT to convert a Word document to HTML—a seemingly simple and common task I use the system for—with detailed instructions (like not stripping URLs and limiting <h2> tags to a single instance), it often completely ignores these instructions.

Frequently, URLs are replaced with # placeholders, and I end up with multiple <h2> tags, despite explicitly stating not to do so. Even when I point out that URLs have been stripped and ask it to refer back to the original document, sometimes it corrects itself, but more often it reevaluates and rewrites the document, either returning the exact same information with URLs stripped out or fabricating URLs entirely.

The documents I’m working with are usually a page and a half, maybe about 150 to 200 words at the most, often much less. The URLs are definitely in the original Word document. To minimize any potential context-related issues, I also ensure that I start a new session after processing a few documents. The instructions are provided in a separate WordDoc, with each individual instruction on it’s own numbered line.

These issues are easy to spot in simple tasks, but if this level of inconsistency occurs during complex data analysis—where errors might be harder to notice—how can ChatGPT be trusted for such tasks? I’ve read articles about how it’s used for analytics, but I wonder how companies manage these kinds of errors. Are the custom built/API powered instances that much more accurate?

Given that ChatGPT sometimes fails to follow straightforward instructions like keeping URLs intact or limiting header usage, how are companies using it for complex tasks like data analysis, and what measures are in place to ensure accuracy and consistency?

I’d appreciate any insights from the community. Thanks!

2

Are you keeping the conversation short? Starting a new one for each request? Or continuing the conversation?

You find a sweet spot of control. Letting any LLM be the driver is a guaranteed bad, probably expensive time. That’s not to say that you can’t have an LLM do some crazy stuff, as long as the tooling is setup correctly.

Additionally, most data analysis is done using the Code Interpreter

Have you tried asking it to convert the markdown first? You can then use a simple, free, online tool/library to convert markdown → HTML. You could probably accomplish this with Code Interpreter. The GPT models are more inclined to write in markdown.

3

Thanks for the quick reply @anon10827405! The conversations are fairly short. I would usually just do three page conversions before starting over. That had its own pros and cons; I noticed after the second document it would do better at following the instructions, especially after I corrected it a few times. But then after that third or fourth conversion it would start doing its own thing, like rewriting some of the copy or creating its own links. (the shame of it is, it’s copy rewrite was often better than the original but I’m locked in to use the client’s copy)

I have not tried to go from Word to mark down. I’m hoping to avoid any extra steps, especially since it gets it right most of the time.

I guess more than trying to solve this immediate issue I’m just curious as to how companies are using it for more complex tasks when these fairly simple tasks/instructions are not followed.

When working with one word doc at a time, one can see when it doesn’t follow instructions. But when working with a even a few hundred rows of data there’s no way to notice issues like that with out looking at all the data.

So this was more a discussion post on the big picture more than fixing this issue (though I’ll definitely try your suggestions to see if I can improve the workflow.)

Thanks!