The issue is LLM’s map everything to a 1 dimensional space. We see the table in 2 dimensions (rows and columns) but the model sees it as a 1 dimensional line. I’m actually planning to write a blog post that digs into how to better reason over table data.
If you can share an example table and what you’re looking for I might be able to help. If nothing else I can at least point out what the model is struggling with.