Given that these models are just pattern matchers they’re actually excellent at reasoning over spreadsheets if you properly format the data as a large sequence of patterns for them to work with… I’ll give a quick example to help build on this intuition…
| Item | In Stock | Price |
| :---------------- | :------: | ----: |
| Python Hat | True | 23.99 |
| SQL Hat | True | 23.99 |
| Codecademy Tee | False | 19.99 |
| Codecademy Hoodie | False | 42.99 |
The model sees that table in 1D as this:
| Item | In Stock | Price || :---------------- | :------: | ----: || Python Hat | True | 23.99 || SQL Hat | True | 23.99 || Codecademy Tee | False | 19.99 || Codecademy Hoodie | False | 42.99 |
If we ask the model “what’s the price of the codecademy hoodie?” the model has to look at all the tokens below in green while ignoring all of the tokens in red that might confuse it.
The biggest issue is that the Price label/anchor is super far distance wise from the other tokens needed to answer the question. You can see how if you add rows and columns that distance just keeps getting farther and the reliability of the models ability to reason is a direct function of distance between the tokens needed to answer a question.
We can help the model by moving all of these tokens closer together:
Item: Python Hat
In Stock: True
Price: 23.99
Item: SQL Hat
In STock: True
Price: 23.99
Item: Codecademy Tee
In Stock: False
Price: 19.99
Item: Codecademy Hoodie
In Stock: False
Price: 42.99
Now all of the information needed to answer our query is within a few tokens of each other. We’ve dramatically increased the reliability with which we can reason over this data… But there’s more going on here then just moving all these tokens closer together…
The model flattens this sequence out to a 1D line and in the process it’s also mapping all of patterns it sees to higher dimensional concepts that intersect with its world knowledge. So with this reshaped table what the model sees concept wise is this:
That clustering of concepts is where the magic really starts to happen at…
note: the bounding boxes i drew are a simplification of the concepts and patterns the model actually sees. It also sees that the first 2 items are things you wear on your head and the second 2 items are things you’d wear to cover your chest. The last 2 items are related to a website where you can learn coding and all 4 items are apparel items programmers would wear. etc…
The important point is that everything is just a pattern to the model and the more you can shape the information you show the model as a series of repeating patterns the better the responses you’ll get from the model.