I’m assuming they were doing this to test the model. I have a whole document corpus full of needles that use to evaluate my systems performance
Right.
I take your points.
Nonetheless, I still think there is a challenge in terms of achieving more comprehensive responses to broader and more complex queries. Certainly you can break down a complex query into multiple components, run multiple queries over your vector DB and pull it together in a comprehensive response. However, vector DB query systems are not inherently designed for completeness. With a properly designed system, you can get decent answers but you don’t have assurance whether the answer really embodies all available information. For me personally, that is still a critical point that needs to be addressed in the context of deeper analysis. This is in part because I work in a domain where accuracy is critical.
I’m hopeful that sometime soon I can do an in-depth medium post or cringe “paper” describing how we’re tackling a bunch of these problems. You guys are dancing around a lot of the ideas but there’s way more to the puzzle. We’re just trying to work out what we’re comfortable open sourcing and what we want to keep secret sauce. Awarity is a for profit org so we’re trying to figure out how to make a profit.
We’re making really good progress on all these fronts and I have very clear line of sight for how to achieve large scale reasoning over document corpuses of any size. There is a solution out there that will work with todays models
Yes they did insert the answer. And yes I know they just did it as a test. What I’m saying is that what they’ve proven may be interesting, but it’s in no way useful, and what they proved was quite obvious without even doing the test in the first place. Of course injecting an answer directly into the middle of context, in some contrived scenario (where you have the answer already in hand, ahead of time) is going to improve results. But so what? How is anyone going to use this knowledge to get any actual improved performance?
One thing I’ll share is that Awareness is the reasoning engine I’m building and I’m currently working on the next generation of Awareness which should have a logical context window size of over 1b tokens. My ultimate goal is to complete eliminate this whole concept of context windows. I’m not there yet but I’m getting closer. And that’s 1b token without any attention issues
That sounds indeed very promising! Keep us in the loop as you progress.
I’m starting to believe you haven’t read either of the posts and for whatever reason are hyper fixated on a typical testing technique.
The test needle in the haystack was used to understand at which lengths parts of context is ignored.
The paper I linked which you haven’t mentioned once indicates that repeating the content inside of this threshold fixes the problem. T
There is no “answer”. I think this is where the problem lies. For some reason you think the solution is to inject the answer elsewhere. Obviously the answer isn’t known but the thresholds which the model tends to “skip” is.
Here’s a very simplified example with an array
[A, B, C, D, E, F] = Length of 6. Model tends to skip indices 1-3 based on empirical evidence. If the answer was inside of “D”, model will fail.
Change array to [A, B, B, C, C, D, D, E, F] = Problem fixed. Extra tokens but now the content isn’t lost.
I built the virtualization system used by the the ListView control in Windows. When I joined windows it could barely display 10,000 items in a single folder. When I was done it could support well over 1,000,000 items per folder. I’m using some of the same tricks I used for the ListView plus some new ones
In EXPERIMENT 1 the “needle” (i.e. answer) that was being injected at various locations was: “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.”
So once again: Yes there was a “answer”, and Yes they did insert it, in various places. And insults like “hyper fixated” have no place on this forum. There’s no reason to attack someone when you could instead, simply ask for a clarification.
I have clarified more than once with a pants on head simple example. I’m sorry but I cannot make it more simple.
You are missing the point. It’s frustrating to have someone try to argue a non-existent point because they can’t be bothered to fully read & understand the linked articles.
You have not bothered to read or engage in anything I have said and quite frankly sound like a broken record. But, one last time.
Yes, the needle in the haystack technique (to which you are referring to) requires an answer. It’s a technique to determine at which depths the model begins to overlook information. This part you understand.
The part you have completely missed and seemingly refuse to understand is that the solution (linked elsewhere) is to repeat the items at this depth. There is no answer. It’s just identifying the token size and then duplicating the content at the depths the model has been measured to fail at.
Not trying to get in the middle of this friendly debate but a reasonable approach to determining “attention disorder” of a model as I call it would be to systematically increase the document size passed into the model and then place a needle (known answer) at varying points within the document at various positions within the document and ideally the model should return that needle every time you ask for it. If it doesn’t the model has an attention problem at that point in the spectrum.
I’m assuming that’s what they did
I avoid the portion of the spectrum with attention problems all together because it’s important to me that I always find the needles I’m looking for so I evaluate model performance a little differently.
I have a hand crafted corpus of the scores, stats, and play-by-play of all 43 games from this past college football bowl season. I then have about 100 Q&A pairs (needles) that I know the answers to like “tell me the score of every game and the name of the player in each game with the most receiving yards” I can run these questions across any model and I’m currently working on a point system that I’m going to use to grade them by (I’ll post those scores soon.) I will tell you the GPT-4 is really good if you stay below 16k tokens. Usually when it misses it’s because I failed to show it the information it needed to answer the question. RAG isn’t perfect. You need to show the model about 30% more data than you think.
Maintaining attention to track information about 43 games is really hard and GPT-4, so far, is the only model that can do it. The other models lose interest after a while and you can typically only get 25 or so scores from them. Even GPT-4 looses interest on occasion.
Hey Steve, I’m just getting into RAG/Vector work and your replies here are interesting. Can you expand a bit more on what you mean by retrieving 30% more text than you think you need?
Shouldn’t you be deciding what to include in the prompt based on how highly it ranks on your vector retrieval of the underlying knowledge? I.e. are you saying you should include more of the lower-scoring documents? Or are you saying repeat some of the higher quality material to address the “forgetfulness” issue people are discussing here?
Semantic embeddings retrieval and injection also can perform well in such a trial - and doesn’t cost $1 a question…
You have to understand how cosine similarity works. In search we think in terms of precision vs recall. Precision is a measure of how accurate the search results match the users query where recall is a measure of how many good answers you show the user. There’s a balancing act here that all search engines play with. Google is really good at precision and ok at recall which means they may not show you all the answers that matched your query but you should get a good one on top.
Cosine similarity, by contrast, is really good at recall but it sucks at precision. That means that you’re probably going to get some great chunks but they’re almost certainly not going to be in the top position. It depends on the embedding model and I forget the exact stat but for even the best embedding models it’s something like 63% precision out of the box which is horrible. Some of the better vector DBs are using a technique called hybrid search. That’s where they use cosine similarity and then they rerank the chunks using TF-IDF. This can boost the precision to around 80% which is still relatively poor.
To work around that you just need to show the model more chunks. The answers probably in there somewhere.
Search for my post on Document Sections (I’m on my phone) if you want to see some other techniques for improving the chunks you show the model.
Found it, thank you.
I work around the issue by ignoring the order of the chunks all together. I use the scores to build a heat map of the document that tells me where the needle most likely is. This works really well but I even miss some times so it’s not perfect
This applies to semantic retrieval. You want the model to validate the returned content and ensure it’s enough to answer the question
The problem with Document Sections is that I start at the center of the heat map then expand out in both directions and sometimes the answer is just below or above the text I render to the model. That’s because Document Sections doesn’t understand the structure of the underlying document.
I have a new library called DSTRUCTure (that I am planning to open source) which should solve that.