# RAG is not really a solution **URL:** \ **Category:** Community\ **Tags:** api, rag\ **Created:** [January 24, 2024, 3:25am UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291 "2024-01-24T03:25:33Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![joyasree78](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/joyasree78/32/35263_2.png) [@joyasree78](https://community.openai.com/u/joyasree78)\ **Post date:** [January 24, 2024, 3:25am UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/1 "2024-01-24T03:25:33Z")
After spending more than a year now with Gen AI, I feel RAG is more of a problem than a solution. it is so brittle and there is no science to it. I wanted to check with this group if anyone is aware of any other technique or ways to work more efficiently in this space. One of the things, I was expecting to see in this space is similar to how automated hyper parameter tuning happens in traditional ML. The idea is, if I provide a set of retrieval techniques and a loss function, is there any product/process that will run the different retrieval techniques, calculate the loss and automate the identification of the best technique. I could not find any such product so far. ---
**Author:** ![PaulBellow](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/paulbellow/32/597962_2.png) [@PaulBellow](https://community.openai.com/u/PaulBellow)\ **Post date:** [January 24, 2024, 3:34am UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/2 "2024-01-24T03:34:14Z")
@stevenic has done quite a bit, but he’s busy with a startup at the moment, I believe. If you search the forum, we’ve got a lot of great hidden gems / wisdom. Hrm… I wonder if someone would want to create a massive “here’s all the best RAG posts” for the forum… ---
**Author:** ![SomebodySysop](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/somebodysysop/32/17613_2.png) [@SomebodySysop](https://community.openai.com/u/SomebodySysop)\ **Post date:** [January 24, 2024, 5:08am UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/3 "2024-01-24T05:08:09Z")
> [@joyasree78](#): > > After spending more than a year now with Gen AI, I feel RAG is more of a problem than a solution. it is so brittle and there is no science to it. After a year of working on my RAG system, I could not disagree more heartily. Here are a couple of information sources I have found useful: - [https://www.youtube.com/live/Y9qn4XGH1TI?si=iUs\_x3yDL8BK7aUb](https://www.youtube.com/live/Y9qn4XGH1TI?si=iUs_x3yDL8BK7aUb) - [[2312.10997] Retrieval-Augmented Generation for Large Language Models: A Survey](https://arxiv.org/abs/2312.10997) > [@joyasree78](#): > > One of the things, I was expecting to see in this space is similar to how automated hyper parameter tuning happens in traditional ML. The idea is, if I provide a set of retrieval techniques and a loss function, is there any product/process that will run the different retrieval techniques, calculate the loss and automate the identification of the best technique. No idea what you’re talking about. But if that is your use requirement, I can certainly see your concern. I just want to make the point that I have found RAG to be extremely useful in my use case: Creating keyword and semantic searchable knowledgebases consisting of thousands of documents and focused on very specific areas of interest. Biggest problem I’ve ran into so far: Some query responses not comprehensive enough. End-users can almost always get to a complete answer using chain-of-thought queries (few-shot). But, the end-users I’ve been working with want complete answers on the first question (zero-shot). This may touch on your issue. My resolution: Deep Dive. Have the model dig through all the possible responses, categorize and analyze those, then return a complete list of best responses. Since I built my RAG system myself, I also have to build this feature. So I’m thinking, whatever you say this technique is you’re missing, you may have to build it yourself. > [@joyasree78](#): > > it is so brittle and there is no science to it. I have felt this way. A lot. But the truth is, so much of it is trial and error and fine-tuning the techniques that work best for your use case. And perhaps, in the end, that _is_ the science of it all. ---
**Author:** ![stanislav.nevyazhski](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/stanislav.nevyazhski/32/208997_2.png) [@stanislav.nevyazhski](https://community.openai.com/u/stanislav.nevyazhski)\ **Post date:** [January 24, 2024, 7:57am UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/4 "2024-01-24T07:57:09Z")
I find RAG extremely useful for giving the model more context. My use case is using LLM as a classifier on the text data, that not just uses the ML techniques, but “know” something about what these texts are about. E.g.I have a dataset with texts and associated numbers (encoded expense accounts for example). So, I receive the new text and I need to find the number for him. Classical classifiers showed accuracy 0.6-0.7. RAG been supplied with 5-10 most similar texts (based on embeddings+cosine similarity) with numbers, so the prompt is formed dynamically every time, and returns the numbers for new lines with accuracy 0.8-0.9. I assume on top of classical classifier “reads” the texts and that gives him more understanding of what he’s returning vs. classical model. So, my “technique” here is to tune the prompt every time i send it and monitor the accuracy afterwards. ---
**Author:** ![joyasree78](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/joyasree78/32/35263_2.png) [@joyasree78](https://community.openai.com/u/joyasree78)\ **Post date:** [January 24, 2024, 2:39pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/5 "2024-01-24T14:39:50Z")
Thanks a lot, are you using ada embedding. I found issues with embedding also where it is not returning the same embedding every time. I converted to encoding\_mod =float, that also is not 100% consistent. I wanted to try with cohere and see if that is better than ada ---
**Author:** ![SomebodySysop](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/somebodysysop/32/17613_2.png) [@SomebodySysop](https://community.openai.com/u/SomebodySysop)\ **Post date:** [January 24, 2024, 7:05pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/6 "2024-01-24T19:05:01Z")
> [@joyasree78](#): > > are you using ada embedding. I found issues with embedding also where it is not returning the same embedding every time. Yes. But I use it via the Weaviate tex2vec-openai transformer. In my experience, I have received fairly consistent cosine similarity results. `text-embedding-ada-002` ---
**Author:** ![stevenic](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@stevenic](https://community.openai.com/u/stevenic)\ **Post date:** [January 31, 2024, 4:24pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/7 "2024-01-31T16:24:32Z")
Thanks @PaulBellow… I will say that RAG is indeed useful and in fact it’s the key to grounding the model and giving it memory. I’m assuming your task @joyasree78 is Q&A related and what you have to realize is that task is simply a compression problem. The model will actually do a great job of answering almost any question but it needs to see the text with the answer in its context window. If you’re getting poor answers you’re likely not showing the model text that contains the answer. I could go on and on about the flaws with current RAG techniques (I’m building a company to address them) but what I’d suggest is to look at the queries where the model came back with the wrong with a bad answer? Was the answer in the prompt text? The model sometimes misses things. It’s rare but they’re not perfect. More often than not you’ll find that you’re simply not showing the model the text that contains the answer and this is when has to guess (hallucinate). The model always wants to answer you. That’s both it’s strength and it’s weakness because it’s a general purpose model that’s trying to cover a wide range of scenarios. ---
**Author:** ![bruce.dambrosio](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/bruce.dambrosio/32/224818_2.png) [@bruce.dambrosio](https://community.openai.com/u/bruce.dambrosio)\ **Post date:** [March 6, 2024, 6:26pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/8 "2024-03-06T18:26:34Z")
Just saw this, so very late reply. IMHO ‘traditional’ RAG is great for answering short questions when the answer is buried in a single chunk. Where it fails is in providing an ‘integrated view’ of a topic or field from a collection of sources. I believe next-gen Augmented LLMs will build more sophisticated background stores, not relying solely on embedding vectors of original document text segments. My current architecture uses a lattice of document/document-section clusters, with relevant abstractions of information at each node, down to actual individual document-sections. Anyone know of others working on things like this? ---
**Author:** ![jdshutt](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/jdshutt/32/265689_2.png) [@jdshutt](https://community.openai.com/u/jdshutt)\ **Post date:** [March 6, 2024, 7:04pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/9 "2024-03-06T19:04:35Z")
Not quite the same thing, but I’m using my [scaffolding](https://github.com/pemulis/sensei) to create networks of AI guides that call each other as functions, and dividing up large documents so that each guide has only one piece of the context, plus instructions on which guides to call for more information. Early tests focused on extremely simple examples—specialized guides have the “secret word” or “secret number,” and the root guide knows about them and can call on them. For the rest of this week, I’ll be testing with a relatively short book, broken into chapters, with each “branch” guide having access to just one chapter and the root guide having access to the table of contents, index, and introduction, and a list of keywords as part of the description for each chapter. I’ll whip up a specialist guide that is focused on generating keywords automatically. The theory here is that any one guide would get overwhelmed by too much context, so having a network of guides that each have limited context, connected by a root guide that knows how to call for support, will outperform a single guide that has the full context, even if it should theoretically fit within its context window. ---
**Author:** ![vidyasagar.mundroy](https://avatars.discourse-cdn.com/v4/letter/v/839c29/32.png) [@vidyasagar.mundroy](https://community.openai.com/u/vidyasagar.mundroy)\ **Post date:** [March 27, 2024, 3:42pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/10 "2024-03-27T15:42:21Z")
Here is my take on it. The intention behind RAG is excellent - Use LLM to answer questions on the external data it was not trained on. In fact, I have implemented a quick toy RAG based chatbot using LlamaIndex, fed four PDF documents that were definitely used for training the LLM (gemini-pro) and the chatbot answered five out of seven questions correctly. It couldn’t answer one question and gave a confident wrong answer (hallucination) to another. When I increased chunk size, it replied correctly to the first question the chatbot didn’t answer earlier. It answered even the second question when the question was slightly rephrased. When it was modified further, it gave wrong answer. In my use case of Q & A, the chatbot was unreliable. I prefer chatbot saying cannot answer instead of giving wrong answer. I don’t think, I want to put such unreliable chatbot in production. Though the RAG concept and intention behind it is quite useful, it is a problem if we can’t make products built on it hundred percent reliable. ---
**Author:** ![merefield](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/merefield/32/14812_2.png) [@merefield](https://community.openai.com/u/merefield)\ **Post date:** [March 27, 2024, 4:33pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/11 "2024-03-27T16:33:00Z")
> [@vidyasagar.mundroy](#): > > if we can’t make products built on it hundred percent reliable. LLMs don’t work deterministically though, so this is likely never going to happen. What I suspect will happen instead, is users will get used to this behaviour and even expect it so they will learn to continue to query. ---
**Author:** ![anon22939549](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@anon22939549](https://community.openai.com/u/anon22939549)\ **Post date:** [March 27, 2024, 4:38pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/12 "2024-03-27T16:38:59Z")
> [@vidyasagar.mundroy](#): > > if we can’t make products built on it hundred percent reliable. That’s an incredibly unrealistic standard, regardless of the tools you’re working with. I doubt you, yourself, are even 100% reliable, why would you expect a piece of software to be. ---
**Author:** ![curt.kennedy](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/curt.kennedy/32/709249_2.png) [@curt.kennedy](https://community.openai.com/u/curt.kennedy)\ **Post date:** [March 27, 2024, 7:21pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/13 "2024-03-27T19:21:02Z")
> [@vidyasagar.mundroy](#): > > It couldn’t answer one question and gave a confident wrong answer (hallucination) to another. To see what’s going on, you’d have to examine the prompt that was retrieved/formed. There are a lot more details you need to get right. First, I would not advise PDF’s as your main data source. These are notorious for being parsed incorrectly. If you have no choice, then you need to pour over the chunks being indexed, to ensure they make sense. Second, did you give the model an out? Like “Your answer will be ‘I don’t know’ if the context provided does not answer the question”. Third, you’ve already shown chunk size matters. It could be that some of the key chunks are incorrectly fragmented, and you need to do more than rely on a simple chunking algorithm to work for you. Chunks should form at least one “complete thought”. So start with examining your prompt. Does it look reasonable? Things getting chopped? Is there an out? Then work backwards. This may require getting your hands dirty, and ultimately not using LlamaIndex, and coding this directly yourself if LlamaIndex doesn’t have the right tuning knobs. ---
**Author:** ![SomebodySysop](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/somebodysysop/32/17613_2.png) [@SomebodySysop](https://community.openai.com/u/SomebodySysop)\ **Post date:** [March 28, 2024, 12:12am UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/14 "2024-03-28T00:12:51Z")
> [@vidyasagar.mundroy](#): > > In fact, I have implemented a quick toy RAG based chatbot using LlamaIndex, fed four PDF documents that were definitely used for training the LLM (gemini-pro) and the chatbot answered five out of seven questions correctly. I have been working with fairly traditional RAG implementations for several months now. There has rarely been a case where a question that was _correctly_ formatted was not answered, and never a case of hallucination. But, this doesn’t happen by magic – a lot of work has gone into developing a RAG system that generates consistently good results. First off, do everything @curt.kennedy suggests. You can’t just chop up text any old kind of way and then expect intelligent answers. I personally use my own version of Semantic Chunking [https://www.youtube.com/watch?v=w\_veb816Asg&t=1s&pp=ygUOc2VtYW50aWMgY2h1bms%3D](https://www.youtube.com/watch?v=w_veb816Asg&t=1s&pp=ygUOc2VtYW50aWMgY2h1bms%3D) But, if you do a YouTube search for “Semantic Chunking”, you will find a lot of different tutorials on how to chunk your data in ways that will make it far more useful in your queries. Also, when you say it answered 5 out of 7 questions, what were the two questions it did NOT answer? And, what makes you say they weren’t answered or that they _could_ be answered in the given text? You have to look at both the EXACT phraseology of the question asked, and the exact text you expected to answer it. You should probably brush up on your Prompt Engineering 101, but I’ll bet you that either your question wasn’t specific enough (based upon the actual text present), or your chunking split the idea of the text in a way that it did not match the cosine similarity search in the way you felt it should. Again, your chunking method. Finally, was the question specific or general? RAG systems, by definition, are excellent at needle in haystack types of queries – typically who, what, when, where? They begin to suck when it comes to sweeping questions and those which involve how, and to a greater extent, why? I have found that a lot of people think, because these machines seem to be so human like in their responses, that they actually “think” about the data they are pontificating on. They don’t. They are simply regurgitating text. They aren’t human, so they don’t posses human intuition – they can’t reason. So when you ask it a vague, sweeping question that is not specifically answered in one or more chunks that are returned to it, it’s going to either hallucinate, or tell you it can’t answer the question (if you have written your system prompt correctly). All this to say that RAG, even simple RAG, is far more than dumping PDFs into a vector store and asking questions with little thought behind how the model is actually capable of responding. ---
**Author:** ![anythxwdo](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@anythxwdo](https://community.openai.com/u/anythxwdo)\ **Post date:** [April 8, 2024, 3:11pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/15 "2024-04-08T15:11:28Z")
Though my experiment with RAG is still limited, I would agree the pre-processing step or “data chunking” is very important. And it seems time consuming to verify each chunk is semantically sound. It would be nice if there’s a proven algorithm that is able to scan a directory and to ensure each chunk/file is a single semantic unit, if not make corrections… maybe it’s too much to ask… ---
**Author:** ![SomebodySysop](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/somebodysysop/32/17613_2.png) [@SomebodySysop](https://community.openai.com/u/SomebodySysop)\ **Post date:** [April 8, 2024, 7:00pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/16 "2024-04-08T19:00:18Z")
> [@anythxwdo](#): > > It would be nice if there’s a proven algorithm that is able to scan a directory and to ensure each chunk/file is a single semantic unit, Theoretically, you could use a model to do this. Right now, I take my semantic chunks and simply chunk them further if they exceed my chunk limit. However, another approach would be to use a model to examine the chunk and have it determine where the cuts should be made. This, however, would be far more time-consuming and expensive – but would also result is semantically perfect chunks. ---
**Author:** ![joyasree78](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/joyasree78/32/35263_2.png) [@joyasree78](https://community.openai.com/u/joyasree78)\ **Post date:** [April 8, 2024, 8:00pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/17 "2024-04-08T20:00:06Z")
This is there to some extent with semantic chunking which is done using embeddings. ---
**Author:** ![anythxwdo](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@anythxwdo](https://community.openai.com/u/anythxwdo)\ **Post date:** [April 8, 2024, 8:19pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/18 "2024-04-08T20:19:46Z")
One may have multiple PDFs or Word documents or many of them. Relevant libraries can easily extract these documents into plain text files, however, they break off at each page and occasionally a whole word into half. But semantically each page does not necessarily form one semantic unit. Hence, some additional data pre-processing work would be necessary. And if we have 1000 or more such text files, going over each of them manually would be extremely time-consuming. That’s the rationale for my initial query for an automated and reliable algorithm to address the issue. ---
**Author:** ![SomebodySysop](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/somebodysysop/32/17613_2.png) [@SomebodySysop](https://community.openai.com/u/SomebodySysop)\ **Post date:** [April 8, 2024, 10:15pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/19 "2024-04-08T22:15:40Z")
> [@anythxwdo](#): > > And if we have 1000 or more such text files, going over each of them manually would be extremely time-consuming. That’s the rationale for my initial query for an automated and reliable algorithm to address the issue. Right. So, what do you think of my idea of having an LLM do this? ---
**Author:** ![anythxwdo](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@anythxwdo](https://community.openai.com/u/anythxwdo)\ **Post date:** [April 8, 2024, 11:18pm UTC](https://community.openai.com/t/rag-is-not-really-a-solution/599291/20 "2024-04-08T23:18:02Z")
A special purpose tiny language model could be a solution. Separate from data pre-processing, there’s an interesting technique to increase semantic similarity match, willing to share with you and a few others via pm if interested (and probably you’ve already experimented with it as well because it’s a logical progression). [edit] and it would be more efficient. [Next page](https://community.openai.com/t/rag-is-not-really-a-solution/599291.md?page=2)