LangChain+LlamaIndex taking too long to give a answer

API
, ,
1

Hi! I’m using LangChain with LlamaIndex to create a assistant with custom knowledge. The assistant will consume the PDF docs in a folder to use as knowledge. I got it to work fine, but the bigger the document, the more it takes to give a answer to the user, even if the correct topic to answer can be found in the beggining of the doc. Here is a piece of my code:

def get_ai_response(user_data):
    os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")

    directory = user_data["discipline"]

    documents = SimpleDirectoryReader(f"data/{directory}", recursive=True).load_data()

    documents = [doc.to_langchain_format() for doc in documents]
    text_splitter = RecursiveCharacterTextSplitter()
    documents = text_splitter.split_documents(documents)

    embeddings = OpenAIEmbeddings()
    retriever = FAISS.from_documents(documents, embeddings).as_retriever(k=4)

    llm = ChatOpenAI(temperature=1, model_name="gpt-3.5-turbo-0125", max_tokens=2048)
[...]
    conversation = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=retriever,
        verbose=True,
        memory=memory,
        max_tokens_limit=1536,
        combine_docs_chain_kwargs={"prompt": qa_prompt},
        condense_question_prompt=CONDENSE_QUESTION_PROMPT_CUSTOM,
    )

Can someone help me? ><