Hi, I’m trying to use Langchain to create a vectorstore from scraped HTML pages, but I encountered an issue where I’m getting embeddings of length 1 when it should be 1536 per OpenAI Platform.
Here’s how my code looks:
from langchain.document_loaders import BSHTMLLoader
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores.faiss import FAISS
...
all_raw_documents = []
for file in html_files:
loader = BSHTMLLoader(file)
raw_documents = loader.load()
all_raw_documents.extend(raw_documents)
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)
documents = text_splitter.split_documents(all_raw_documents)
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)
On the last line, I’m getting the below error:
Traceback (most recent call last):
File "/ingest.py", line 128, in <module>
ingest_docs(customers)
File "/ingest.py", line 34, in ingest_docs
vectorstore = FAISS.from_documents(documents, embeddings)
File "/env/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 272, in from_documents
return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)
File "/env/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 385, in from_texts
return cls.__from(
File "/env/lib/python3.10/site-packages/langchain/vectorstores/faiss.py", line 348, in __from
index.add(np.array(embeddings, dtype=np.float32))
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (13238,) + inhomogeneous part.
After some investigation, I found that the problem is due to getting the following erroneous embeddings:
{
"embedding": [
NaN
],
"index": 520,
"object": "embedding"
}
As you can see, the embedding returned has length 1 instead of a real embedding with length 1536.
Does anyone know how to resolve this issue? Thanks!