I passed a 1GB of file using openai’s text-embedding-3-small model, im getting OOM error, these are the limits passed for each api call by using multi threading…
MAX_TOKENS_PER_DOC = 8192
MAX_DOCS_PER_BATCH = 2048
MAX_TOKENS_PER_BATCH = 300000
is there a way to achieve high through put, so we can embed the large data of files which are in GB’s in very less time? how speed up the embedding process?