⬛ Splitting / Chunking Large input text for Summarisation (greater than 4096 tokens....)

API
14

Hi @jdc2106

Sorry, no. The use case you describe above is not suitable for embeddings.

Embeddings are useful for (from the OpenAI docs):

  • Search (where results are ranked by relevance to a query string)
  • Clustering (where text strings are grouped by similarity)
  • Recommendations (where items with related text strings are recommended)
  • Anomaly detection (where outliers with little relatedness are identified)
  • Diversity measurement (where similarity distributions are analyzed)
  • Classification (where text strings are classified by their most similar label)

Hope this helps.

See also:

OpenAI Docs: Embeddings

3 Likes