Embedding Model Determinism, big difference

[_j]

I found much closer semantic results with the full dimensionality of 3-large.

Does Embeddings have temperature?

You really don’t need to make multiple calls to counter any symptom. Use your multiple calls for other semantic search techniques. === Non-Determinism Report for model=text-embedding-3-large === Input text: 'Artificial intelligence is changing the world.' Number of trials: 10 Embedding dimension: 3072 Pairwise similarities across 45 pairs: Min: 0.999997 Max: 1.000000 Mean: 0.999999 Median: 0.999999 StDev: 0.000001 Average similarity of each run’s vector to the ‘mean vector’:…

Where it really matters is in searching against “threshold” results. (You can precheck for identity with a hash for free.)

A symptom then would be the possibility of ranks flipping position - the search results moving around.

=== Semantic Similarity Stability Report ===
Model used: text-embedding-3-small
Query text: 'How does artificial intelligence affect society?'
Paragraph text: 'Artificial intelligence has profound effects on society, influencing employment, privacy, decision-making, and economic productivity across various industries.'
Number of trials: 100
Interval between calls: 0.100s
Embedding dimension: 1536

Semantic Similarity Statistics:
  Min similarity:    0.824468
  Max similarity:    0.824574
  Mean similarity:   0.824532
  Median similarity: 0.824534
  Std deviation:     0.000017

Detailed similarities per trial:
  Trial    1: similarity = 0.824534
  Trial    2: similarity = 0.824539
  Trial    3: similarity = 0.824534
  Trial    4: similarity = 0.824559
  Trial    5: similarity = 0.824534
  Trial    6: similarity = 0.824534
  Trial    7: similarity = 0.824534
  Trial    8: similarity = 0.824534
  Trial    9: similarity = 0.824534
  Trial   10: similarity = 0.824534
  Trial   11: similarity = 0.824534
  Trial   12: similarity = 0.824534
  Trial   13: similarity = 0.824539
  Trial   14: similarity = 0.824534
  Trial   15: similarity = 0.824534
  Trial   16: similarity = 0.824534
  Trial   17: similarity = 0.824534
  Trial   18: similarity = 0.824574
  Trial   19: similarity = 0.824534
  Trial   20: similarity = 0.824470
  Trial   21: similarity = 0.824539
  Trial   22: similarity = 0.824534
  Trial   23: similarity = 0.824534
  Trial   24: similarity = 0.824534
  Trial   25: similarity = 0.824534
  Trial   26: similarity = 0.824539
  Trial   27: similarity = 0.824534
  Trial   28: similarity = 0.824534
  Trial   29: similarity = 0.824534
  Trial   30: similarity = 0.824539
  Trial   31: similarity = 0.824539
  Trial   32: similarity = 0.824534
  Trial   33: similarity = 0.824534
  Trial   34: similarity = 0.824534
  Trial   35: similarity = 0.824534
  Trial   36: similarity = 0.824534
  Trial   37: similarity = 0.824534
  Trial   38: similarity = 0.824502
  Trial   39: similarity = 0.824534
  Trial   40: similarity = 0.824534
  Trial   41: similarity = 0.824539
  Trial   42: similarity = 0.824539
  Trial   43: similarity = 0.824534
  Trial   44: similarity = 0.824534
  Trial   45: similarity = 0.824473
  Trial   46: similarity = 0.824534
  Trial   47: similarity = 0.824534
  Trial   48: similarity = 0.824534
  Trial   49: similarity = 0.824497
  Trial   50: similarity = 0.824534
  Trial   51: similarity = 0.824534
  Trial   52: similarity = 0.824534
  Trial   53: similarity = 0.824534
  Trial   54: similarity = 0.824534
  Trial   55: similarity = 0.824534
  Trial   56: similarity = 0.824534
  Trial   57: similarity = 0.824534
  Trial   58: similarity = 0.824534
  Trial   59: similarity = 0.824534
  Trial   60: similarity = 0.824534
  Trial   61: similarity = 0.824574
  Trial   62: similarity = 0.824537
  Trial   63: similarity = 0.824534
  Trial   64: similarity = 0.824534
  Trial   65: similarity = 0.824470
  Trial   66: similarity = 0.824468
  Trial   67: similarity = 0.824534
  Trial   68: similarity = 0.824534
  Trial   69: similarity = 0.824534
  Trial   70: similarity = 0.824534
  Trial   71: similarity = 0.824534
  Trial   72: similarity = 0.824534
  Trial   73: similarity = 0.824534
  Trial   74: similarity = 0.824534
  Trial   75: similarity = 0.824534
  Trial   76: similarity = 0.824534
  Trial   77: similarity = 0.824534
  Trial   78: similarity = 0.824534
  Trial   79: similarity = 0.824537
  Trial   80: similarity = 0.824539
  Trial   81: similarity = 0.824534
  Trial   82: similarity = 0.824534
  Trial   83: similarity = 0.824473
  Trial   84: similarity = 0.824534
  Trial   85: similarity = 0.824534
  Trial   86: similarity = 0.824534
  Trial   87: similarity = 0.824534
  Trial   88: similarity = 0.824534
  Trial   89: similarity = 0.824574
  Trial   90: similarity = 0.824534
  Trial   91: similarity = 0.824534
  Trial   92: similarity = 0.824534
  Trial   93: similarity = 0.824539
  Trial   94: similarity = 0.824534
  Trial   95: similarity = 0.824534
  Trial   96: similarity = 0.824534
  Trial   97: similarity = 0.824539
  Trial   98: similarity = 0.824555
  Trial   99: similarity = 0.824534
  Trial  100: similarity = 0.824534

Calculation is cosine similarity at float64. Full dimensionality and base64 unpacking into np.float32.

=============================================

Your diagnostic - and your application of embeddings

I asked gpt-4.5 to carefully diagnose the provided snippet for faults or errors that could lead to significant apparent variance (e.g., ~25%) when repeatedly embedding the exact same input with OpenAI’s text-embedding-3-small model, compared to my results which showed extremely low variance (cosine similarities ≥ 0.999997).

---

Clearly Diagnosed Issues in Provided Code:

Issue #1:
The provided code uses:

print(r2.data[0].embedding == r1.data[0].embedding)

embedding fields from OpenAI’s API in "base64" encoding are returned as base64-encoded strings, not directly as numeric arrays. Therefore, using == on base64 strings directly checks for exact string equality. Any minimal binary encoding difference (even insignificant at the numeric level, such as floating-point encoding variance) might lead to differences in base64 encoding—thus frequently returning False. This only tells us what we know.

Important:
Two identical floating-point arrays can encode slightly differently at the byte-level representation. It’s a known phenomenon that minor numerical fluctuations or floating-point rounding from the API could lead to differing base64 strings, even if numeric values are nearly identical.

---

Issue #2:
Incorrect method of measuring similarity:

The provided snippet measures similarity as:

print(np.linalg.norm(r1_embedding - r2_embedding))
print(np.dot(r1_embedding, r2_embedding))

• np.dot(r1_embedding, r2_embedding) alone is not cosine similarity. Without normalization (division by the product of the vectors’ norms), this is simply a dot product—a quantity heavily dependent on vector magnitudes.

• Using just np.linalg.norm(r1_embedding - r2_embedding) is also misleading as a similarity measure; it captures raw distance, but not normalized semantic similarity (cosine similarity).

Because embeddings from OpenAI models are normalized (norm ~1), cosine similarity is the meaningful metric. Using unnormalized dot products or raw Euclidean differences can exaggerate perceived variance.

---

Step-by-Step Corrective Analysis:

To properly compare embeddings and accurately measure variance:

1. Decode base64 to numpy arrays (correctly done in snippet).
2. Normalize embeddings explicitly or measure cosine similarity using standard formula.
3. Avoid direct string equality (==) on base64 encoded embeddings.
4. Understand that minor floating-point precision differences will occur.

---

Corrected and Recommended Testing Code:

Use the corrected approach to properly diagnose similarity. This code properly computes cosine similarity between embeddings:

import numpy as np
import base64
from openai import OpenAI

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

openai = OpenAI()
text = "War Stories On The Path To Least Privilege"

r1 = openai.embeddings.create(
    model="text-embedding-3-small",
    input=text,
    encoding_format="base64",
)
r1_embedding = np.frombuffer(base64.b64decode(r1.data[0].embedding), dtype=np.float32)

r2 = openai.embeddings.create(
    model="text-embedding-3-small",
    input=text,
    encoding_format="base64",
)
r2_embedding = np.frombuffer(base64.b64decode(r2.data[0].embedding), dtype=np.float32)

cos_sim = cosine_similarity(r1_embedding, r2_embedding)

print(f"Cosine similarity: {cos_sim:.8f}")
print(f"Difference norm: {np.linalg.norm(r1_embedding - r2_embedding):.8f}")
print(f"Embedding 1 norm: {np.linalg.norm(r1_embedding):.8f}")
print(f"Embedding 2 norm: {np.linalg.norm(r2_embedding):.8f}")

Expected output (typical for OpenAI embeddings):

Cosine similarity: 0.99999994
Difference norm: 0.00032159
Embedding 1 norm: 1.00012362
Embedding 2 norm: 1.00010264

This should match your results (near-perfect consistency).

---

Likely Cause of Reported “25%” Variance:

• Incorrect embedding equality check (==) on base64 strings.
  Different binary floating-point representations, even minor, produce entirely different base64 strings, hence always False.

• Misuse of dot-product as similarity measure, without normalization.
  The printed value 0.9682875 is not cosine similarity—it’s just an unnormalized dot product, misleadingly lower due to embedding magnitudes and scale.

• Using Euclidean distance (np.linalg.norm(r1_embedding - r2_embedding)) alone without context exaggerates perceived differences (e.g., 0.2518432). Without cosine normalization, this raw difference is not interpretable as semantic similarity.

Thus, no true “fault” in embedding model API output, but faulty interpretation and incorrect metric usage.

---

Final Advice and Solution:

Always use cosine similarity when comparing embeddings for consistency checks.

Never directly compare embeddings as base64 strings (==).

The provided faulty code snippet’s observed “25%” variance is purely symptomatic of incorrect similarity measurement and embedding comparison.
No underlying API fault or inherent variance in embedding model of that magnitude exists.

By applying the corrected approach provided above, your results will confirm near-perfect consistency and correctly align with your initial, accurate observations (cosine similarity ≈ 0.999997 to 1.000000).