You could keep your full 13,778 token chunk in tact, but embed only pieces of it, and then return the full 13,778 token chunk that correlates to the smaller embedded piece.
This way your chunk is coherent and you can feed it into a large context model.
As stated, the AI may not fully digest the larger chunk, because the attention is diluted over this larger span of tokens, and some details may get glossed over. But at least you have a coherent chunk in your retrieval, and hopefully, over time, the attention mechanisms in the LLM’s will improve.
As for determinism, I’m not sure why you need this as long as the cosine similarities are within roundoff error. To make an embedding engine fully deterministic, would cause a massive latency hit, because all the arithmetic ops in the computer have to be done in a very specific synchronized sequence.
The reasoning here is that it is a little known fact that the distribution laws are not reliable with floating point numbers inside the computer, so in general:
A(B + C) \neq AB + AC
So this is relaxed to:
A(B + C) \approx AB + AC
Therefore it’s not deterministic.
This is why there will never be determinism, nor would I ever want or desire this, simply for the benefits of increased speed and lower latencies.
PS. These models also may be running at lower than 64 bits. They might be 16 or even 8 bits. This lower number of bits, in conjunction with parallelized non-synchronous arithmetic would also increase the observed non-determinism. Devoting smaller bits to the weights also improves latencies and reduces costs, but shows up as more non-determinism. So as long as this randomness isn’t drastically throwing off your rankings, it’s seen as a speed and cost benefit to you.