At a high level I understand what you are saying, which is, you need high scores on semantic meaning and not word overlap. Got it. Then you say you can achieve this by a NN (two-tower). Got it. Then you say the fine-tuned embedding is the output of your NN. Got it. All of this is fine and good and doesn’t need a direct fine-tune of the original embedding engine, since you are creating them in the output of your NN. I think you answered your own question, which is yes, you can create a fine-tuned embedding, which is created by the output of you own neural net. Totally feasible and makes sense. But you can’t upload some training file to the OpenAI API for embedding-ada-002 and get the same thing. Which is what I thought your original post was about.
And FYI, you can improve the geometry of the embeddings too, I did this in this thread. Some questions about text-embedding-ada-002’s embedding - #42 by curt.kennedy
It removes the mean embedding vector and uses PCA to reduce the dimensions and increase the spread without altering the meaning too much.
So yeah, post-processing of the embeddings is certainly advised and encouraged in certain situations.