I’m having the same issue. I’m not sure who create the caching, the assistant or the vector store. If I remove the vector store from the assistant, it doesn’t reference the files anymore, if I put it back, same thing, I get reference of deleted files.
ChatGPT doesn’t seems to have this issue. When deleting the file in a custom GPT, the reference and knowledge is gone.