You MAY find Assistants SDK helpful which by the docs DOES support file_search and you can specify Vector Store ID to look into files - Does voice agents realtime api supports the vector stores or file from platform open ai? - #3 by tiblu
NOTE that Assistants API is deprecated and will go away late August 2026. Here is the migrations guide - https://platform.openai.com/docs/assistants/migration