Ai assistant search tool response

API
1

i have been working on this problem for months. i am emitting a record from the search tool. it is in the message response but i have no reliable way of getting that content from the messages because attachments are supposed to be sent to the search tool, not from. i have tried every way i can think of with no luck. i am parsing the message and getting it out that way but any change in the instructions and the parsing criteria falls apart. is the any reliable way to send this record out of the search tool response and pick it up from a message ?emit:{
[
{
“ai_process_summary”: {
“record_id”: “generate_unique_id()”, # Generate unique record ID
“index_tag”: index_tag,
“category_id”: category_id,
“process_details”: {
“process_name”: “Process Analysis Test”,
“algorithms”: [
{
“algo_name”: algo[“algo_name”],
“description”: algo[“description”],
“inputs”: algo.get(“inputs”, ),
“outputs”: algo.get(“outputs”, )
}
for algo in selected_algos
]
}
}
}
]
}

i finally got the message out by doing an if on the record being emitted in json. even that has sketched on me a few times but it is working. that was a pain. using dash i got that and a usable check for display option that streamlit wouldnt let me do.

1 Like
2

What you are attempting to do is hard to understand, because it doesn’t follow any convention of where file search is useful.

File search is just for instilling knowledge, and the AI even has to know what to write a query about to get some top-ranked document chunks out-of-order.

Here’s a simplified step-by-step breakdown of the process that actually happens when uploading as user attachment to an assistants message for file search and attempting to use that:

File Upload and Processing:

  1. User Uploads File: A user uploads a document (e.g., PDF, TXT).
  2. Document Extraction: OpenAI’s system extracts the text content from the uploaded file.
  3. Chunking: The extracted text is divided into smaller pieces (chunks). Each chunk is roughly 800 tokens in size, with a 400-token overlap between chunks.
  4. Vector Embedding: The chunks of text are converted into numerical representations called “embeddings”. This is done using a model like text-embedding-3-large.
  5. Vector Store: The chunks and their embeddings are stored in a vector database (called a Vector Store), which is optimized for semantic and keyword search.

AI Interaction:

  1. AI Receives User Query: The AI assistant receives a question from a user.
  2. AI Queries Vector Store: If the AI deems it useful (somehow), AI uses the file_search tool to send a query in its own language and search the Vector Store to find the best matching chunks for answering the user’s question.
  3. Ranked Chunks Returned: The Vector Store returns the most relevant chunks in ranked order, based on their similarity to the AI’s query.
  4. AI Creates Response: The AI then uses these most relevant ranked chunks of information from the documents to formulate a response to the user.

You can see that placing instructions within some documents is very unreliable, and the AI instruction hierarchy actually demotes tool return text from taking over the AI with any injection attempts that may be within.

You likely want to:

  1. Craft a system prompt with the overall mission;
  2. provide the full text you are working with as a “documentation” section in a system or user message following that;
  3. by user input, instruct the exact generation (step) that you wish to receive;
  4. use the chat completions endpoint so you have ultimate control of the input.

The AI is not a code execution environment; it understands natural language.

If it is simply a JSON output that you wish to instruct the AI in, you can use “structured outputs” documentation, giving a response_format: json_schema, a specification of the output format that must be followed.