I have a request that provide file inputs and text. E.g. a base64 file is provided along with a request “please summarize the file”.
How can I build evals for this type of prompt that includes files and text?
I’ve tried setting the test data with base64 encoded file data to feed as input into a jsonl file. When calling the eval run, i declare input messages similar to how I would when making a request to responses.
const run = await openai.evals.runs.create("eval-...", {
name: "Categorization summary",
data_source: {
type: "responses",
model: "gpt-4.1",
input_messages: {
type: "template",
template: [
{ role: "developer", content: "prompt...." },
{
role: "user",
content: [
{
type: "input_file",
filename: "draconomicon.pdf",
file_data: "{{item.file_content}}"
},
{
type: "input_text",
text: "What is the first dragon in the book?",
},
],
},
],
},
source: { type: "file_id", id: "file-..." },
},
});