Hi there everyone.
I’m a newbie in working with llms. I would like to make an evaluation of the model given as input an image and a prompt and then getting as output just a word. In particular i would like to string-check that word so that it should match a “correct_label”.
I’ve tried a few options and the best i got launching the eval was a server error (500).
I’ve tried contacting help.openai.com and they said it’s technically possible but now I’m really not sure. I know there is already a topic where it says it’s not possible with “gpt_4o_mini” but i don’t really care about what model i need to use, I’m really open on that. Can you help me?
Attached I’ll leave my code as well as the jsonl file i used.
url = "https://api.openai.com/v1/evals"
headers = {
"Authorization": f"Bearer {openai.api_key}",
"Content-Type": "application/json"
}
data = {
"name": "Image + Correct label",
"data_source_config": {
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"input": { "type": "image_url" },
"correct_label": { "type": "string" } # the expected one-word label
},
"required": ["input", "correct_label"]
},
"include_sample_schema": True
},
"testing_criteria": [
{
"type": "string_check",
"name": "Match output to label",
"input": "{{ sample.output_text }}",
"operation": "eq",
"reference": "{{ item.correct_label }}"
}
]
}
{"item": {"input":{"url": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRvqPTmHSPQNv6z3FGXMV6aexJWQ4EjwkQGWQ&s"}, "correct_label": "Dog"}}
{"item": {"input":{"url": "https://i.natgeofe.com/n/548467d8-c5f1-4551-9f58-6817a8d2c45e/NationalGeographic_2572187_16x9.jpg?w=1200"}, "correct_label": "Cat"}}
{"item": {"input":{"url": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSon1zCtSmEcmvdpCKR3gV41mS5frDNLqvX2w&s"}, "correct_label": "Car"}}
{"item": {"input":{"url": "https://media.istockphoto.com/id/1364917563/it/foto/uomo-daffari-sorridente-con-le-braccia-incrociate-su-sfondo-bianco.jpg?s=612x612&w=0&k=20&c=fxEx8bGP-UfpVRwdmX_mxQIs2E0ojhxw1bxHcB_ltzs="}, "correct_label": "Man"}}
eval_id = "correct eval id as defined before"
file_id = "correct json file id as defined before"
url = f"https://api.openai.com/v1/evals/{eval_id}/runs"
headers = {
"Authorization": f"Bearer {openai.api_key}",
"Content-Type": "application/json"
}
data = {
"name": "Image + Label Test Run ",
"data_source": {
"type": "completions",
"model": ENGINE,
"input_messages": {
"type": "template",
"template": [
{
"role": "developer",
"content": (
"You are an expert on recognizing what an image is about. "
"Respond with just one word representing the main item in the image. "
"Don't use commas or dots."
)
},
{
"role": "user",
"content": "{{item.input}}"
}
]
},
"source": {
"type": "file_id",
"id": file_id
}
}
}
resp = requests.post(url, headers=headers, data=json.dumps(data))
print(resp.status_code)
print(resp.json())
ERROR:
500
{‘error’: {‘message’: ‘The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error. (Please include the request ID req_f22ff81123b920e8ba4de860140cd437 in your email.)’, ‘type’: ‘server_error’, ‘param’: None, ‘code’: None}}
It would be perfect if someone corrected my code in some way. It’s for a school project and I’m really getting frustrated hahah
Thank you guys!