# Getting GPT Vision To Return Coordinates **URL:** \ **Category:** Prompting\ **Tags:** gpt-4, gpt-4-vision\ **Created:** [March 7, 2024, 9:59pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669 "2024-03-07T21:59:59Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![dano1066](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/dano1066/32/305758_2.png) [@dano1066](https://community.openai.com/u/dano1066)\ **Post date:** [March 7, 2024, 9:59pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/1 "2024-03-07T21:59:59Z")
Vision has been doing a really good job at describing the images i provide but i want to try and also make it give me back the x/y coordinates of the specific things in the image so that i can use javascript to create a selection of the data the vision model found on the image. To test this i have an image that contains a list of items and on the left side of each item is a small icon. Ive tried using the following prompt and it is giving me back coordinates but they arent what I am looking for. “Each item on the list on the left contains a small icon on the left side. Give me the list of x,y coordinates of the top left and bottom right corner of each of these icons and return it as a json array” ---
**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\ **Post date:** [March 7, 2024, 10:04pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/2 "2024-03-07T22:04:16Z")
That is not something that gpt-4 vision can do, alone. > [@Estimation of measurements in images or PDFs](https://community.openai.com/t/estimation-of-measurements-in-images-or-pdfs/574214/3): > > Obtaining dimensions and bounding boxes from AI vision is a skill called grounding. You can, for example, see how Azure can augment gpt-4-vision with their own vision products. Other AI vision products like [MiniGPT-v2 - a Hugging Face Space by Vision-CAIR](https://huggingface.co/spaces/Vision-CAIR/MiniGPT-v2) can demonstrate grounding and identification. Such metrics are needed as a basis for measurement. gpt-4-vision alone might give you a description and be coaxed into extrapolation, but it is unlikely to be reliable. ---
**Author:** ![bulundum](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/bulundum/32/448499_2.png) [@bulundum](https://community.openai.com/u/bulundum)\ **Post date:** [June 6, 2024, 2:50pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/3 "2024-06-06T14:50:33Z")
I tried something similar, then I learned that the Vision API does not actually receive the original image. It gets a twice-resized image and messes the coordinates horribly. For more information on the resizing behavior, check out [https://platform.openai.com/docs/guides/vision](https://platform.openai.com/docs/guides/vision) title ‘calculating costs’. ---
**Author:** ![brendan.whiting](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/brendan.whiting/32/511502_2.png) [@brendan.whiting](https://community.openai.com/u/brendan.whiting)\ **Post date:** [June 6, 2024, 7:15pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/4 "2024-06-06T19:15:15Z")
You might try the Segment Anything model instead: [https://segment-anything.com/](https://segment-anything.com/) There’s a great course on it here: [https://learn.deeplearning.ai/courses/prompt-engineering-for-vision-models](https://learn.deeplearning.ai/courses/prompt-engineering-for-vision-models) ---
**Author:** ![rrmehdi](https://avatars.discourse-cdn.com/v4/letter/r/c67d28/32.png) [@rrmehdi](https://community.openai.com/u/rrmehdi)\ **Post date:** [June 13, 2024, 8:53pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/5 "2024-06-13T20:53:33Z")
Hi, I can return coordinate of a logo in png image, just with promps. ![AI_obtained](https://us1.discourse-cdn.com/openai1/original/4X/6/6/6/666bbf50dd8607cdc29320f328b32efd1b06be8b.png) This is done in chat session with gpt-4-vision-preview. The blue is the ground truth box, and blue is computed by AI, or other way around. The match is perfect. But, I don’t know how to convert that chat session trial into a reliable python script, because whatever I do , AI ChatGPT complain that it cannot analyze images with API yet. Any ideas how to do it? Perhaps, other AI frameworks (Azure OpenAI?) could be more flexible or advanced? My goal is to loop over hundreds of images and collect detected logo objects of many kinds (with coordinates) and their parent images into a dataframe. Any suggestions/recipes? ---
**Author:** ![aayush\_shah](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/aayush_shah/32/77050_2.png) [@aayush\_shah](https://community.openai.com/u/aayush_shah)\ **Post date:** [September 17, 2024, 8:17am UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/6 "2024-09-17T08:17:35Z")
A couple of good threads to make GPT-4o return co-ordinates! 🔥 1. [Reddit - Dive into anything](https://www.reddit.com/r/ChatGPT/comments/17b33c5/has_anyone_come_up_with_a_good_consistent_way_to/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button) (Main thread) 2. [Reddit - Dive into anything](https://www.reddit.com/r/ChatGPT/comments/17bdmst/comparing_gpt4vision_opensource_llava_for_bot/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button) (Sub thread - great discussion) ---
**Author:** ![Bai\_Lan\_Blues](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/bai_lan_blues/32/234782_2.png) [@Bai\_Lan\_Blues](https://community.openai.com/u/Bai_Lan_Blues)\ **Post date:** [November 1, 2024, 9:30am UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/7 "2024-11-01T09:30:27Z")
Has there been any improvement in this area? From what I find, it seems like it’s still not quite reliable. Though in my case I’m looking to identify the location of text instead of objects ---
**Author:** ![hiramreisneto](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/hiramreisneto/32/449693_2.png) [@hiramreisneto](https://community.openai.com/u/hiramreisneto)\ **Post date:** [November 14, 2024, 6:24pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/8 "2024-11-14T18:24:52Z")
I have found online a really clever solution for this kind of issue (using LLM to find coordinates on images) that I would like to share with you. The repository is not mine and I have not tested it, but the idea is there and from the examples presented it seems to work quite well. I am planning on testing it in a near future… ![1000209070](https://us1.discourse-cdn.com/openai1/original/4X/b/0/7/b07f2e0326988d6b093cd18cc1c1f54b403cb553.png) (Apparently I cannot post external links here, but you can find a repository called GridGPT on github, from a user called quinny1187…) In summary, the idea is just a way to give spatial context/awareness to the LLM, layering a numbered grid on the image sent to the model. ---
**Author:** ![honghoa2k2z](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/honghoa2k2z/32/547947_2.png) [@honghoa2k2z](https://community.openai.com/u/honghoa2k2z)\ **Post date:** [February 4, 2025, 1:50pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/9 "2025-02-04T13:50:00Z")
Hello, did you try to test it and how does it work ? I tested it and it doesn’t seem to work very well ---
**Author:** ![Saimon\_Saikia](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/saimon_saikia/32/512024_2.png) [@Saimon\_Saikia](https://community.openai.com/u/Saimon_Saikia)\ **Post date:** [May 19, 2025, 1:28am UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/10 "2025-05-19T01:28:51Z")
Well hello everyone. Yes the grid overlay technique actually does work, you can use 100×100 grids (1000 boxes instead of 100) if you want better accuracy…but you need to adjust some things (opacity, thickness, colour) programmatically.. With what I’ve observed Gemini models seem to work in providing bounding box coords with just raw image prompt.. (but I haven’t tested it well for UI elements). But the grid technique seems excellently well with GPT 4o and 4.1. You can test it yourself.. (if you want free access you can try GitHub Models). If you want faster results.. (eg. For realtime automated interaction use cases) you might find Gemini 2.0 Flash (live variant) as the best fit for this (with the grid overlay technique). Its quality is lesser than GPT (yet accurate 80-90% of the time), however it gives near realtime responses which might be a very critical factor in realtime automation systems. Try Gemini Realtime Stream (Google AI Studio) [\*If you specifically are looking for automating the web, then you might want to change the approach and use Gemini 2.5 Flash (Google AI Studio) for DOM analysis, and dynamic control of chromium via selenium/playwright (the anti bot prevention is your own part)] ---
**Author:** ![15yo\_Seal\_Opener](https://avatars.discourse-cdn.com/v4/letter/1/e274bd/32.png) [@15yo\_Seal\_Opener](https://community.openai.com/u/15yo_Seal_Opener)\ **Post date:** [July 30, 2025, 9:07am UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/11 "2025-07-30T09:07:56Z")
You can try gemini-2.5-flash to get bounding box/coordinates. > **[Bounding box detection  |  Generative AI on Vertex AI  | ...](https://cloud.google.com/vertex-ai/generative-ai/docs/bounding-box-detection)** I guess it’s the only model I know that can perform excellent object detection. Hope OpenAI bring support to their gpt-5 models ---
**Author:** ![Ana\_Wyatt](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/ana_wyatt/32/511791_2.png) [@Ana\_Wyatt](https://community.openai.com/u/Ana_Wyatt)\ **Post date:** [March 2, 2026, 9:23pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/12 "2026-03-02T21:23:14Z")
Eu testei o gemini-3-flash-preview e realmente ele foi muito bom na função de identificar blocos visuais e coordenadas praticamente perfeitas. A questão que me impediu de continuar com ele é muita restrição a depender do conteudo inputado (mesmo que tenhamos direitos autorais ele bloqueia muito e pelas pesquisas que fiz esse comportamento é de fato recorrente) Tive que abandonar a estrategia por conta desse bloqueio e estou tentando realizar um pipeline hibrido com Azure Document Iteligence e o GPT vision (4o). O GPT esta falhando muito no retorno de coordenada, entao estou tentando usar o mapeamento do DI como um guia para ele, mas ainda esta muito grosseiro os recortes. No meu caso busco as coordenadas para poder recortas as imagens ---
**Author:** ![OpenAI\_Support](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/openai_support/32/561845_2.png) [@OpenAI\_Support](https://community.openai.com/u/OpenAI_Support)\ **Post date:** [September 19, 2026, 6:33pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/13 "2026-09-19T18:33:03Z")
Thanks for sharing your findings, everyone, and sorry for the delayed response. As you may know, image analysis is supported through the API. See the current [Images and Vision guide](https://developers.openai.com/api/docs/guides/images-vision?api-mode=responses) for examples. For coordinate-based workflows, it's important to specify the coordinate system clearly - for example, pixel coordinates with (0,0) at the top-left - and account for any image resizing. The guide recommends detail: "original" for coordinate-sensitive tasks when supported by the model, though resizing limits can still apply. If you're still encountering issues, please open a new topic with the model, prompt, image dimensions, and an example showing the expected versus returned coordinates. Farah ---
**Author:** ![avinash.api](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/avinash.api/32/741581_2.png) [@avinash.api](https://community.openai.com/u/avinash.api)\ **Post date:** [September 19, 2026, 6:33pm UTC](https://community.openai.com/t/getting-gpt-vision-to-return-coordinates/671669/14 "2026-09-19T18:33:08Z")