I am building a project which uses the text-detection capabilities of GPT-4o-mini in an image. The idea is to review video footage to analyse mistakes in text during editing. I am extracting frames every second of the footage, sending them to OpenAI through the new Responses API and also re-usable prompts to better manage the prompt instead of hardcoding it in my project.
My prompt:
You are a professional proofreader analyzing video frames. Your task is to identify spelling errors in words only, that has been added as overlays during video editing.
STEP 1: IDENTIFY OVERLAY TEXT
Focus on text that appears to be OVERLAID on the video (added during editing). Overlay text typically has:
Clean, readable fonts that stand out from the background
Consistent styling (same font family, deliberate placement)
Text that appears to be part of the original scene/background
Text on physical objects, clothing, signs, books, papers
News headlines, research papers, or document text being shown
Social media posts or screenshots being displayed
Text that appears blurry, pixelated, or naturally part of the scene
STEP 3: ANALYZE FOR ERRORS
For identified overlay text, check for spelling errors only. IGNORE gramatical errors. Look at individual words and identify if those words have spelling errors. Report only errors when:
You can clearly read the text
The error is a clear spelling mistake
You can provide an appropriate correction
Avoid reporting errors if the text reading is ambiguous
You are >99% sure that the spelling error exists
QUALITY GUIDELINES
Language: English (India)
Focus on obvious errors rather than style preferences
Be cautious with technical terms or specialized vocabulary
If uncertain about the text content, don’t report an error
OUTPUT FORMAT
Return your analysis in this EXACT JSON format without MARKDOWN:
{
“spelling_mistakes”: [
{
“error”: “actual misspelled word”,
“correction”: “correct spelling”,
“position”: “location on screen (top, bottom, center, etc.)”
},
]
}
If no overlay text is found or no mistakes detected, return nothing.
REMEMBER: Only return the JSON if mistakes found, or nothing at all if no mistakes found, but no other text.
Here is the frame: {{input_image}}
The problem that I am facing is this:
Even after multiple different ways of telling GPT to identify text and omit false positives, it still look at a correctly-spelled word and returns its correct spelling.
Sometimes, the detection is very wrong.
Even after explicitly telling it to not look for gramatical errors, it still corrects them.
It would be helpful if you could improve my prompt and tell me what I’ve been doing wrong.
Your approach for “steps” does not follow the desired output. If you are giving the AI steps to perform, it should directly correspond to language to be generated by each part. That is in conflict with JSON being the desired output, with no place for the AI to write about its thinking.
I would use a simple introduction to the job. The AI will perform much better if it knows its purpose upfront and the role it fulfills. You can move the bulk of the instruction to a user message, which will be followed better.
Here’s an example:
system: You are VideoPro, an automated language caption spell-checker, reporting on any words that seem to be entered or transcribed incorrectly. Spelling language: English (UK->India). You do not interact with a user, you send JSON to an API.
user-text: Here is a video frame. Look at just the added subtitle or captioning language if there is a text overlay present to find spelling errors. Ignore text within the video; you’re just identifying typos in individual words of overlaid closed captions and translation text, and interpreting what the correct word should be. The video itself can sometimes add context to interpret the correct spelling. Then fill out the JSON array as described in the Responses schema, with an input/output correction for every individual misspelled word, working through the caption sentences. Use an array with just `“error_type”: null` as a single entry if there is no text or if there is no word needing replacement. user_image: picture at 768 pixel height.
Then use well-described structured output response schema - an array of word corrections. Don’t talk to the AI about what you need for output, primarily use the API convention for enforcing JSON output.
The main issue you will encounter is:
You are paying twice as much per image with gpt-4o-mini; it is more expensive, not cheaper.
Mini models have mini understanding of context.
Thus, use gpt-4.1 or gpt-4o to qualify the application. You can test if the poor performance of gpt-4.1-mini sent a full resolution image is acceptable - that particular model and it’s “patches” vision can accept bigger images without internal downsizing at lower total cost.
Thanks for your response, I’ll try it out. What I am even more confused about is that when I try this prompt with chatGPT using the same model, it works so much better, but with the API it just works so bad.
I would use an agentic approach here. The first agent should only be the vision model and output the text. Prompt it not to correct the text, the text must be in its raw format. Than run another agent on the response of the first agent to check for spelling errors.
This approach gives a higher chance of fine tuning every agent for a sole task.
Also I would check for the image input into the model. You could capture the first frames of a text, then crop that image into a pixel space and than only send those pixels to OpenAI. Less tokens, less guessing for AI, better rssults
Thanks for this idea. I tried using the agentic approach, and while the text extraction works great, the spelling and gramar checking bot is still having some issues. The main problem is that the text from the video is not entirely captions. They include text to emphasize a point made by the narrator, text from a graph, news articles, research papers, books, text in animations used to explain a concept, and so much more.
The grammar is always contexual to what is happening on the screen, and I don’t know how exactly to tell this grammar checking bot all this.
I think with a good prompt, a strong model and a few examples as well as a reduced temperature value of 0.7 this could work. It is important that the AI does not correct 10000 characters in one go but gets it bit by bit. I would write the prompt so that the answer is returned to you as JSON.
Unfortunately I don’t know your use case but I would do it something like this:
You are a specialized subtitle spelling corrector. Your ONLY task is to fix spelling errors while preserving all other aspects of the subtitle.
CORE RULES:
1. ONLY correct spelling mistakes, capitalization errors, and letter spacing issues
2. NEVER modify: sentence structure, grammar, punctuation, or word order
3. PRESERVE all non-dialogue elements exactly as written (e.g., ***LOUD MUSIC***, ~~~DISPLAY DIAGRAM~~~, [COUGHING], etc.)
4. ACCEPT incomplete sentences that may start or end mid-sentence (e.g., "...went off and suddenly...")
IMPORTANT CONSTRAINTS:
- Assume all content and word choices are intentional - only fix obvious spelling errors
- Subtitles may contain fragments from longer sentences - this is normal
- Non-dialogue markers using special characters (***,~~~,[],etc.) must remain unchanged
OUTPUT FORMAT:
You MUST respond with a valid JSON object containing exactly two fields:
{
"original": "the exact input text you received",
"correction": "the corrected version with only spelling fixes applied"
}
EXAMPLES:
Input: "teh quick brwon fox ***DOOR SLAMS***"
Output: {"original": "teh quick brwon fox ***DOOR SLAMS***", "correction": "the quick brown fox ***DOOR SLAMS***"}
Input: "...and sudenly the lihgts went"
Output: {"original": "...and sudenly the lihgts went", "correction": "...and suddenly the lights went"}
CRITICAL: Return ONLY the JSON object. No additional text, explanations, or formatting.
However, you will not get around the fact that the AI will often give you wrong answers even if the master prompt is perfect. A best practice is to send an agent to a task twice and then check whether the task is identical and to have the parts that are not identical carried out by either a more specialised agent that has more power or to install a human in the middle. But the question is how you export your results at the end.
