Using GPT Assistant with image and Wolfram

Using GPT Assistant with image and Wolfram

We want to build the best maths app for students.
Mathematical calculations are unfortunately quite bad with GPT, so we would like to have them solved by Wolfram.

Our idea.
The student takes a picture of his maths homework.
We use GPT4 to recognise the topic
We use GPT-Vision to extract the tasks.
If we have more information on the topic in our training data, then we use this to solve the task
If they are arithmetic maths problems, then send them to Wolfram. We display the result.

Our idea was to use the GPT wizard, but we read in the documentation that the wizard does NOT work with Vision.
Does anyone have any idea how we can implement this?

Flow would be

  1. user uploads an image
  2. GPT recognises the topic
  3. if we have more information about the topic, then we use that to solve the task
  1. GPT divides the tasks into subtasks
  1. if it is an arithmetic task, we send it to Wolfram

You could use the API for all of this.

Make GPT-4V extract the topic and coordinate subtasks. Then, make GPT-4 execute the subtasks through a function to send API requests to Wolfram to solve the arithmetic problems.

I had the same thought at first.
However, the costs here could explode.

First I send the picture => costs
I then receive the topic of the worksheet back
=> OK

Then I send the image back to GPT to get it and identify what Wolfram should solve
==> New costs

Then I tell GPT which task to solve and which tungsten.
==> Additional costs again.

With the API approach, I often repeat the same information and the response takes quite a long time.