Have not tested this in particular with Ultra. No API available yet.
So, waiting to see what happens once the Gemini Pro 1.5 API is released. But, looks like we’re a few months away for that.
I will say that, in more testing, I have been able to at least get some answers from Gemini Pro. Again, in the Bible scenario, I am able to get answers to “hard fact” kinds of questions: who, what when where.
But, it absolutely sucks at higher level questions that require some degree of introspection and depth, like this question which is easily answered by gpt-4.5-turbo and claude v3:
Having worked with Gemini Pro 1.0 now for going on 3 months, mostly struggling, I am thinking it’s problem is not so much it’s ability to comprehend, but “rail guards” put on it. That is, censorship. And I can say further that it’s performance has varied over the past few weeks – a period that happens to coincide with the whole Gemini controversy in the media.
And, I found at least one other person on the Gemini API Discord forum who has documented the same sort of up and down performance over the same period. Discord
In conclusion, while it may be great at recognizing and drawing images, or writing games or answering puzzle questions, I find it severely lacking as the “generation” model in a RAG scenario where it is tasked with comprehending large amounts of fairly sophisticated texts. You are better off, as much as I hate to admit it, with gpt-3.5-turbo-16k.