Is there a way to help o1-mini take less time to think? Are there any recommended ways to achieve this? Would the number of reasoning tokens used be an accurate measurement of time performance?
Could you share how many secs it takes for you and the number of reasoning tokens? In my case it takes on average 3k of reasoning tokens at about 15-20sec wait time which is roughly the same number of input tokens
You could try and limit the output by specifying output size, this would in turn limit the thinking steps…
‘Design a complete reference guide to creating your very own AI in 1000 words.’
I have tried a few things and that’s all that has worked for me so far
Maybe being more specific in the output you wanted could tailor steps a bit more too though not necessarily quite so consistently
I tried a few ways to tell it ‘in x reasoning steps’ though that didn’t work either more or less
My use case is returning a specific json that is dynamically generated so the number of tokens varies, though is less than 1000 words for sure. In the prompt I do include the expected structure of the json, not examples though. I could include that and see
Placing limitations (ie ranges or areas to cover) on the scope of each JSON section may help reduce uncertainties.
The more refined the potential output one would expect the less reasoning to be done.
It’s hard to see without understanding your use case but another approach might be to compute or reason parts of the JSON separately (even with different and cheaper/faster models)
Either in parallel process…
Or parts of the process that maybe don’t need the same depth of reasoning
Hierarchical Reasoning ie dividing up the JSON into a Tree may reduce redundancy
I came across this the other day…
https://arxiv.org/html/2305.10601v2
I have only skimmed it, I’m not very good at technical papers like this I don’t get how to read the formulas if not code but I did see some ideas like depth/breadth searching in Trees from projects I have worked on, you also might also consider CRUD in terms of trees. I think it might give more suggestions.
I’m not sure if your wait is excessive or not but part of 01 is the fact that it takes longer to give a response, mainly because it’s devivsing an aswer after going thru it’s own chain of thought then running an answer she’s checking it against the chain of though if your looking for speed you probably be better off using the 40 model and refining your prompts to get the quality of answer your do from 01. That or teach it give it 3 examples of every possibly scenario or variation you can thing of and then in the instructions tell 01 to not scrutinize it’s answer so much and to give it to your faster
Thanks! Of course, I’m not saying to not wait just looking into ways to optimize it from my end. I didn’t think just asking it to do it faster would be an option. I’ll try that today!