Team: I’m using the openAI API to generate responses to an arbitrary set of user inputs, then creating a relation graph to illustrate similarities between the questions. The end result will be a graph that displays n user inputs and their calculated relation to each other.
Since I anticipate this data set eventually being quite large, indexing and search is going to be an issue. I’d like to use the tokenized input text as part of my addressing scheme, as I think that would provide options for a useful search parameter. Is there a way to get those already?
If not, would it be possible to add an optional field “returnTokens” to openai.Completion.create, such that when returnTokens = true, the response object includes the tokens generated from the input text?
I realize I could use another library to tokenize the input and index that way, but if the project continues to use openai, having direct access to the generated tokens could prove useful for later development.