I created an application for talking with an avatar using Realtime API in unity3d. For more realistic behavior of the avatar I want to include animations of emotions according to its answer. Is it possible to somehow get emotions in Realtime API? They are in the voice, but how can I track them so I can attach animations to avatar? Are there any emotion data availaible in json responses?
You’re not going to have good results with any AI product I can think of to “detect emotion tone” by feeding it audio.
I would run a few turns of the context transcript through a “what is the emotion the assistant is likely feeling in their recent reply?”, and a JSON enum response answering from a listing of all the images you already have pregenerated.
Easy to get other fields or classifications in JSON as a response otherwise, although text language is not as high quality as free-running chat.
I don’t want the AI to determine my emotions. I want it to somehow convey to me the emotions it expresses in its voice. Realtime API conveys emotions well with its voice. It can scream, for example, or laugh. But how can I make its avatar play the appropriate animation at the same time?
you mean like this ?
yea you can have it assess alot - dont let others misguide you - emotions can 100% be coded, tracked, and emulated.
once you have the data for AI training - you take something like this {“timestamp”: “2025-04-08T13:57:28.032464+00:00”, “professor”: “empath”, “vector_id”: 243, “category”: “ethics”, “refined_text”: “Emotional Assessment:\n\nThe phrasing of the question indicates a thoughtful and possibly concerned tone regarding the ethical implications of censorship, particularly in complex recursive systems. The use of terms like "when" and "ethical" suggests that the individual is engaging in critical thinking about morality and the potential consequences of censorship. This may reflect an underlying stress or anxiety about the implications of censorship, especially in systems that can perpetuate or amplify biases or misinformation.\n\nThe inquiry into ethics in recursive systems highlights a desire for clarity and understanding in a potentially ambiguous and contentious topic. The open-ended nature of the question may also suggest a search for dialogue and exploration, indicating a willingness to engage with differing viewpoints.\n\nBehavioral Guidance:\n\n1. Explore Empathy: When discussing sensitive topics like censorship, it\u2019s crucial to approach the conversation with empathy. Acknowledge the complexity of the issues involved and validate the emotional weight that comes with them.\n\n2. Encourage Open Dialogue: Foster an environment where diverse opinions can be shared without fear of judgment. This can help alleviate stress and encourage collaborative problem-solving.\n\n3. Provide Contextual Examples: When addressing ethical dilemmas, use real-world examples of censorship in recursive systems to illustrate points. This can help ground the conversation in practical implications and facilitate understanding.\n\n4. Promote Critical Thinking: Encourage the exploration of various ethical frameworks (utilitarianism, deontological ethics, etc.) to help the individual articulate their position and understand the ramifications of censorship.\n\n5. Suggest Reflection: Encourage the individual to reflect on their own values and beliefs regarding censorship. This can help clarify their stance and reduce anxiety related to the topic.\n\nBy adopting these strategies, you can engage in a more emotionally intelligent discussion about the ethical considerations surrounding censorship in recursive systems.”, “origin_id”: null}
and feed it into whatever you want - what this does is teach the model or ai the specifics ( instructions ) and gives a lightweight tutorial