I wanted to create a human chatbot that will listen to the questions
of users and answer it and lip of human will be synced with the answer.
so kind of real time voice converstional avatar interaction users can have.
I will be using GPT natural language converstion.
But i am not getting any solution how can i implement the real time lip sync of the avatar, what tools or models i need to use in order to achive this.
Interesting subject. You’ll be surprised that there are not many TTS solutions that support out-of-the-box lip sync.
Azure and Amazon TTS produce built-in lipsync data. It’s called viseme. Look them up.
You’ll find a lot of AI video solutions, but none of them are suitable for real-time animations.
There is a very comprehensive Facebook group on this topic at Virtual Beings, just Google it.
Hey we have done a hackathon project before using RealTime lip-sync solution.
neat part is that it can be integrated as a react component. no complicated game engine setup
“github dot com /BennyKok/leaked-zoom”
you can ask any questions in their discord “discord dot gg / ZXKaZq4gMR”
Hi,
Did you make any good progress on this? We want to do something similar. Hoping you could help point us in the right direction.
I am also interested in this, so please let me know if you come up with any solution
Interested as well. So far I found these solutions:
Character API by Media Semantics (available on AWS Marketplace) offers real-time animation with lip-sync.
This is what i found in the internet but implementing might have to do bit of googling.