How to implement Real time lip sync of avatar chatbot, powered by gpt

I wanted to create a human chatbot that will listen to the questions
of users and answer it and lip of human will be synced with the answer.
so kind of real time voice converstional avatar interaction users can have.
I will be using GPT natural language converstion.

But i am not getting any solution how can i implement the real time lip sync of the avatar, what tools or models i need to use in order to achive this.

1 Like

Look into HeyGen: Quick Start

Interesting subject. You’ll be surprised that there are not many TTS solutions that support out-of-the-box lip sync.

Azure and Amazon TTS produce built-in lipsync data. It’s called viseme. Look them up.

You’ll find a lot of AI video solutions, but none of them are suitable for real-time animations.

1 Like

There is a very comprehensive Facebook group on this topic at Virtual Beings, just Google it.

1 Like

Hey we have done a hackathon project before using RealTime lip-sync solution.
neat part is that it can be integrated as a react component. no complicated game engine setup
“github dot com /BennyKok/leaked-zoom”

you can ask any questions in their discord “discord dot gg / ZXKaZq4gMR”

1 Like