How to implement Real time lip sync of avatar chatbot, powered by gpt

GPT builders Plugins / Actions builders
, , ,
24

Hello, can you share that please? Would be super helpful

25

Super Excited and interested as well.

26

at the end we developed a webapp that allows you to create your own avatars, i think we are keep improving it.

27

I want to make a Realtime Listening Learning Tutor App, which tool should I use to do it

28

That sounds interesting.

I recently came across a company called YepAI, which develops highly humanised AI avatars capable of real-time lip-syncing and natural conversations.

These avatars are remarkably lifelike and used to enhance the user experience by delivering engaging interactions with 24/7 availability.

29

Did you see on MuseTalk on GitHub?

30

Iā€™m surprised and tbh disappointed that this hasnā€™t been laid out by now. Itā€™s all possible. HeyGen streaming API is built specifically for this. What weā€™re missing is some clever person to make a quick screenshare of putting the pieces together (for no-coders like me).

31

$699 per month. Totally ridiculous. Someone please record a point and click of connecting with HeyGen streaming avatar - then weā€™ll all be better off.

32

For real-time lipsync with GPT-powered avatars, Mascotbot might be worth checking out.

It provides:

  • A TTS-agnostic real-time lipsync API (works with any TTS, including self-hosted)
  • A React SDK for seamless integration & animation control
  • Pipecat integration for self-hosted AI voice calls

You can use pre-built avatars or do your own. Itā€™s built on Rive, so you get high-performance, 120 FPS animation with full programmatic control (same tech stack duolingo uses for their characters).

33

Mascotbot is nice. Priceyā€¦and a little slow too.

34

HeyGen Streaming Avatar SDK is free to test. It does exactly what is asked of this post. What is needed is a community member to make the API and share the steps in human form :grinning:

35

If you are interested in 3D avatars with real time lip sync and the ability to move around (take action) and perceive the environment, check out Convai.