JoyHallo

Why we included this project

JoyHallo turns a portrait photo and an audio track into a talking-head video, syncing lip movement, facial expressions, and head pose to the speech. It is built for Mandarin, using a Chinese wav2vec2 encoder to handle the language's intricate mouth shapes, and the authors note it still produces believable English clips. The project ships with a Gradio demo and command-line inference, plus fine-tuning support so you can adapt it with your own portrait-video data. If you plan to modify the model rather than just run it, the semi-decoupled design that separates lip, expression, and pose features is worth a look. Expect a research-grade codebase: it needs a capable GPU and careful data prep, not a turnkey product.