FaceFormer

Why we included this project

The FaceFormer repository is the PyTorch code for a CVPR 2022 paper that helped popularize transformer-based methods for speech-driven 3D facial animation. Give it raw audio and a neutral 3D face mesh, and the model autoregressively predicts a sequence of facial motion with accurate lip shapes, along with upper-face expressions rather than just the mouth. Pretrained weights for the VOCASET and BIWI datasets are included, so you can run the demo and animate a mesh without training first, and the code is readable enough to see how wav2vec 2.0 audio features and biased attention are wired together. Anyone prototyping lip-synced avatars or virtual characters should treat it as a research baseline to study and build on, not a plug-and-play SDK: you must request the datasets, install the MPI-IS mesh library, and adapt the data pipeline to your own rig.