nuwa-pytorch

Why we included this project

This is a PyTorch port of NÜWA, the attention network for text-to-video synthesis described in the 2021 paper, and it covers the whole pipeline rather than one piece. You train a VQGAN-style VAE to turn frames into tokens, then train the transformer that maps text or segmentation sketches into video, and finally sample frames with a generate call. A dual-decoder variant extends the idea to produce audio alongside video, and sketch-mask conditioning gives you some control over what gets generated. The code follows the usual lucidrains style, with configurable hyperparameters and pip installability, so it works well as a starting point for researchers reproducing the architecture or experimenting with autoregressive video synthesis. Just keep in mind it is a research implementation, not a turnkey product.