open-musiclm

Why we included this project

This PyTorch reimplementation of Google's MusicLM is worth a look if you want to generate music from text and are ready to train your own model rather than call a hosted API. It follows the paper's three-stage design, modeling semantic, coarse, and fine tokens, but swaps the original components for openly available ones: CLAP replaces MuLan, Encodec stands in for SoundStream, and MERT takes the place of w2v-BERT. That swap matters because you can actually get the pretrained checkpoints and run the pipeline, instead of being stuck with Google's proprietary weights. The repo walks through the whole training process, from the residual vector quantizer and K-means layer through the three transformer stages, and includes a preprocessing step to cache tokens ahead of time. It also compares itself directly to the more faithful musiclm-pytorch implementation, with honest notes on early results, which helps when you're deciding which codebase to build on.