audiolm-pytorch

Why we included this project

Google's AudioLM showed that audio generation can be treated as a language modeling problem: compress sound into discrete tokens, then predict the next token. This PyTorch reimplementation gives you that whole pipeline to run yourself, with SoundStream or EnCodec as the encoder and a transformer trained on the resulting token sequences. It also extends the paper with classifier-free guidance and T5 conditioning, which opens up text-to-audio and even a VALL-E-style text-to-speech setup without assembling pieces from several repositories. Anyone who already trains transformer models will find it a practical base for experiments and for adapting the approach to their own audio data. Expect to supply your own data and compute, though; this is research code, not a hosted service.