transfusion-pytorch

Why we included this project

MetaAI's Transfusion paper proposed training a single model on both text and continuous data like images, and this repo is a compact PyTorch implementation of that idea. The code swaps the paper's diffusion objective for flow matching, the approach behind Flux, and keeps the API simple: you hand it interleaved sequences where integer tensors are text and float tensors are modalities, and it produces a combined loss for training plus a sampling method for joint generation. It also handles multiple continuous modalities, lets you set different latent dimensions for each, and accepts custom encoders and decoders. Because it's a training-oriented codebase rather than a pretrained model, it's best for studying and adapting the architecture for your own experiments.