mmdit

Why we included this project

Stable Diffusion 3's MMDiT block is the cross-attention layer that lets text steer image generation, and this repo implements it as a single, readable PyTorch module. Rather than a full training pipeline, you get one block you can drop into your own diffusion code, plus a generalized version that extends the same idea beyond two modalities, so text, image, audio, and video tokens can all be mixed in one transformer. The code is short enough to read through quickly, which helps if you want to understand how rectified-flow transformers combine conditioning signals or adapt the design to your own generation experiments. There's also an improvised self-attention variant with learned gating for anyone curious about alternatives to the standard formulation.