MetaTransformer

Why we included this project

MetaTransformer tackles a question that keeps coming up in multimodal research: can one transformer backbone handle many different input types? The authors show that a single encoder with frozen weights can process twelve modalities, from images and text to point clouds, audio, time series, and even IMU sensor data, by mapping each raw input into a shared token space first. That design lets you reproduce how a vision-pretrained backbone transfers to other domains without paired multimodal training data, which is an unusual setup compared with the usual separate-encoder-per-modality approach. The code is organized around tokenizers, a shared encoder, and task-specific heads, so adapting the pipeline to your own benchmark is fairly direct. Treat it as a research blueprint for unified multimodal perception rather than a turnkey application. For teams studying cross-modal feature sharing, it is a clear and testable reference.