#66 · Primary category： Foundation Models
NExT-GPT
Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model
Project last updated：05/13/25
GitHub Stars
3.6K
Forks
360
Contributors
4
License
BSD-3-Clause
Why we included this project
NExT-GPT is a research model that takes in and outputs text, images, video, and audio in any combination, so a single system can describe a video in words or generate both an image and a sound clip from a text prompt. The repo accompanies the ICML 2024 paper and ships checkpoints, the modality-switching instruction dataset, and documented training steps, making it a workable end-to-end reference implementation rather than something you have to rebuild from the paper. Its training approach tunes only a small fraction of parameters, which keeps it light enough to study. Researchers and engineers comparing multimodal understanding-plus-generation architectures will find an inspectable baseline here. Expect to handle your own serving and integration, since this is a research release, not a turnkey product.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities