nanoT5

Why we included this project

nanoT5 is a PyTorch template that lets you pre-train a T5-base encoder-decoder model from scratch on a single GPU in under 24 hours, then fine-tune it on instruction data. That combination is rare: most lightweight training repos focus on decoder-only GPT or encoder-only BERT, so this fills a gap for anyone who wants to experiment with encoder-decoder architectures on limited hardware. The pipeline is designed to be read line by line, from C4 data streaming through training and evaluation, and because it sticks closely to the HuggingFace T5 implementation, any model changes you make remain portable and compatible with the larger ecosystem. There's also a simplified T5 model and training loop meant for teaching, which makes it a solid baseline for testing research hypotheses before spending compute on a bigger run.