#116 · Primary category： Foundation Models
PaLM-pytorch
Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways
Project last updated：11/09/22
GitHub Stars
824
Forks
81
Contributors
3
License
MIT
Why we included this project
PaLM-pytorch is a compact PyTorch port of the transformer architecture Google described in its PaLM paper, and the whole thing fits in under 200 lines. That brevity is deliberate: the author wrote it as a teaching tool, not a production model, so you can trace the attention mechanism, feed-forward blocks, and normalization choices straight from the paper without digging through a large codebase. You can spin up a small PaLM-style model, train it on something like Enwik8, and see the design decisions play out. It won't scale to 540B parameters, but for anyone learning transformer internals or borrowing PaLM-style ideas for their own experiments, the clean, dependency-light code is easy to adapt.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities