#112 · Primary category： Foundation Models
pytorch-openai-transformer-lm
🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI
Project last updated：08/09/21
GitHub Stars
1.5K
Forks
285
Contributors
9
License
MIT
Why we included this project
If you want to see how the original GPT model was actually built, this is a good place to start. It's a PyTorch port of the TensorFlow code that shipped with OpenAI's GPT paper, and the standout piece is the weight-loading script, which transfers the official pre-trained weights into the PyTorch model so it behaves like the published results. The repo also includes the paper's modified Adam optimizer with fixed weight decay and scheduled learning rate, plus a classifier head and training script that reproduce the ROCStories Cloze test accuracy. Since it predates the Hugging Face transformers library, which largely replaced it, treat it as an educational reference rather than production software.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities