scibert

Why we included this project

General BERT models are pretrained on web text, which handles ordinary language well but misses how terms like gene names and chemical compounds behave in scientific papers. SciBERT was pretrained from scratch on over a million full-text research papers from Semantic Scholar, with a custom vocabulary built for scientific writing instead of general web text. That domain focus pays off on standard scientific information-extraction tasks, including biomedical named-entity recognition, relation extraction, and citation-intent classification, where it posts strong benchmark results. The repo ships cased and uncased checkpoints with both the custom and the original BERT vocabularies, and the pretrained weights load directly through the Hugging Face Transformers API, so you can fine-tune them for your own document pipeline. Evaluation scripts and data are included, which helps if you want to reproduce the reported numbers before adapting the model to your own corpus.