bookcorpus

Why we included this project

The original BookCorpus is no longer publicly distributed, which is a real obstacle for anyone trying to reproduce the sentence-encoder and decoder experiments built on it. This repo ships working Python scripts that crawl the same source the dataset came from, smashwords: the pipeline collects a list of free book URLs, downloads each title as plain text, and falls back on extracting text from epub files when that is the only format available. The result is a sentence-per-line file you can feed straight into a tokenizer, with an optional check that discards epub conversions whose word count diverges too far from the official stats. If you would rather not crawl at all, the README points to public copies of the data and is frank about the flaws in each, including a paper on the known deficiencies of BookCorpus and its replicates. Just remember that the underlying books remain under copyright and subject to the source site's terms of service.