weibo_terminater

Why we included this project

Anyone building Chinese-language dialogue models quickly hits the same wall: clean, large-scale conversational data is hard to come by. This scraper pulls public Weibo posts, comments, and follower information so you can assemble your own corpus instead of buying or scraping from proprietary sources. Its distinguishing piece is a script that turns comment threads into conversational reply pairs, which is what makes the output suitable for chatbot training rather than just raw text dumps. The crawler also rotates multiple accounts to dodge rate limits and can resume from where it stopped, useful when running long collection jobs. The author explicitly labels it for academic research and hasn't touched the code since 2019, so treat it as a source of working techniques and raw material to adapt, not a maintained service. Developers who need Chinese dialogue data will find the reply-pair extraction and anti-ban logic worth studying even if they rewrite the scraping layer.