DeepSeek-671B-SFT-Guide

Why we included this project

Fine-tuning a 671B-parameter MoE model is a serious undertaking, and this guide is one of the few places that documents the whole process openly. It extends the xtuner framework with the DeepSeek-V3/R1 training logic, and includes the modeling files, training scripts, and data-format handling needed to run full-parameter SFT across a multi-node cluster using DeepSpeed ZeRO data parallelism plus sequence parallelism. The authors, from the Institute of Automation of the Chinese Academy of Sciences and Beijing Wenge Technology, also wrote down the pitfalls and problems they hit during training and deployment, along with the solutions, and recorded experiment results across different parallel strategies, including memory footprints and which configurations ran out of memory. The reasoning-data format extension and loss-field handling for multi-turn dialogue are small details that save real debugging time. If you have access to a large H100 cluster and want to adapt DeepSeek for domain-specific reasoning tasks, this is a useful reference implementation to start from.