DualPipe

Why we included this project

DualPipe is the bidirectional pipeline parallelism algorithm DeepSeek used to train V3 and R1. It overlaps forward and backward computation with communication so GPUs stay busy instead of idling in pipeline bubbles between micro-batches. The repo includes runnable Python examples for both the original DualPipe schedule and the simpler DualPipeV variant, plus a comparison table that lays out the bubble and memory trade-offs against the standard 1F1B and ZB1P approaches. Teams training very large language models across many GPUs will find it a useful reference for getting more throughput out of pipeline-parallel setups. Treat it as a scheduling blueprint to adapt into your own stack rather than a turnkey framework: real deployments require wiring in a custom overlapped forward-backward method.