OmniNet

Why we included this project

OmniNet is a compact codebase for anyone curious about training one model on multiple tasks across text, images, and video. It is the code released by the OmniNet paper's authors, so the pretrained weights, dataset download script, and training commands reproduce the published results rather than being a loosely related third-party port. The architecture routes each input domain through its own neural peripheral into a shared Transformer-based processor, and that shared core is what lets a single model handle POS tagging, image captioning, visual question answering, and video activity recognition at once. The repo also demonstrates multi-GPU hogwild training across heterogeneous tasks and includes experimental zero-shot prediction for video captioning and video QA, which makes it useful for teaching or digging into unified multimodal architectures. Just keep in mind it has not been maintained since 2020 and is not built for production deployment.