Mish

Why we included this project

Mish is the reference implementation for the self-regularized non-monotonic activation function introduced in the BMVC 2020 paper of the same name, a smoother alternative to ReLU that can improve accuracy on convolutional and transformer-style networks. It provides working implementations for PyTorch, Keras, TensorFlow, JAX, and several other frameworks, so you can drop it into your existing stack without writing custom layers. The repo also documents where the activation performed best on real benchmarks like ImageNet and MS-COCO object detection, giving you a sense of whether it's worth swapping in for your own architecture. Useful extras include pointers to faster CUDA builds and memory-efficient variants, plus a variance-based initialization trick that pairs well with Mish when you adopt it at scale.