lambda-networks

Why we included this project

The lambda layer from the LambdaNetworks paper is a different take on attention. It captures long-range interactions by turning context into linear functions, called lambdas, that get applied to each input separately. This repo packages that idea as a single pip-installable PyTorch layer, so you can drop it into a vision model in a few lines and set a global or localized receptive field with a couple of parameters. It reaches SOTA on ImageNet with less compute, which makes it worth studying. Don't expect a drop-in classifier: it's a reference implementation for reproducing experiments or prototyping new architectures. There's also a Keras port for TensorFlow teams.