RT-2

Why we included this project

RT-2 is a compact PyTorch reimplementation of Google's vision-language-action model, small enough to read end to end and easy to modify. It pairs a PaLM-E style language backbone with a vision encoder, so camera images and text instructions land in a shared embedding space and the model emits action tokens a robot can execute. For teams trying to understand how web-scale pretraining transfers into robotic manipulation, the code is a friendlier entry point than the paper alone. It installs with pip, exposes a single RT2 class that bundles the vision and language pieces, and includes straightforward examples for feeding in images and captions. Treat it as a research and prototyping reference rather than a production control stack; real deployment still needs the robotics data pipeline and hardware integration described in the original work.