OmniParser

Why we included this project

Computer-use agents run into a familiar problem: a vision-language model can reason about a screenshot, but it can't reliably tell which pixel is a clickable button or what an icon does. OmniParser addresses that by converting a raw UI screenshot into structured elements, marking each interactive region with a bounding box and a short caption describing its function. It combines a fine-tuned detection model with a captioning model, and the output feeds into any off-the-shelf VLM such as GPT-4V or Qwen-VL, letting the model ground its actions in the correct screen coordinates. It's a building block rather than a full agent framework; you wire it into your own pipeline, and the companion OmniTool shows a working Windows VM setup. For teams prototyping a vision-only agent that has to operate real applications on desktop or mobile, this is a well-documented starting point for the screen-understanding layer.