OmniParser for Pure Vision Based GUI Agent - Hugging face

Model Summary

模型概况

OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent. Training Datasets include:

1. an interactable icon detection dataset, which was curated from popular web pages and automatically annotated to highlight clickable and actionable regions, and
2. an icon description dataset, designed to associate each UI element with its corresponding function.
   OmniParser是一个通用的屏幕解析工具，该工具可以将UI屏幕截图给解释/转化成结构化的格式，去改进现有基于LLM的UI代理。它训练的数据集包括：
   1）一个可交互的图标检测数据集，它从热门网页中进行精选，并自动标注以突出显示可点击和可操作的区域，并且
   2）一个图标描述数据集，用于将UI元素和对应功能关联起来。

This model hub includes a finetuned version of YOLOv8 and a finetuned BLIP-2 model on the above dataset respectively. For more details of the models used and finetuning, please refer to the paper.
该模型中心分别包含一个YOLOv8的微调版本，和基于上述数据集的微调BLIP-2模型。关于模型使用和微调的更多信息，请参考文章。

Responsible AI Considerations

对AI的负责任考虑

Intended Use

使用目的

• OmniParser is designed to be able to convert unstructured screenshot image into structured list of elements including interactable regions location and captions of icons on its potential functionality.
  OmniParser 被设计用于将非结构化屏幕截图图像转换为结构化元素列表，包含可交互区域位置及关于潜在功能上的图标标题。
• OmniParser is intended to be used in settings where users are already trained on responsible analytic approaches and critical reasoning is expected. OmniParser is capable of providing extracted information from the screenshot, however human judgement is needed for the output of OmniParser.
  OmniParser 主要用于用户已经接受过负责任分析方法的培训并且需求批判性推理的环境。OmniParser 能够提供从屏幕截图中提取的信息，但其输出仍需要人工判断。
• OmniParser is intended to be used on various screenshots, which includes both PC and Phone, and also on various applications.
  OmniParser 可以用于不同的屏幕截图，包括PC和手机，以及各种应用程序。

limitations

限制

• OmniParser is designed to faithfully convert screenshot image into structured elements of interactable regions and semantics of the screen, while it does not detect harmful content in its input (like users have freedom to decide the input of any LLMs), users are expected to provide input to the OmniParser that is not harmful.
  OmniParser 被用于将屏幕截图图像给准确转换成屏幕上可交互区域和语义的结构化元素，但同时它无法检测到输入中的有害内容（就像用户可以自由决定任何LLM的输入一样），用户也需要提供无害的输入内容给OmniParser。
• While OmniParser only converts screenshot image into texts, it can be used to construct an GUI agent based on LLMs that is actionable. When developing and operating the agent using OmniParser, the developers need to be responsible and follow common safety standard.
  尽管 OmniParser 只能将屏幕截图图像转换成文本，它还可以用来构建基于LLM的可以执行操作的客户端代理。当使用 OmniParser 来开发和使用代理时，开发者需要负责任和遵循公共安全标准。
• For OmniPaser-BLIP2, it may incorrectly infer the gender or other sensitive attribute (e.g., race, religion etc.) of individuals in icon images. Inference of sensitive attributes may rely upon stereotypes and generalizations rather than information about specific individuals and are more likely to be incorrect for marginalized people. Incorrect inferences may result in significant physical or psychological injury or restrict, infringe upon or undermine the ability to realize an individual’s human rights. We do not recommend use of OmniParser in any workplace-like use case scenario.
  对于 OmniPaser-BLIP2，它可能会错误地推断图标图像里的个人的性别或其他敏感属性（如种族，宗教等）。对于敏感属性的推断很可能是依赖于刻板印象和概括，而不是根据特定个人的信息，这对边缘人群来讲很可能是不对的。不正确的推论可能导致严重的身体上或心理上的伤害，或限制，或侵犯实现个人人权的能力。我们不建议在任何类似工作场所用例场景中使用 OmniParser。

License

许可

Please note that icon_detect model is under AGPL license, and icon_caption_blip2 & icon_caption_florence is under MIT license. Please refer to the LICENSE file in the folder of each model.
请注意 icon_detect 模型是采用AGPL许可，而 icon_caption_blip2 和 icon_caption_florence 模型是采用MIT许可。具体请参阅每个模型文件夹下的LICENSE文件。