Introducing the Gemini 2.5 Computer Use Model: A Breakthrough in AI-Powered User Interface Interaction
October 7, 2025 — Google DeepMind today unveiled the Gemini 2.5 Computer Use model, an advanced AI system designed to interact directly with user interfaces (UIs) in web and mobile environments. This new offering, now available in preview via the Gemini API on Google AI Studio and Vertex AI, empowers developers to build intelligent agents capable of navigating and operating graphical interfaces much like a human would.
Key Capabilities and Innovations
Unlike traditional AI models that engage with software through structured APIs, many digital tasks still require direct interaction with graphical user interfaces — for example, filling and submitting forms, manipulating dropdowns, or navigating behind login screens. The Gemini 2.5 Computer Use model enables agents to perform these complex tasks by clicking, typing, scrolling, and interacting seamlessly with UI elements.
Built upon the visual understanding and reasoning strengths of the Gemini 2.5 Pro model, this specialized version offers superior performance and lower latency on numerous web and mobile control benchmarks. It provides a significant step forward in building powerful, general-purpose AI agents capable of managing real-world software environments and workflows.
How It Works
The model is accessed through the new computer_use tool within the Gemini API and is designed to operate iteratively in a loop. Its inputs include the user’s request, a screenshot of the current UI environment, and a history of recent actions taken. Developers can customize this interaction by excluding certain UI actions or adding bespoke functions to suit their needs.
Upon receiving input, the model analyzes the context and generates a response typically structured as a function call—such as clicking a button or typing into a text field. For sensitive actions, like making purchases, the model can request end-user confirmation before proceeding. After the client-side system executes the selected action, fresh visuals and the current URL are sent back to the model to continue the task until completion, error, or termination.
Though primarily optimized for web browsers, Gemini 2.5 also shows promising capabilities for mobile interface control. However, it is not yet suitable for controlling desktop operating system-level functions.
Demonstrations in Action
Google has shared demonstrations highlighting Gemini 2.5’s ability to complete complex multi-step tasks. For instance, one demo shows the model retrieving pet care information from a registration website and then using that data to book a follow-up appointment in a spa CRM system. Another example highlights organizing chaotic virtual sticky notes into designated categories on a collaborative web app, showcasing the model’s versatility in handling real-life UI challenges.
Performance Excellence
The Gemini 2.5 Computer Use model leads the field with high accuracy and low latency. Evaluations by independent groups like Browserbase, alongside Google’s internal testing, confirm the model’s excellence on multiple benchmarks related to web and mobile control.
Safety and Responsible AI Use
Given the risks inherent in giving AI agents control over computer functions, Google has integrated robust safety mechanisms directly into the Gemini 2.5 model. These safeguards address potential misuse, unexpected behaviors, and common attack vectors such as prompt injections or scams.
Key safety features include:
- A per-step safety service that evaluates each proposed action before execution to prevent harmful operations.
- System instructions allowing developers to require user confirmation or block specific high-risk actions, such as compromising security or bypassing CAPTCHA systems.
- Comprehensive documentation providing developers with best practices for maintaining safe and secure AI deployments.
Google urges developers to thoroughly test these systems pre-launch to ensure robust protections.
Early Adoption and Use Cases
Several Google internal teams have already implemented the Gemini 2.5 Computer Use model for production uses, including speeding up UI testing workflows that accelerate software development cycles. It powers components of Project Mariner, the Firebase Testing Agent, and features within AI Mode in Search, illustrating its broad versatility.
Early access program participants similarly leverage the technology for personal assistant applications, automation of workflows, and UI testing, reporting strong results and value.
Access and Next Steps
Developers eager to explore the Gemini 2.5 Computer Use model can start building today via the Gemini API on Google AI Studio and Vertex AI platforms. Feedback is encouraged through Google’s Developer Forum to help refine and improve this cutting-edge technology.
For detailed technical information, evaluation metrics, safety guidelines, and integration instructions, visit the official Google DeepMind blog and Gemini API documentation.
The Gemini 2.5 Computer Use model marks a pivotal advancement in AI’s ability to seamlessly interact with digital environments—ushering in a new era of intelligent automation and user interface control.
