What’s the Point of Computer Use Agents?

Computer use agents (CUAs) are agents that work autonomously across different apps, websites, and tools to complete a goal on a computer. The key difference between CUAs and other agents is that CUAs operate directly on the same graphical user interface (GUI) as human users: they take screenshots as inputs and act in the same action space as humans (e.g., click/type/scroll). This makes CUAs the most general form for automating computer work. It is also quite mesmerizing to watch them work:

In early 2024, I shifted my research focus from multimodal LLMs and image generation to focus almost entirely on CUAs. This was motivated by two conclusions that I arrived at after many conversations at ICML 2023:

But as cool as CUAs are, there are some obvious downsides compared to other types of agents. It’s common for me to receive pushback about the usefulness, relevance, or longevity of CUAs. Despite this, I remain more excited about CUAs than ever. It really feels like this is the year where CUAs start to become as important as (or dare I say, more important than) other already ubiquitous agents. In this article, I’ll explain why I think CUAs are here to stay, and the challenges that remain before they will be broadly deployed.

This is the first argument someone typically brings up regarding the practicality of CUAs. Text-based LLMs have historically progressed a lot faster than multimodal models. Even today, many frontier models are also disproportionately better at text and coding than vision-heavy tasks. Consequently, one of the most common arguments against CUAs is that structured text interfaces (APIs, MCPs, or bash) is more reliable, cheaper¹, and easier to validate programmatically. Why should we bother with CUAs, which require visual grounding and reasoning capabilities? I think that there are a few compelling reasons:

Computer use is the most general interface for digital work. Any task done on the computer can be treated as a CUA task (even SWE!), but not all tasks can be easily completed through APIs or text-based interfaces. Some examples include:

CUAs are necessary for 100% automation coverage. We can attempt to build APIs for every single app and feature that we need, indefinitely maintaining an agent-parallel computer interface, or we can let CUAs handle the long tail. APIs will never have 100% coverage, especially for rarely used user-facing features. There will always be a long tail of actions that cannot be done through text interfaces or APIs. Using CUAs to cover this long-tail allows us to fully automate any task without worrying about API coverage.

User interface and app testing. Any software that will be used by a real human will need to be eventually tested through the same human user interface. By definition, this is not completely covered by headless or text-based testing. Using a CUA for app testing can expose human-facing bugs, or verify the intuitiveness of the UI and UX³.

Of course, I don’t really believe that we will end up with pure CUAs (even though I’d actually quite like this :)). We’ll end up with hybrid agents that use whatever tool is best for the job at hand⁴. For tasks that involve crossover between a wider variety of apps, act over longer horizons, or rely on proprietary software, the amount of CUA intervention required will also increase.

Aside from capabilities, one major feature of CUAs is that they use the same UIs and action spaces that humans already use for computer work. This allows CUAs to trivially plug into existing workflows, and inherit the permissions, information privileges, security mechanisms, and safeguards of existing software.

Presumably in the future, we’ll continue to have user interfaces (broadly defined) that humans use, and we’ll need agents that are also capable of operating on these UIs. CUAs enable more natural user-agent collaboration. With CUAs, it becomes significantly more straightforward for an agent to take over a task. For example, a user may open a few windows, hand off something to a CUA, and resume control once the task is complete to inspect or give additional work to the CUA. Agents operating through a different interface introduce contextual gaps in terms of both UI understanding and communication.

For work that is very heavily visually grounded, operating in the GUI can also be more intuitive for the agent. App testing as described above is one, and graphically heavy tasks, such as 3D modeling with CAD software or video editing, also intuitively seem like they would continue to benefit from agents that operate visually. This also makes it far easier for an experienced user to supervise the agent’s work, or interrupt to take over and fix minor mistakes.

Hopefully, the above points have convinced you that we require a capable CUA to automate the full suite of economically valuable computer work. Aside from this, another reason I originally got interested in CUAs was because they provide grounding for interesting research problems that other agents do not. To solve computer use, we require multimodal LLMs with several fundamental capabilities:

Computer use makes for a very natural and organic setting for testing these multimodal capabilities. I also consider CUAs to be in the goldilocks region between non-embodied multimodal tasks (such as chat-based tasks) and embodied robotics tasks:

A strong CUA possesses many capabilities that transfer well to vision-language action (VLA) models: grounding, visual reasoning, long horizon planning, and the ability to act in real time. Solving computer use will directly result in research that is useful for robotics. For example, we’ve seen recently that Astra seems to have very compelling robotics capabilities, and many have hypothesized that this is due to transfer from its strong computer use (especially Blender and CAD) abilities.

For researchers who are more interested in grounded sequential decision-making, long-horizon interaction, or multimodal perception, CUAs present an idealized setting to study these problems. Experiments are cheaper to run, infrastructure is simpler, and iteration cycles are much faster. CUAs present a useful testbed for isolating and improving these foundational capabilities in VLMs and VLAs. We will of course still need important advancements in embodiment and motor control to optimally transfer research findings in these areas to actual real world robots.

The past year has brought extremely rapid progress on computer use. For me personally, the release of the GPT-5.4 and Opus 4.6 models marked a turning point where I felt like the models were actually useful for simple to medium difficulty tasks in my day-to-day life. GPT-5.6 was another step change as well. Unlike a year ago, I now regularly use CUAs for menial and repetitive tasks on a daily basis. The models are still not perfect. For many tasks that are a bit more complex, CUAs are still worse than me in terms of speed and quality. But they have far surpassed the best models from a year ago, and are already good enough to improve the productivity of most people.

So what’s left to do? A lot of the low hanging fruit has been picked, and the remaining challenges are more interesting and difficult. Here are a couple that are top of mind for me. We have a few projects coming out soon along these directions that I’m very excited about:

Speed and cost: While models have been getting steadily faster, frontier CUAs are still far slower than humans when performing even simple tasks end-to-end⁵. There are two main sources of bottlenecks. The first is prefill and generation throughput: CUAs have to process large screenshots and generate long reasoning traces and actions. This (especially keeping screenshots in history) compounds and fills up the context window quickly, slowing things down over the course of many turns. A second factor is action efficiency. Many modern CUAs execute more actions than a human would on the same task, which makes them spend a lot more (expensive) turns on the same problem. Existing work explores speeding up CUAs along several axes: systems and infrastructure, model architecture, multi-action prediction, multi-agent parallelization, hardware, and more. All of these will continue to be important, and general LLM speedups and efficiency gains will also transfer to CUAs. Model architecture research is also a very interesting direction to speed models up, and will probably be necessary to get a 10x or 100x speedup for realtime CUAs (e.g., to play video games).

Personalization: Being attuned to a user’s preferences, data, and past behavior is especially important for CUAs. CUAs will likely operate locally on personal devices, referencing and working off the user’s personal data, and will need to adapt on the fly as the context and habits of the user change. MyPCBench is one benchmark that we’ve built to measure how CUAs behave when required to perform tasks that reference past user context and personal data. I expect that this will become a central focus of CUA research this year as models have already mostly saturated benchmarks for testing general non-personalized tasks. Another interesting direction along these lines is always-on CUAs: models that continuously monitor and learn from your computer history. This has the potential to unlock personal assistants with a perfect context and memory of your working preferences.

Infrastructure and UX: What is the right way to deploy and use CUAs? Prior to 2025, most CUA products were deployed as a blocking agent on the user’s computer (or virtual machine), and you would sit there and watch it (slowly) do its thing. This has changed in the past couple of months. Several frameworks have come up with engineering tricks to enable CUAs to run in the background. Codex was the first to show that this was technically possible and not as hard as previously thought. This was probably the most important CUA UX breakthrough this year, and it makes them a lot more feasible to be used in day to day work. There are still many interesting alternative interfaces that we can explore, especially those that enable realtime human-agent collaboration (rather than turn based). Another interesting idea is that of ambient proactive CUAs, which observe and provide suggestions rather than act only when prompted.

Multi-Agent Systems: Almost all CUA frameworks and papers today focus on testing and deploying these systems as single serial agents. This is not how they will operate in the real world. We will need to be able to seamlessly integrate CUAs with other types of agents (tool calls, coding, reasoning, etc.) to effectively handle the full spectrum of tasks that we care about. We will also need to start orchestrating CUAs as multi-agent systems, which will allow them to work faster, divide-and-conquer, and generally perform better than single-agents overall.

Security and Privacy: As a consequence of being more general, CUAs can also be harder to gate compared to coding and bash agents. CUAs have access to more personal data and computer permissions (as they essentially inherit user permissions). As CUAs are deployed more widely, it becomes increasingly important to ensure they operate safely and align with user intents. We have already seen evidence that CUAs are more susceptible to content injection, and personal agents with permissive access possess an immense potential to wreak havoc on your personal data. Being able to ensure that these models are well aligned and operate within confined and safe scopes is an increasingly important research direction for future CUA deployments.

Computer use is messier and less verifiable than coding, math, and other axiomatic domains, which is partly why it has trailed these domains in terms of progress. For myself, this is what also makes CUAs a lot more interesting, and why I believe it will be harder to completely solve. As a community, we’ve made a ton of breakthroughs in the past year, and I think we’re very close to having CUAs be a part of the daily workflows of regular people.

We already have useful products with CUA as a core capability: Codex, Claude in Chrome, Grok Bot, are some that I quite like. However, it feels like CUA applications and interfaces are still in their infancy. Personally, I have not found any of these existing products to be polished or robust enough for my hardest tasks, and while they’re fine for self-contained tasks, they don’t really work very well for longer horizon work.

There’s still a lot of important research to be done, but it’s looking like we will have superhuman CUAs within the near year (arguably, we already do on certain domains). I find it very exciting to think about the products and workflows that these will bring. Once we have models that can operate any digital interface with superhuman speed and expertise, we unlock drastically new directions for automation, products, and experiences.

Acknowledgements. Thank you to Pranjal Aggarwal, Daniel Fried, Lawrence Jang, Wendy Kua, Graham Neubig, Ruslan Salakhutdinov, and Jeremy Tien for helpful discussion and feedback on an earlier draft of this blog post.