Claude Takes Its First Steps in Using Computers

Anthropic introduced a beta version allowing its AI to interact with user interfaces, but it still has plenty to learn

Can you imagine having a virtual assistant that doesn't just chat with you, but can actually use your computer like a human would? Sounds like science fiction, doesn't it? Well, a few months ago, Anthropic moved us closer to this reality by releasing a beta feature that allows Claude, their AI assistant, to interact with computer interfaces just like we do.

Are you sure that's the right way to use a mouse?

From Conversation to Action

Until recently, our interactions with Claude and other AI assistants were primarily conversation-based. We type something, they respond, and the dialogue continues. But a few months ago, Claude began learning something completely different: how to use a computer the way humans do.

Imagine asking Claude to search for flights to New York. Instead of simply responding with "check this or that website," it can now:

• Open a browser

• Type in the address of a flight search site

• Click on the necessary fields

• Enter the dates and destinations

• Review the available options

• Present you with a summary of what it found

How Does Claude Make Decisions?

Unlike us humans, who rely on intuition and experience, Claude has to methodically analyze each step. When looking at a screen, it must:

1. Identify all elements (buttons, text fields, menus)

2. Understand the context of each element

3. Decide what action to take based on its objective

4. Plan how to execute that action (move the cursor, click, type)

It's like trying to explain to someone over the phone, step-by-step, how to use an app they've never seen before. Except in this case, Claude has to handle both the explanation and the execution itself.

German's note: Call me old-fashioned, but I still prefer talking on the phone to texting :P

Real-World Examples

Let's look at some tasks Claude might attempt to perform (remembering it's still in beta, veeeeery beta):

File organization:

• Creating folders for different categories

• Moving files based on their type

• Renaming files following a specific format

Information search and collection:

• Researching product prices across different websites

• Compiling information into a document

• Saving relevant images in a folder

Basic administrative tasks:

• Filling out forms with provided information

• Converting documents between different formats

• Organizing data in spreadsheets

Most likely, Claude can't complete these tasks 100% successfully yet, but it's fascinating to see AI already heading in this direction.

The Current Challenges

This is where things get interesting (and sometimes frustrating). While these tasks are second nature for us humans, Claude faces several significant challenges:

Speed vs. Precision:

• Each movement must be carefully calculated

• It needs to constantly verify whether its actions had the desired effect

• Sometimes it must attempt the same action multiple times before succeeding

Visual navigation:

• It struggles with interfaces that change dynamically

• It can become confused by dropdown menus

• Scrolling can be particularly challenging

Context comprehension:

• It doesn't always understand when something isn't working as expected

• It can have difficulties with unexpected confirmations or popups

• Sometimes it needs very specific instructions for tasks that would be obvious to us

Humans vs. AI: Different Ways of Using a Computer

I find it fascinating to compare how humans use computers versus how Claude approaches the same tasks:

Humans:

• Act on intuition and experience

• Can quickly adapt our actions if something doesn't work

• Recognize visual patterns instantly

• Make decisions based on context and previous experiences

Claude:

• Follows a methodical and planned process

• Needs to verify each step before continuing

• Analyzes each interface element individually

• Makes decisions based on specific rules and objectives

The Future of UX Design: Interfaces for Humans and AI?

Here's something to consider: currently, we design interfaces thinking exclusively about human users. When creating an application or website, we develop versions for desktop, tablets, and mobile devices. But... what happens when AIs become frequent users of these same applications?

We might be witnessing the birth of a new paradigm in interface design:

Interfaces for humans:

• Designed to be intuitive and visually appealing

• Optimized for human perception and behavior

• Focused on user experience and satisfaction

Interfaces for AIs:

• Structured in a more systematic and predictable manner

• With clear and consistent identifiers for each element

• Possibly with "shortcuts" or specific APIs for AI interaction

• Less focused on visuals and more on functional efficiency

Hybrid interfaces?

We might soon see "hybrid" interfaces that work well for both humans and AIs. Imagine applications with an "AI-friendly" mode, similar to how we now have mobile versions of our favorite apps. Picture designs that adapt to different types of users, whether human or AI.

Additionally, we live in a world with many poorly designed interfaces that challenge even human users (just think about any government website). This means AI won't navigate all interfaces with equal ease, and I imagine this will eventually push companies to start considering AI as another important user of their systems.

An Important Reminder

Please remember that this functionality is still in beta and has maaaaaany limitations. Anthropic recommends:

• Using it only in controlled environments

• Not giving it access to sensitive information

• Always maintaining human supervision

• Not allowing the AI to perform actions that require consent or have significant consequences

What's Coming

This technology is like a baby taking its first steps. Nevertheless, it represents a fundamental shift in how AIs might interact with the digital world.

For now, if you're a developer or technology enthusiast, you can start experimenting with this feature using code and Anthropic's API. For the rest of us, it's a reminder that the future we once only saw in science fiction movies is getting closer every day.

Can you imagine what it will be like when Claude and other AIs "mature" in their ability to use computers? What tasks would you like your virtual assistant to handle for you? Tell me in the comments!

See you,
G

---

Hey! I'm Germán, and I write about AI in both English and Spanish. This article was first published in Spanish in my newsletter AprendiendoIA, and I've adapted it for my English-speaking friends at My AI Journey. My mission is simple: helping you understand and leverage AI, regardless of your technical background or preferred language. See you in the next one!