When @ethansdyer and I joined Anthropic last Dec and spearheaded the discovery team, we decided to focus on unlocking computer-use as a bottleneck for scientific discovery. It has been incredible to work on improving computer-use and witness the fast progress. In OSWorld for example, the performance of E2E foundation models have improved from ~8% a year ago to ~61% today with human performance at 72%. Some interesting observations/challenges with computer-use🧵

Claude Haiku 4.5 is released today! It is as strong as Sonnet 4 on coding and much stronger in computer-use. But more importantly, it is 2Xa faster! Watching computer agent doing things on your computer is much more exciting now!

My PhD was a roller coaster 🎢:

My personal story of how I got into PhD program:

Why you should think really hard before committing to a PhD program in ML (this is from 3 years ago and even more true today given the current AI trends):

I still get a lot of questions about doing a PhD in ML so I'm resurfacing a three threads I wrote about this including some fun personal stories 🧵

Very creative!

You can try this on “Claude for Chrome” research preview to get a sense of where things are: claude.ai/chrome

It has been amazing to work with many talented people in Anthropic on this. Some that I could find here are @the_marwell @oh_that_hat @katie_kang_ @shaya_zarkesh I originally got interested in computer-use when I was at Google thanks to very insightful conversations with @anmol01gulati who has been thinking about this and working on this for a long time.

10) Current techniques/models are very slow and we need the time per action to be 0.1 to 1 second (particularly for collaborative agents).

9) For computer-use agents to be very useful, their error rate should be much much lower. For example, for ~3h of autonomous work, the error rate per action should be ~1e-5 (assuming 1 action/sec)

8) We still need to figure out the right interface for both collaborative and autonomous computer-use agents for maximum productivity. That needs some clever product work and many start ups are working on that.

7) We are now transitioning from GPT-2 moment of computer-use to GPT-3 moment when we start to have few-shot behavior working reliably. Imagine being able to upload a video your workflow once (perhaps with your voice explaining what you are doing) and then model can do all future similar workflows with that one example.

6) Computers are very rich and empowering environments. You can do A LOT from your computer including collaborating with others to accomplish real tasks in physical world.

5) Compared to domains like math and code, computer-use has many many more interactions with the environment and in many cases, it is easier for humans or models to verify success of a given trajectory.

4) Using UI is easier for many people so understanding how to do things in UI is going to be a big bottleneck for any collaborative agent that is observing what you do and help you get it done faster.

3) There are many tasks that are either impossible or very hard to do using terminal/code and therefore computer-use ability will likely be a bottleneck for building autonomous agents for very long horizon tasks.

2) Today, a very significant portion of scientific and knowledge work is being done using GUI. This means there is plenty of demonstration data (in some cases with chain-of-thought in voice/transcript format) available and it is very easy to collect such data.

1) Computer-use is the most general interface for the non-physical world designed for maximal cross-task generalization.

GOOD MORNING NEW YORK CITY COME DO YOUR BEST THINKING AT OUR THINKING SPACE IN THE WEST VILLAGE SAY NO TO SLOP