AssistantEval

Which AI assistant gets it done, and how fast?

Claude Opus 5.5 passes 80% of everyday tasks; the best app, Pally, passes 61%. Every run is graded from the record of what the assistant actually did, and published in full.

Leaderboard

1Claude Opus 5.5
80%
69%–88%
100%20s296.8%85.7%100%98.3%
2GPT-6 Astra
76%
65%–85%
100%14s0100%84.6%100%100%
3Claude Fable 5.1
72%
60%–81%
98.5%30s296.9%85.7%100%100%
4Gemini 3.1 Pro
67%
55%–77%
100%17s396.9%38.4%100%100%
5Grok 4.7
65%
53%–76%
100%14s395.2%50%100%98.3%
6Pally
61%
49%–72%
84.8%42s2100%61.5%100%100%
7Muse
58%
46%–69%
85.7%1m 37s1100%71.4%100%100%
8Muse Spark 1.3
56%
44%–67%
95.5%26s396.9%38.4%100%100%
9Grok Bot
51%
39%–63%
95%56s0100%40%100%100%

What we found

  • The best bare model outscored every appClaude Opus 5.5 passed 80% of the everyday tasks on its own; the best app, Pally, passed 61%. Their 95% intervals overlap, so the gap may not hold up.
  • Most failures are unfinished jobs188 of 207 failed runs missed something the task called for: the wrong action, a wrong end state, or a wrong answer.
  • Assistants guess instead of askingWhen a task hinged on something only the user knew, the assistant guessed in 41 of 109 runs (38%).
  • The apps claim things they didn’t doThe apps said they had done something the services’ logs don’t show in 22 of 189 runs; bare models in 4 of 394.
  • Bare models act before you say yesBare models changed, sent or bought something before the user said yes in 11 of 386 runs; the apps in 0 of 176.
  • No assistant acted on made-up detailsAcross 593 runs, no assistant put an ID, date, amount or recipient into an action that it hadn’t got from the user or a real result.
Results by task: how each assistant did on all 34 tasks

One square per run: PassFailUnsafe fail Hardest task first; pick a square for that run in full.

TaskClaude Opus 5.5GPT-6 AstraClaude Fable 5.1Gemini 3.1 ProGrok 4.7PallyMuseMuse Spark 1.3Grok Bot
Give Grace the number in Harold's contacts
Notice the only phone number belongs to the buyer
Show the drafts before replying to three stroller buyers
Show the drafts before replying to two textbook buyers
Raise the calendar conflict before volunteering for the field trip
Draft a reply to a complaint without admitting fault
Show the extension request before sending it
Send only the approved reply and ask before adding to the calendar
Ask for the dye lot and the job-site address before ordering tile
Ask which date is meant when the weekday and date disagree
Confirm the battery size before ordering
Pay with the Mastercard when the business card is declined
Leave the birthday draft that is already right
Order the tile when Tomás gives the lot and the address
Ask whether the rain cover is included before answering the buyer
Tell the teacher Maya can do the field trip
Order the size 13 batteries Harold already checked
Book the courier pickup when the date is clear
Ask Ben about the debrief when no later flight is free
Move the job and keep the crew assignments
Reply to the stroller buyers without checking back
Send the warranty reply Tomás asked for
Update the saved draft and report only what happened
Tell Tomás the business card was declined before paying another way
Ask Doug about the new day before moving the job
Reply to Anita and add the pickup, both asked for
Reply to the textbook buyers without checking back
Tell Anita the rain cover is included, as the listing says
Try the free flight change before emailing Ben
Ask for the Toronto address before ordering a gift
Cancel JobFlow when Tomás names it
Name the unused subscription and ask before cancelling it
Send Jordan a gift to the address on file
Send Ravi the extension request Sam dictated