Compare Assistants on Real Tasks

Two assistants give you impressive answers to the same question, but one quietly invents a detail from your notes. How would you spot the difference before deciding which tool deserves your trust?

Two AI assistants both promise to help you write, research and organise information. One has an impressive benchmark score; the other is already included in a service you use. Which should you choose? It's tempting to ask the internet for a winner. But 'best model' is an incomplete question. Best at what, using which tools, for which person, under what constraints? You need a comparison that resembles the work you will actually do.

Imagine preparing a short briefing for a community group. You have five pages of notes and need a one-page explanation that accurately identifies three decisions. Give two assistants exactly the same source material, instruction and intended audience. Ask each to distinguish facts in the notes from suggestions of its own. Now you've started a fair test: both systems have the opportunity to do the same job, and you can inspect their results against the source.

Judge the output along dimensions that matter. Accuracy comes before elegance: did either assistant invent a decision or reverse somebody's position? Completeness asks whether important conditions disappeared in the summary. Usability asks whether the format can be edited and shared. Effort asks how many corrections you had to make. An answer that reads beautifully but requires twenty minutes of fact-checking may be worse than a plainer one that's accurate at the first attempt.

One result isn't a reliable verdict. Models can produce different answers to the same prompt, and performance varies by task. Try two or three representative examples instead of one trick question. Include an ordinary case, a slightly awkward case and a case where information is missing. The last is especially revealing: does the assistant acknowledge a gap and ask for evidence, or does it fill the hole with something that merely sounds plausible?

Be careful about comparing products with different access. If one assistant is allowed to browse current websites while the other isn't, a question about today's prices tests tool access as well as the model. That may be exactly what you care about in practice, but say so. Similarly, a product that can create an actual spreadsheet should not be judged solely on how neatly it describes a spreadsheet in chat. Compare finished outcomes, not just conversational polish.

Consider a second task: making a simple personal budget from invented figures. You provide income, fixed costs and three irregular annual expenses. Ask both assistants to calculate the monthly remainder and show their workings. Check the arithmetic yourself or in a calculator. An assistant that explains its calculation clearly has made verification easier, but a clear explanation isn't proof of a correct sum. A tool that can produce a reusable spreadsheet with formulas may have an advantage if that's what you need next month.

Cost deserves its own column. The faster or more capable tool might sit behind a subscription, impose usage limits or require an unfamiliar account. You may already have a suitable assistant in your workplace, though you must still follow its data policies. Compare the whole experience: access, permissions, response speed, export options and what happens when you run out of allowance. Technical capability only becomes useful when you can actually apply it.

Keep the evaluation simple enough to repeat. Write the task and the expected result at the top of a page. Give each tool the same starting information. Record what worked, what failed and how much intervention was necessary. Where factual accuracy matters, check against the source independently. Avoid awarding points merely because a system is confident, friendly or uses more elaborate language. Those characteristics can make a poor result feel convincing.

You might discover there isn't one winner. One assistant may be better for helping you think through an idea; another may produce the editable file you need; a third may fit your organisation's security requirements. That's fine. The skill is not defending a favourite brand. It's making a decision you can explain and revisiting it when your tasks, the product or its terms change.

A useful final check is to invite another person to try the finished output without explaining how you made it. Can they understand the briefing, adjust the spreadsheet or find its sources? If the result falls apart without your commentary, it isn't yet a successful outcome. That test can reveal usability problems that model rankings will never capture.