Surge AI's DAYJOB: Tax finds AI misses what to ask

Surge AI's point of departure is that most existing tax benchmarks hand the AI a pre-specified job — answer a tax-law question, compute a figure for a return. Real advisory work starts earlier: the client may not know they have a tax problem, and the documents may disagree with each other. The benchmark was built with tax professionals to force that earlier step, and the file counts show the scale involved — a median of 30 source files per task.

The results split along that seam. In the home-sale case, GPT-6 Astra correctly judged that selling at either date Anna was weighing would satisfy the residence test, but it never told her about the deadline implied by the rules: past roughly late August 2027, more than three of the prior five years would fall after she moved out, putting the two-year residency requirement out of reach. Claude Opus 5.5 flagged that deadline in every run. In the retirement-split case the roles reversed: Claude Opus 5.5 proceeded on the assumption the couple was not legally married, while GPT-6 Astra raised the need to confirm marital status first in four of five runs.

Surge AI's conclusion is that handling individual pieces of work is not the same as reconciling a client's account with the documents, deciding what else must be checked, and carrying that through into a return or advice. The two examples show why that distinction is concrete rather than philosophical — the same model that caught one verification step missed another.