Investment bankers reject all AI outputs in first major real-world benchmark — even GPT-5.4 fails to meet client standards

Handshake AI and McGill University tested nine top AI models (GPT-5.4, Claude Opus 4.6, Gemini 2.5 Pro, and others) against 100 realistic investment banking tasks using BankerToolBench, an open-source benchmark created with input from 500 current and former bankers at Goldman Sachs, JPMorgan, Morgan Stanley, and other major firms. Not a single model output was ready to send to a client as-is; 41% needed major rework and 27% were completely unusable.

The benchmark grades actual deliverables junior bankers produce — Excel financial models with working formulas, PowerPoint presentations, PDFs, and Word memos — against 150 individual criteria covering technical correctness, compliance, and client readiness. GPT-5.4 scored highest at 58.1 out of 100 but still failed nearly half the criteria. Claude Opus 4.6 produces polished-looking outputs that hide a critical flaw: key numbers are hardcoded as fixed values rather than linked to formulas, making it impossible to run scenario analyses (e.g., changing a purchase price updates nothing).

For investment banking teams, this means AI cannot yet replace junior analyst work or streamline client deliverables, despite vendor claims. The benchmark also revealed why: models make four recurring mistakes — buggy code generation (41% of failures), broken business logic like adding costs to the revenue line (27%), aborted data queries (18%), and fabricating missing numbers and presenting them as sourced (13%).

The full benchmark, including test data, rubrics, and an AI verifier called Gandalf, is publicly available and can also be used to train models via reinforcement learning. Early experiments with smaller models like Qwen showed five- to thirteen-fold performance gains, though from very low baselines.