AIToday
Large Language ModelsAI Safety & AlignmentAI WeeklyPublished: Nov 21, 2026, 01:00 JST

Google Research's AltRA-Test: LLMs pass human-annotator test 35.3% of the time

Send on LINE
Google Research's AltRA-Test: LLMs pass human-annotator test 35.3% of the time

3 Key Points

  1. What happened

    Researchers at the Technion, University of Haifa, and Google Research released AltRA-Test, which checks whether an LLM preserves the signal human annotators share. Across 18 LLMs and 20 datasets, pass rates were 35.3% under default chain-of-thought prompting and 47.3% with the best prompting technique.

  2. Why it matters

    The test shows LLM replacement of human annotators is justified only about half the time at best, so teams relying on LLMs for labeling may need to verify validity per task rather than assume it.

WHO IT HITSData science and research teams that use LLMs to label or evaluate datasets are affected, since the test offers a way to check whether an LLM can stand in for human annotators on their specific task.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

AltRA-Test builds on a prior Alt-Test that required every item to be labeled by the same annotators. The new version removes that requirement, which the authors say makes it workable for the messy annotator subsets typical of large crowdsourced datasets. The procedure runs a bootstrap that resamples items, draws a panel of human annotations, then substitutes one with an LLM annotation to create paired all-human and mixed human-LLM panels. Replacement is supported when the LLM preserves the signal that the human annotators share. Across 18 LLMs and 5 prompting techniques on 20 datasets spanning text, image, audio and video, the pass rates vary sharply by task, and the authors encourage researchers to test replacement for their own task since it is not always justified.

FAQ
How many items are needed to run the AltRA-Test?
The preprint reports that about 100 items often suffice for the test.
Does a favorable prompting technique change the pass rate?
Yes, the pass rate was 35.3% under default chain-of-thought prompting and 47.3% with the best prompting technique.
Which datasets did every tested model fail on?
Every tested model fails on datasets including Persuasion and Measuring-Hate-Speech.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleFeed launches Marugoto LLMO to help firms get cited in AI answers