
What happened
Researchers at the Technion, University of Haifa, and Google Research released AltRA-Test, which checks whether an LLM preserves the signal human annotators share. Across 18 LLMs and 20 datasets, pass rates were 35.3% under default chain-of-thought prompting and 47.3% with the best prompting technique.
Why it matters
The test shows LLM replacement of human annotators is justified only about half the time at best, so teams relying on LLMs for labeling may need to verify validity per task rather than assume it.
WHO IT HITSData science and research teams that use LLMs to label or evaluate datasets are affected, since the test offers a way to check whether an LLM can stand in for human annotators on their specific task.
Summaries like this, every morning.
AltRA-Test builds on a prior Alt-Test that required every item to be labeled by the same annotators. The new version removes that requirement, which the authors say makes it workable for the messy annotator subsets typical of large crowdsourced datasets. The procedure runs a bootstrap that resamples items, draws a panel of human annotations, then substitutes one with an LLM annotation to create paired all-human and mixed human-LLM panels. Replacement is supported when the LLM preserves the signal that the human annotators share. Across 18 LLMs and 5 prompting techniques on 20 datasets spanning text, image, audio and video, the pass rates vary sharply by task, and the authors encourage researchers to test replacement for their own task since it is not always justified.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
The Nov 19 arXiv paper found ICLR critique overlap rose from 12.0% in 2026 to 21.3% in 2027, with LLM-flagged…

Feed Inc. began offering "Marugoto LLMO" on November 17, 2026, covering AI research, question design, page rew…

Co-Labo AI published the first of three explainer installments on ontology on November 20, 2026, aimed at manu…

The guide maps input types to fields — images and video to computer vision (CV), text to natural language proc…

Across 52 laboratory tasks in physical sciences, life sciences and engineering, Claude Fable 5.2 led 13 agents…

Meta's AI assistant Muse can list and unfollow Instagram accounts that don't follow you back; Khalil Roberts u…
