
Summaries like this, in your inbox every morning.
Sign up free →LLMs fail when obvious surface cues (like distance mentions) conflict with unstated constraints, with distance cues wielding 8.7-38x more influence than goal statements
The Heuristic Override Benchmark tests 14 models across 500 instances with 4 heuristic types and 5 constraint families, finding presence constraints are hardest (44% accuracy)
Token-level analysis reveals models rely on keyword associations rather than compositional reasoning, suggesting surface-level pattern matching rather than deep understanding
Simple hints like emphasizing key objects improve performance by +15 percentage points, indicating the problem is constraint inference rather than fundamental capability gaps
No comments yet. Be the first to share your thoughts!
Log in to join the discussion




Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.
Get Started FreeFree · takes 30 seconds · unsubscribe anytime
1 minute a day. The AI essentials.
200+ sources · Email / LINE / Slack