How well do Large Language Models actually reason when faced with questions humans find straightforward? Simple Bench answers this with a free benchmark of 213 adversarial multiple-choice questions designed to expose logical blindspots in spatio-temporal reasoning, social intelligence, and linguistic strongness — areas where even top LLMs like Claude 3.5 Sonnet score below 42% while humans reach over 83%.
Uncovering LLM Reasoning Gaps
Simple Bench provides a multiple-choice text benchmark with over 200 questions specifically crafted to assess LLMs’ reasoning capabilities. These questions explore areas that aren’t covered by standard benchmarks areas such as spatio-temporal reasoning, social intelligence, and linguistic adversarial resilience, often referred to as "trick questions." The tool highlights instances where LLMs struggle with basic reasoning and common sense, even when humans find the questions straightforward. It evaluates models using standardized prompts and presents a leaderboard for performance comparison.
Who Benefits from This Benchmark?
This tool primarily targets AI researchers, developers, and anyone interested in understanding the limitations of current LLMs. It serves as a critical instrument for gauging the progress of AI models in reasoning relative to human-level performance. By exposing areas where LLMs lack logical coherence and contextual understanding, Simple Bench helps the community identify specific challenges in AI development.
Technical Deep Dive into Evaluation
The benchmark consists of 213 multiple-choice questions, each offering six options. To ensure reliable evaluation, models run each question five times, typically with a temperature of 0.7 and a top-p value of 0.95. The evaluation setup includes prompts designed to elicit maximal behavior and induce chain-of-thought reasoning in models. The platform features a public leaderboard showcasing the performance of various LLMs, providing transparent insights into their reasoning abilities.
The Persistent Gap: Humans vs. LLMs
Simple Bench consistently reveals a significant performance disparity between humans and LLMs on its reasoning tasks. Human baselines typically range from 83.7% to 92% accuracy. In stark contrast, even leading LLMs like Claude 3.5 Sonnet score considerably lower, achieving around 27% or 41.4%, while o1-preview reaches 41.7%. This highlights fundamental limitations in LLMs’ common sense, temporal reasoning, and social intelligence. While prompt engineering can sometimes improve scores, it often points to deeper issues in the models’ reasoning rather than just a lack of knowledge. The benchmark is still under development, with ongoing efforts to explore more failure modes and refine its assessment capabilities.
Why Simple Bench Stands Out
Simple Bench excels at identifying and quantifying the gap between human and LLM reasoning, particularly for questions that are simple for humans but challenging for AI. Unlike many traditional benchmarks that advanced models have "saturated," Simple Bench focuses on exposing the lack of logical coherence and contextual understanding in LLMs. This provides a more precise measure of their reasoning capabilities against a human baseline, making it particularly useful for evaluating how well LLMs handle "trick questions" and scenarios requiring real-world common sense.
Accessing the Tool
Simple Bench is a free tool. Users can explore the benchmark results and model comparisons directly on its official website: https://simple-bench.com/