AI benchmark charts are like test tracks for cars. A lap time tells you something useful, but it does not tell you which car handles school runs, dirt roads, or repair bills best.
Model leaderboards work the same way. The useful question is not “Who is number one?” It is “Does this test resemble the work I need done?”
Match the benchmark to the task
Start with the job, then choose the test. Use public results to build a shortlist of two or three candidates—not to make the final decision.
| Your task | Start with | Where to look | Main caveat |
|---|---|---|---|
| General chat and writing | Arena Text | Arena leaderboard | Measures human preference, not factual correctness |
| Coding agents in repositories | SWE-bench Pro, with Verified as historical context | SWE-bench and SWE-bench Pro | Agent scaffolds, retries, and tests can change the result |
| Algorithmic coding | LiveCodeBench | LiveCodeBench | Contest problems are not messy production repositories |
| Capability versus price and speed | Artificial Analysis | Artificial Analysis | A composite still reflects its chosen tasks and weights |
| Finding benchmark rows quickly | BenchLM | BenchLM | Aggregated rows must be checked against their original source |
| Frontier scientific reasoning | FrontierMath or HLE | Epoch AI | Expensive reasoning settings may not match normal use |
What the main leaderboards actually measure
Arena: which answer people prefer
Arena, formerly LMArena, shows two anonymous model responses and asks the user to choose one. Those pairwise votes produce rankings with uncertainty ranges. This is useful for tone, instruction-following, and whether a model feels helpful in open conversation. It is weak evidence for objective accuracy: a polished wrong answer can still win a preference vote.
On the Text Arena snapshot dated June 25, 2026 (Archive), Claude Fable 5 ranked first at 1508 ±9 from 4,366 votes. Treat that as a dated preference result for that board—not proof that it is the best model for coding, research, or your account.
Artificial Analysis: a controlled multi-test dashboard
Artificial Analysis runs models under documented settings and combines nine evaluations in its Intelligence Index v4.1 methodology. It also reports price, output speed, time per task, and cost per task. That makes it a strong first stop for capability-versus-cost questions.
Its v4.1 model comparison checked July 2, 2026 (Archive) reported scores of 56 for Claude Opus 4.8 at max effort and 55 for GPT-5.5 at xhigh effort. The settings belong beside the numbers: a maximum-effort benchmark run may be slower and more expensive than your production configuration.
SWE-bench: can an agent fix a real issue?
SWE-bench gives a system a historical GitHub issue and repository, applies its patch, and runs tests. Verified is a human-reviewed set of 500 tasks. But the full leaderboard compares systems—not just base models—and retrieval, tools, retries, and review loops can materially affect the resolution rate. Its bash-only mini-SWE-agent view is the cleaner model comparison, and even its 1.x and 2.x releases are not directly comparable.
Verified is now historical context rather than an unquestioned frontier standard. In February 2026, OpenAI reported that at least 59.4% of the 138 frequently failed tasks it audited had material test or problem-description issues, alongside contamination evidence. OpenAI recommends SWE-bench Pro, which uses harder and more diverse work, including held-out and commercial repositories. Multi-SWE-bench addresses a different weakness by expanding beyond Python.
LiveCodeBench: was the coding problem actually unseen?
LiveCodeBench dates problems from LeetCode, AtCoder, and Codeforces so evaluators can select tasks released after a model’s training cutoff. It covers code generation, self-repair, execution, and test-output prediction. Artificial Analysis’s current controlled implementation runs 315 tasks with three repeats.
That makes LiveCodeBench a useful contamination check beside repository benchmarks. It does not tell you whether an agent can navigate your monorepo, preserve local conventions, or understand an underspecified ticket.
BenchLM: the map, not the territory
BenchLM’s methodology (Archive) says it tracked 264 models across 101 benchmarks as of June 27, 2026. It separates sourced “verified” views from provisional rows and links results gathered from benchmark leaderboards, model cards, and provider material.
Use it to discover which tests exist and find candidate models. Then open the original result. An aggregator can preserve a source’s number without making different harnesses comparable.
For the hard end of the spectrum, Epoch AI’s FrontierMath Tiers 1–3 v2 (Archive) reported GPT-5.5 Pro at 87.7% ±1.9% in its internal runs. More instructively, the June 2026 v2 update corrected errors in 42% of problems. Even expert-built benchmarks need revision.
Three questions before trusting a number
- Is it saturated? If frontier models cluster near the ceiling, tiny rank changes may be noise rather than useful separation.
- Could it be contaminated? Public questions and solutions may appear in training data. Prefer held-out, private, or post-cutoff tasks when possible.
- Does the test match your deployment? Record reasoning effort, tools, retries, context limits, and agent version. A score from an expensive max-effort scaffold may not describe the model you will actually run.
Why the leaderboard winner can lose in your repository
Benchmark tasks are curated and bounded. Your repository has half-written documentation, framework quirks, generated files, long-running tests, and conventions that exist only in review comments.
The wrapper also matters. Giving a model search, a terminal, two retries, and a reviewer is a different system from giving the same model one prompt. Language mix matters too: a Python-heavy score says less about a Rust service or React Native application.
So let leaderboards narrow the field. Let your own work choose the winner.
Build a mini-eval in an afternoon
Choose five to ten recent tasks: a bug fix, a small refactor, a test-writing job, a code-review task, and one awkward repository-specific problem. Write the acceptance rule before running anything. Give every candidate the same context, tools, retry limit, and cleanup allowance.
Review and cleanup time belongs in the calculation because cheap tokens can still produce expensive work. Keep clean pass rate next to CAR so you can distinguish “low cost after editing” from “correct on the first attempt.” With only five to ten tasks, the result is directional—not a universal model ranking—but it is more predictive for your workflow than somebody else’s leaderboard average.
Related links
- LLM model guides
- Compare AI models by price tier
- GLM-5.2 vs Kimi K2.6/K2.7 eval checklist
- Smart Spend Guide
- The Honest Limitations of AI Tools
- How AIHackers Verifies Claims
Benchmark examples checked July 2, 2026. Rankings, model availability, and benchmark versions change; open the linked live source before making a production decision.