Has Your Model Gotten Dumber? How NerfBench Tests Models and Scores Them

Has your model gotten dumber? That feeling can come from two different things.

Sometimes a model really does get worse. In April 2026, Anthropic publicly confirmed three changes. The default effort setting in Claude Code, which controls how much the model thinks, was lowered. A bug cleared earlier thinking on every later turn in conversations that had been idle for more than an hour. And one system prompt instruction asked for shorter answers. All three changes affected the product. That incident did not affect the API.

Other times, it only feels worse. When discussing the GPT-4 controversy, Arvind Narayanan and Sayash Kapoor suggested one possibility: the more people use a model, the more likely they are to notice its limitations. They also warned that changes in a model’s behavior can feel much like a loss of ability to users.

“Nerf” is a gaming term for making a character or weapon weaker. NerfBench aims to replace “I feel like it got worse” with repeated tests.

How NerfBench tests models

NerfBench is a tracker from BridgeBench. It works like giving the same private test over and over: 50 questions, with 3 tries per question. The first result becomes the 100% baseline, and each later result is compared with it.

Scores from 90% to 110% count as normal variation, because scores naturally go up and down just by chance. A result only counts if it goes beyond the range in the same direction twice in a row.

What the leaderboard looks like now

The latest test on the leaderboard was on October 16. It tracks 4 models, and none is outside the normal range.

Power trends for four models on NerfBench, with the normal range shown as 90% to 110% — Power trends for four models and the normal range of 90–110%. Source: BridgeBench NerfBench leaderboard, redrawn by Penchan

How to read the score

NerfBench’s score is called Power. I’ll call it the score below. Usually about 60% of it comes from accuracy. The rest comes from how many tokens (the units used to measure the text a model processes) the model uses and how much it costs, and using less earns more points. So a high score does not mean a model is smarter. Codex does not report costs, and its score is weighted differently.

Calculations using the official formula show two cases worth knowing about. Accuracy can drop quite a bit while the score stays in the normal range. A model can get cheaper and answer fewer questions correctly, yet still get a higher score.

One example from the leaderboard: Opus 5.5 went from 94.2% to 105.8% over two weeks. Both numbers are normal.

Keep these points in mind when you look at a score:

The test covers single answers with no tools, which is different from using Claude Code every day to work on projects.
A changed score does not show what caused it.
The full questions and answers are not public. The public page also does not show the accuracy, token use, or cost for each test, so outsiders cannot recalculate the scores.
BridgeBench says it tests at least three times a week. The public leaderboard has fewer data points, and results are reviewed before they are published.

Who runs this leaderboard

BridgeBench and NerfBench were built by BridgeMind, a company that sells an AI coding tool. In April 2026, results from another BridgeBench test drew criticism. Critics pointed out that the two runs had different numbers of questions. When they compared only the questions shared by both runs, the difference was small. In October, someone publicly asked how the range and weightings were chosen. BridgeBench then published its testing method, but kept the questions private.

Other trackers

AI Stupid Level: frequently tests the APIs of different companies.
LiveNerf: open source, and tests Claude Code every day.
MarginLab: tests Claude Code. As of October 31, it has paused detection and is rebuilding its baseline.

What to check if a model seems worse

Start a new chat and paste in the same question. Long chats can also affect quality. See The Sweet Trap of One Million Tokens to learn why.
Check the effort or thinking setting.
Check the app or CLI version and release notes. Anthropic’s documentation says the weights (the trained model parameters) for the same model ID do not change, but the surrounding service can.
Try the same question several times before deciding, because luck plays a part in any single result. NerfBench also repeats each question.
Look at the tracker last.

A clean leaderboard does not mean nothing has changed for you. And when the leaderboard does show an anomaly, it cannot tell you why.

FAQ

What is NerfBench?

A model tracker from BridgeBench. It repeatedly tests models on the same private set of questions. The first result is the 100% baseline, and each later result is compared with it. Scores from 90% to 110% count as normal. A result only counts if it goes beyond that range in the same direction twice in a row.

Does a higher score mean a model is smarter?

No. Usually about 60% of the score comes from accuracy. The rest is based on how many tokens the model uses and how much it costs, with fewer tokens and lower costs earning more points. A model can get cheaper and answer fewer questions correctly, yet still get a higher score.

What should I check first if a model seems worse?

Start a new chat and try again. Then check the effort or thinking setting and the app or CLI version. Next, try the same question several times. Look at the tracker last.