Vibe coders have long suspected that the models and reasoning effort they pay for have been quietly downgraded since release. For instance, on October 23, a user filed an issue on GitHub saying they had requested Sol 6.1 at xhigh in Codex but the account UI showed Luna. Either the model selection or the UI is wrong. Opus 5.5, which scores higher than Fable 5.1 in Anthropic’s benchmarks yet costs less, is on everyone’s radar as a potential next victim. In this context, when Opus is accused of being “nerfed”, users are asking for consistency, not a one-time peak score.

LiveNerf tracks Opus 5.5 through Claude Code, with results free to read on GitHub. It repeats fixed questions and compares accuracy and output-token counts against a baseline from the benchmark’s first ten days. Most readers can follow the public charts in a browser. Those who want to run the benchmark can use its Python runner on Linux, macOS or Windows, and code contributions are welcome through GitHub.

The first completed comparison reports “no change”. Do be mindful the benchmark, as all benchmarks are, is limited to the scope of its own criteria. “Nerf” is a colloquial catch-all term. Subscribers are judging not only the mere quality of the answers, but the model’s general capability, speed, and its cost performance too.

I see this project as a community effort to keep Anthropic, and possibly other AI companies, in line. It gives us a tool to measure and question what is being delivered. The companies selling the service should have been upfront about that in the first place.