https://benchr.org/ Pricing, benchmarks, and use-case fit for AI model selection. en-us Mon, 07 Sep 2026 21:14:01 +0000 corrections@benchr.org (benchr) https://benchr.org/articles/kimi-k2-7-code-review https://benchr.org/articles/kimi-k2-7-code-review Tue, 08 Sep 2026 00:00:00 +0000 Moonshot's coding model claims ~30% fewer thinking tokens than K2.6, on a rate card where output costs 4.2x input. The high-speed variant doubles every line. https://benchr.org/articles/gemini-3-8-flash-review https://benchr.org/articles/gemini-3-8-flash-review Mon, 07 Sep 2026 00:00:00 +0000 Gemini 3.8 Flash is priced to the cent like 3.7 Flash. Google says it spends more tokens on purpose, which is where the cost difference lives. https://benchr.org/articles/deepseek-v4-flash-vision-exp-review https://benchr.org/articles/deepseek-v4-flash-vision-exp-review Mon, 07 Sep 2026 00:00:00 +0000 DeepSeek charges nothing extra for image input on this checkpoint. What you trade is stability: it is experimental and has no published license. https://benchr.org/techniques/write-the-shot-not-the-vibe https://benchr.org/techniques/write-the-shot-not-the-vibe Fri, 04 Sep 2026 00:00:00 +0000 A shot description that fixes camera, lens, movement, light and the constants is doing the job a shot list and a cinematographer would do. Short prompts are… https://benchr.org/techniques/write-the-repo-rules-down-once https://benchr.org/techniques/write-the-repo-rules-down-once Fri, 04 Sep 2026 00:00:00 +0000 Put the build command, the test command, the untouchable files and the definition of a complete change in a file at the repository root. Agents read it. So… https://benchr.org/discover/an-agent-tuned-another-agent-until-it-stopped-failing https://benchr.org/discover/an-agent-tuned-another-agent-until-it-stopped-failing Fri, 04 Sep 2026 00:00:00 +0000 The optimiser was not a training run or a hyper-parameter sweep. It was a coding agent editing another agent's prompts and tools, judged by a benchmark. https://benchr.org/fix/the-video-shimmers-and-nothing-holds-still https://benchr.org/fix/the-video-shimmers-and-nothing-holds-still Fri, 04 Sep 2026 00:00:00 +0000 Shorten the shot. Most shimmer becomes invisible below about two seconds, and a cut costs nothing. https://benchr.org/fix/the-bill-was-much-bigger-than-expected https://benchr.org/fix/the-bill-was-much-bigger-than-expected Fri, 04 Sep 2026 00:00:00 +0000 Find the largest unchanging block in your prompt. If it is re-sent every turn and not cached, that single change is usually most of the bill. https://benchr.org/fix/the-video-changes-the-thing-between-shots https://benchr.org/fix/the-video-changes-the-thing-between-shots Fri, 04 Sep 2026 00:00:00 +0000 Generate from a still of the actual product rather than from a description of it, for every single shot. https://benchr.org/fix/the-text-inside-the-image-is-gibberish https://benchr.org/fix/the-text-inside-the-image-is-gibberish Fri, 04 Sep 2026 00:00:00 +0000 Keep the text to a few words, put it in straight quotation marks in the prompt, and say where on the image it goes. https://benchr.org/fix/the-eyes-and-hair-drift-across-a-set https://benchr.org/fix/the-eyes-and-hair-drift-across-a-set Fri, 04 Sep 2026 00:00:00 +0000 Name the attributes that must not move — eye color, hairline, one distinctive mark — and check those three specifically rather than judging the face as a whole. https://benchr.org/fix/the-agent-clicks-the-wrong-thing https://benchr.org/fix/the-agent-clicks-the-wrong-thing Fri, 04 Sep 2026 00:00:00 +0000 Refer to targets by their accessible name, not by where they are. 'The button labeled Continue' survives a layout change; 'the button on the right' does not. https://benchr.org/techniques/schema-first-prompt-second https://benchr.org/techniques/schema-first-prompt-second Fri, 04 Sep 2026 00:00:00 +0000 Write the output schema before the instruction. The prompt then only has to describe the judgement, because the shape is already enforced. https://benchr.org/fix/the-person-changes-between-images https://benchr.org/fix/the-person-changes-between-images Fri, 04 Sep 2026 00:00:00 +0000 Stop re-describing the person and start re-supplying them: pass the first image back in as a reference for every subsequent generation. https://benchr.org/techniques/quote-the-string-leave-the-space https://benchr.org/techniques/quote-the-string-leave-the-space Fri, 04 Sep 2026 00:00:00 +0000 Two moves, in order of reliability: quote a short string exactly, or generate the composition with deliberate empty space and set the type in an editor. https://benchr.org/techniques/put-the-rule-last https://benchr.org/techniques/put-the-rule-last Fri, 04 Sep 2026 00:00:00 +0000 Instructions placed after a long attachment are followed more consistently than the same instructions placed before it. Recency is doing work that emphasis… https://benchr.org/techniques/outline-then-one-unit-per-turn https://benchr.org/techniques/outline-then-one-unit-per-turn Fri, 04 Sep 2026 00:00:00 +0000 Split the job before the output limit splits it for you, at a boundary you chose rather than mid-sentence. https://benchr.org/discover/maintainers-started-writing-the-rules-for-robots https://benchr.org/discover/maintainers-started-writing-the-rules-for-robots Fri, 04 Sep 2026 00:00:00 +0000 The quote that explains the whole shift is Tindle's: an agent-written pull request is "basically somebody else paying for your compute". https://benchr.org/techniques/one-reference-one-variable https://benchr.org/techniques/one-reference-one-variable Fri, 04 Sep 2026 00:00:00 +0000 Stop describing the subject and start re-supplying it. Then change exactly one thing per generation so you can tell what caused a drift. https://benchr.org/discover/the-web-tasks-that-stop-agents-stop-people-too https://benchr.org/discover/the-web-tasks-that-stop-agents-stop-people-too Fri, 04 Sep 2026 00:00:00 +0000 One agent beats the human baseline on partial progress and still loses on finishing. https://benchr.org/fix/the-agent-broke-code-that-was-working https://benchr.org/fix/the-agent-broke-code-that-was-working Fri, 04 Sep 2026 00:00:00 +0000 Work on a branch, and read the diff before the tests. A large diff for a small request is the signal, not the test result. https://benchr.org/techniques/make-it-restate-the-brief https://benchr.org/techniques/make-it-restate-the-brief Fri, 04 Sep 2026 00:00:00 +0000 One paragraph, in its own words, before it starts. You find out whether the constraints landed while it is still cheap to correct. https://benchr.org/fix/the-run-dies-halfway-because-i-hit-a-limit https://benchr.org/fix/the-run-dies-halfway-because-i-hit-a-limit Fri, 04 Sep 2026 00:00:00 +0000 Read the error before retrying. A 429 with a retry hint wants a wait; a quota error wants a smaller job, because no amount of backoff refills a balance. https://benchr.org/fix/the-agent-passes-the-test-and-fails-the-job https://benchr.org/fix/the-agent-passes-the-test-and-fails-the-job Fri, 04 Sep 2026 00:00:00 +0000 Take ten tasks from your own week, write down what a correct answer looks like, and run those. That is your benchmark; everyone else's measures someone… https://benchr.org/fix/the-pdf-comes-back-half-read https://benchr.org/fix/the-pdf-comes-back-half-read Fri, 04 Sep 2026 00:00:00 +0000 Ask it one question it can only answer from the end of the document — a figure from the last table, the final section heading. If it cannot, the file did… https://benchr.org/fix/the-json-comes-back-broken https://benchr.org/fix/the-json-comes-back-broken Fri, 04 Sep 2026 00:00:00 +0000 Stop asking for JSON and start supplying a schema. The API feature is called structured outputs and it is a request parameter, not a prompt trick. https://benchr.org/fix/it-stops-halfway-through-a-long-job https://benchr.org/fix/it-stops-halfway-through-a-long-job Fri, 04 Sep 2026 00:00:00 +0000 Ask for one section, not the whole thing, and then ask for the next. Two short answers beat one truncated one. https://benchr.org/fix/it-cannot-read-the-number-off-the-chart https://benchr.org/fix/it-cannot-read-the-number-off-the-chart Fri, 04 Sep 2026 00:00:00 +0000 Give it the numbers, not a picture of the numbers. A CSV export, the underlying table, or the data endpoint removes the entire failure mode. https://benchr.org/fix/the-prompt-works-on-one-model-and-fails-on-another https://benchr.org/fix/the-prompt-works-on-one-model-and-fails-on-another Fri, 04 Sep 2026 00:00:00 +0000 Before blaming the prompt, check that the feature it depends on exists on the target model at all. Half of these failures are a missing parameter, not a… https://benchr.org/techniques/hold-back-a-validation-set-the-agent-never-sees https://benchr.org/techniques/hold-back-a-validation-set-the-agent-never-sees Fri, 04 Sep 2026 00:00:00 +0000 Your own tasks, split in two. You iterate against one half; the other half is only ever run, never tuned against. The second number is the real one. https://benchr.org/fix/it-forgot-what-i-told-it-earlier https://benchr.org/fix/it-forgot-what-i-told-it-earlier Fri, 04 Sep 2026 00:00:00 +0000 Re-state the two or three rules that matter immediately before the request, not at the top of the conversation. Recency beats position. https://benchr.org/techniques/give-it-the-data-not-a-picture-of-the-data https://benchr.org/techniques/give-it-the-data-not-a-picture-of-the-data Fri, 04 Sep 2026 00:00:00 +0000 The single highest-value substitution in analysis work. A CSV removes an entire measured failure mode that no prompt fixes. https://benchr.org/techniques/cache-the-part-that-never-changes https://benchr.org/techniques/cache-the-part-that-never-changes Fri, 04 Sep 2026 00:00:00 +0000 Find the largest block of your prompt that is identical on every call — the system prompt, the style guide, the reference document — and stop paying full… https://benchr.org/techniques/ask-for-the-last-page-first https://benchr.org/techniques/ask-for-the-last-page-first Fri, 04 Sep 2026 00:00:00 +0000 A thirty-second check that tells you whether the model actually has the whole document before you trust a summary of it. https://benchr.org/discover/openclaw-outgrew-the-encyclopaedia-entry-about-it https://benchr.org/discover/openclaw-outgrew-the-encyclopaedia-entry-about-it Fri, 04 Sep 2026 00:00:00 +0000 The interface is the surprise. No terminal, no dashboard, no browser tab — the agent is a contact in a chat app you already have open. https://benchr.org/discover/ten-million-tokens-and-eighty-five-minutes-to-fail https://benchr.org/discover/ten-million-tokens-and-eighty-five-minutes-to-fail Fri, 04 Sep 2026 00:00:00 +0000 The cost is the headline, not the score. Eighty-five minutes and ten million tokens is a real bill, and it buys a first-time finish about one time in seven. https://benchr.org/discover/a-95-minute-feature-shot-by-fifteen-people-in-two-weeks https://benchr.org/discover/a-95-minute-feature-shot-by-fifteen-people-in-two-weeks Fri, 04 Sep 2026 00:00:00 +0000 Eighty percent of the budget was compute, not people — the opposite of how a film has ever been costed. https://benchr.org/discover/a-two-thousand-dollar-film-premiered-at-tribeca https://benchr.org/discover/a-two-thousand-dollar-film-premiered-at-tribeca Fri, 04 Sep 2026 00:00:00 +0000 A festival premiere is the bar that separates a demo from a film, and this one was cleared at a hobby budget. https://benchr.org/discover/no-api-to-publish-so-the-model-built-a-browser-extension https://benchr.org/discover/no-api-to-publish-so-the-model-built-a-browser-extension Thu, 03 Sep 2026 00:00:00 +0000 The missing interface did not stop the task; the model built the interface. https://benchr.org/discover/the-model-installed-its-own-blender-tooling https://benchr.org/discover/the-model-installed-its-own-blender-tooling Thu, 03 Sep 2026 00:00:00 +0000 Repairing its own working environment was not the task it was given; it was what the task required first.