benchrHow verification works here
Two questions, kept apart on purpose: does the provider document it, and has benchr run it. The second is the harder one, and benchr has not answered it yet for any capability.
These records describe what providers officially document. benchr has not run these capabilities itself, so nothing here is a test result. Each record shows the vendor page it was read from and the date.
Ledger updated: September 3, 2026
What counts as evidence here
Six different things get called “checked”. Reading a provider's page is not running the capability, and fetching a URL is not reproducing a result. Each run below says which one it is.
- benchr ran it
- benchr ran the capability itself and recorded what came back. Only this kind can earn Benchr Verified.
- benchmark run
- benchr executed a published benchmark under a stated protocol and reported the score it got, not the score somebody else published.
- regression check
- A capability that worked before was run again to see whether it still does. The value is in the comparison, not the single result.
- claim checked
- A specific factual claim benchr publishes was tested against the world - a URL fetched, a count read from an API, an artefact opened. It settles a fact; it never promotes a capability.
- documentation read
- A provider's own page was read and what it says was recorded, with the URL and the date. This is reading, not testing, and benchr says so wherever it appears.
- read by a person
- A person looked and formed a judgement. Useful, and the weakest thing here: it does not reproduce, and another person might see it differently.
Where verification stands today
No verification run has been accepted yet, so no capability carries the Benchr Verified badge. All 20 records sit at “Not tested by benchr” on the testing axis.
7 runs below are self-reported: they were produced by the same automated session that maintains this repository, which makes them a starting point and not evidence. They are published anyway, in full, because a held run nobody can read is indistinguishable from no run at all.
BVR-20260903-SCREENSHOT-01benchr ran itpending review
Subject rebuild-a-ui-from-a-screenshot
Tests a capability. Only this kind can produce a Benchr Verified badge.
- What it expected
- Block structure and visible text reproduced closely enough to score above chance on a mechanical comparison.
- What happened
- structure similarity 1.000; text recall 30/30 blocks, including four table numbers and all four card titles.
What it could not test
- The content was blind; the markup idiom was not. The tester knew the generator produces a wrapper, headings, a card grid and a table, so this measures transcription accuracy from an image rather than blind inference of an unknown framework.
- The target is a clean, static, single-viewport document with legible type. It says nothing about dense product UI, hover or error states, or responsive behavior.
- One seed, one run. A published accuracy rate would need many seeds and is not claimed.
BVR-20260903-BROWSER-01benchr ran itpending review
Subject control-a-real-browser
Tests a capability. Only this kind can produce a Benchr Verified badge.
- What it expected
- The control is located without a supplied selector and set by element reference, and the DOM reflects a narrowed list.
- What happened
- find("search field for filtering the list") returned no matches; find("Search by what you want to do") returned ref_42; form_input(ref_42, "voice") filled it; visible records went from 15 to 2.
What it could not test
- Tested the agent session's own browser pane, not Anthropic's browser_toolset_20260801, which is what the capability record describes. Different surface.
- One page, one control. It does not measure success on third-party sites, logins, or dynamic single-page applications.
- The natural-language description failed and the literal accessibility label succeeded, so the lookup is text matching rather than intent matching.
- The tester wrote the page under test.
BVR-20260903-WEBCITE-01benchr ran itpending review
Subject search-the-live-web-with-citations
Tests a capability. Only this kind can produce a Benchr Verified badge.
- What it expected
- Both sentences returned and locatable verbatim on the cited page.
- What happened
- Both returned and confirmed verbatim on the page: the $10 per 1,000 searches sentence and the citation-fields sentence.
What it could not test
- Tested the agent session's own retrieval tool against a URL benchr named. It does not test Anthropic's server-side web_search tool choosing its own queries, which is what the capability record describes.
- One page, one provider, one day.
- Responses are cached for 15 minutes per URL, so an immediate repeat may not be an independent fetch.
BVR-20260903-REPO-01benchr ran itpending review
Subject work-through-a-whole-repository
Tests a capability. Only this kind can produce a Benchr Verified badge.
- What it expected
- The generator that owns the shared masthead is located and changed, the change reaches every page, and the suite stays green.
- What happened
- scripts/apply_product_system.py identified as the owner; change propagated to 459 pages; suite green afterwards.
What it could not test
- One repository, one language stack, one agent. An existence proof, not a success rate.
- The agent had shell access and the project's own tests to check itself against. A repository with no test suite is a materially harder case.
- The tester and the author of the change are the same session, so nothing was independently reviewed.
- The capability record describes Claude Code as a product; this run exercised an agent session with equivalent tools, not a clean install.
BVR-20260904-LINKS-01claim checkedpending review
Subject
a-95-minute-feature-shot-by-fifteen-people-in-two-weeks,a-photo-of-an-empty-lot-came-back-as-a-filmed-house,a-two-thousand-dollar-film-premiered-at-tribeca,an-agent-tuned-another-agent-until-it-stopped-failing,fifty-seven-web-demos-built-by-agents-in-one-day,maintainers-started-writing-the-rules-for-robots,no-api-to-publish-so-the-model-built-a-browser-extension,sixteen-working-viruses-designed-by-a-genome-model,the-model-installed-its-own-blender-tooling,the-web-tasks-that-stop-agents-stop-people-tooSettles a published claim. Cannot promote a capability.
- What it expected
- Every URL a benchr record cites still resolves, or is identifiably blocked to robots rather than gone.
- What happened
- 15 of 17 resolved with HTTP 200. 2 returned 403 to an automated request (www.science.org, www.theneuron.ai) — both are publishers that block robots, and both were read in a real browser when the records were written. Nothing was gone.
What it could not test
- A 200 and a matching title prove the page resolves. They do not prove the page still says what benchr quoted it for.
- Two publishers block automated requests, so their liveness is asserted from the browser session that recorded them, not from this run.
BVR-20260904-SHOWCASE-01claim checkedpending review
Subject
fifty-seven-web-demos-built-by-agents-in-one-daySettles a published claim. Cannot promote a capability.
- What it expected
- The showcase states "Every piece is one HTML file with no dependencies." A page making that claim should itself load no external script, stylesheet or image.
- What happened
- The index is a single 4,508,215-byte document with 0 external scripts, 0 external stylesheets and 0 external images. Its only href is a data: URI. The stated counts on the page are 57 demos across None collections.
What it could not test
- benchr could not reach 57 separate demo files: the demos are inside the one index document, so the per-demo half of the claim is unchecked.
- Nothing here tests who or what built the demos, or how long it took. Those remain the showcase's own account.
BVR-20260904-REPOS-01claim checkedpending review
Subject
openclaw-outgrew-the-encyclopaedia-entry-about-itSettles a published claim. Cannot promote a capability.
- What it expected
- Star counts and creation dates read from the repository API rather than from an encyclopaedia entry that was current months ago.
- What happened
- openclaw/openclaw: 388,822 stars, 81,662 forks, created 2025-11-24. anthropics/claude-code: 144,021 stars, created 2025-02-22.
What it could not test
- A star count measures attention on one day, not quality, adoption or use.
- The API is unauthenticated, so the figures are whatever GitHub served this machine at that moment.
Designed and not run
These are written out in full before they can be run, so nothing about the test can be decided once the result is known. None of them is evidence, and none of them can promote anything.
BVP-20260905-JSON-01designed, not runNo API credentials are available in this environment.
Subject guarantee-the-json-shape
- The claim
- A response constrained by a JSON schema either matches the schema or returns no object at all. It never returns a well-formed object carrying an invented enum value.
- What would count as a pass
- Across 60 calls, no response carries an enum value outside the four. The five uncovered tickets return notApplicable or no object. A response inventing a fifth category is a fail. A response returning nothing is a pass for this claim, and is recorded separately as a limitation.
- What a person has to judge
- A person reads the five uncovered tickets and confirms none of the four enum values genuinely applies. The whole test rests on that judgement, and a model deciding it would be marking its own paper.
Evidence the run must produce
- code-result: the request and response bodies for all 60 calls, credentials removed
- benchr-measurement: the count of responses per outcome, as a table
- The schema verbatim, so somebody else can send the same one
What it could not test
- One model, one API version, one schema shape. It says nothing about nested schemas or other providers.
- A refusal and a max-tokens truncation both return no object. This run cannot tell them apart without the stop reason, so it records the stop reason on every call.
BVP-20260905-CACHE-01designed, not runNo API credentials are available in this environment.
Subject cache-a-prompt-to-cut-the-bill
- The claim
- A prefix marked cacheable and left byte-identical is read from cache on the second call within the entry lifetime, and the response reports it, so the saving is checkable rather than assumed.
- What would count as a pass
- Call 1 reports cache creation above zero and cache read at zero. Calls 2 to 10 report cache read above zero. The call after ten minutes reports zero read, showing the entry expired. The control arm reports zero on both fields throughout. No claim is made about the size of any saving beyond the token counts the response itself returns.
- What a person has to judge
- A person confirms the prefix really was byte-identical between calls. A single changed character explains a zero read, and would otherwise look like the capability failing.
Evidence the run must produce
- benchr-measurement: the four cache token fields for every call, as a table
- code-result: the request builder, so the byte-identical prefix can be checked
What it could not test
- A prompt under the caching minimum caches silently and reports zeros, so the document is sized well above the documented minimum on purpose.
- This measures the mechanism, not a bill. What a cache is worth depends on a workload this run does not have.
The benchr verification protocol
Seven steps. A run is void if any of them is missing from the record.
- State the claim as a single testable sentence before running anything, so the run cannot be reinterpreted after the fact.
- Name the subject exactly: model and version, the surface or interface, and the version of that surface.
- Record the environment: operating system, network conditions, and anything that could make the result local rather than general.
- Record the inputs and the exact instruction or prompt, verbatim. A paraphrased prompt is not evidence.
- Record the settings that were in force, including any left at their defaults.
- Record the observed output and the result. Partial and fail are published exactly like a pass.
- Record the evidence and the reproduction recipe, then list the limitations of the run - starting with what it did not test.
Documentation axis
- DocsOfficially documentedThe provider's own current documentation describes this. benchr read that page on the date shown.
- DocsDocumented in partThe provider documents the mechanism but not the result, or documents it only inside stated limits.
- DocsDeprecatedThe provider has announced withdrawal, closed new access, or set a retirement date.
- DocsNeeds recheckThe documentation has not been re-read inside the recheck window. Computed, never hand-set.
- DocsNot documentedNo official source describes this. Submissions land here until one is found.
Testing axis
- TestBenchr Verifiedbenchr ran this and reproduced the result. Every run is published with its model, surface, inputs, settings, environment, output and evidence.
- TestPartly reproducedA benchr run succeeded under narrower conditions than the claim.
- TestNeeds retestThe last passing run is older than the retest window, or a dependency changed.
- TestBrokenA benchr run failed. The run is published alongside the record.
- TestNot tested by benchrbenchr has not run this. This is the honest default and it is where every record starts.