How we measure
See what our tests cover before choosing a model.
Current measured coverage
Fresh passing files by exact hardware class. A file may appear in more than one class. Finder checks current file availability separately.
- M4 Mac 16 GB: 44 files
- Google Cloud E2 8 GB: 2 files
- Google Cloud E2 16 GB: 2 files
- Google Cloud E2 32 GB: 1 file
- Google Cloud C3 16 GB: 6 files
- Google Cloud C3 32 GB: 4 files
- Google Cloud G2 · NVIDIA L4 24 GB: 6 files
Snapshot:
What we test
We test one exact model file, runner version and hardware class at a time. A passing run does not promise the same result on your computer or establish answer quality.
What a signature means
Each signed result records the file fingerprint (SHA-256), command, software version, hardware, test time and transcript fingerprint. The signature identifies the lab signer; it does not guarantee future execution.
Time, speed and memory
Time to first token (TTFT) measures the wait for the first piece of an answer. For request timing, the server is already ready. Download, installation and server startup time are excluded. Runner-reported timing can use a different start point. Answer speed is measured in tokens, or pieces of text, per second. Peak process memory (RSS) may exclude other processes. Each run includes its timing and memory scope.
The homepage number
The homepage needs at least 30 observations across three distinct files before it shows the median from the latest passing, fully measured run for each file, runner and hardware class in the last 24 hours. Only request-to-first-content timing after server readiness counts. Each run has equal weight. A newer failure removes the older pass from this sample. The number describes our lab sample, not every user or model.
Coverage and limits
Coverage is partial. The nightly scheduler works within a budget, so not every file or hardware class is retested each night. A total of files ever tested is not a count of current passing results. E2 and C3 are cloud CPU classes. G2 uses an NVIDIA L4; it does not establish RTX or Mac compatibility.
Why finder may show no match
Proven-only finder needs a currently hosted file with the same path, fingerprint and size, the same tested hardware class, and a fully measured passing run less than 24 hours old at 4096 context tokens. That context is the amount of text the model can work with at once. Unknown hardware, missing measurements or stale results produce no tested recommendation.