Agent evaluations
Observed first-call success, tool choice, tokens and latency from real agent clients.
Scripted tasks through Claude's MCP connector and Codex CLI. Plain JSON-RPC checks transport and response contracts separately.
Status: measured. Six fixed tasks per client. Failed and unavailable attempts are retained. Internal client HTTP retries are not instrumented; the three-tool prompt limit is not a hard provider-side cap.
Report source: public-gcs. This report is over 36 hours old or has no recorded run time.
Last run: 2026-11-09T23:30:10.848615+00:00. Target: https://huggingbay.xyz/api/mcp?profile=core.
| Client | Coverage | Attempted / planned | First call | Tool choice | Task success | Total tokens | Median latency (ms) |
|---|---|---|---|---|---|---|---|
| claude | agent | 6/6 (0 unavailable) | 5/6 | 4/6 | 2/6 | 31248 | 5846 |
| codex | agent | 6/6 (0 unavailable) | 4/6 | 2/6 | 1/6 | 551773 | 19620 |
| jsonrpc | transport | 6/6 (0 unavailable) | 4/6 | Not applicable | 4/6 | Not reported | 824 |
First call means the first tool result succeeded without a retry. Tool choice grades the first tool against the task rubric. Task success checks call arguments, bounded result fields and client completion, including expected terminal errors. It does not grade the wording of the final answer. Unavailable clients and absent token counts remain explicit in the JSON. These observations do not prove model inference on your computer.