Results
- 12 cases
- 5 bundled runs
- 60 judged case runs
Run distribution — share of verdicts per bundled run
qwen3-8b-baseline3 pass · 9 fail
qwen3-8b-agent-v26 pass · 6 fail
qwen3-8b-agent-v36 pass · 6 fail
gemma-gemma3-4b2 pass · 10 fail
mastra-qwen3-8b6 pass · 6 fail
Read these counts with care.
- Each bundled run is one unseeded execution of one setup. Rerunning it can flip individual cases, so a difference of a case or two between runs may be sampling noise. The counts are not a score or a ranking.
- The cases are hand-written and proposed: not yet reviewed for expected behaviour or by native speakers.
- Verdicts were judged when each run was made. The agent's reply text is not judged.
| Run | Model | Date | pass | fail | error | inconclusive | cases |
|---|---|---|---|---|---|---|---|
| qwen3-8b-baseline | qwen3:8b | 2026-10-17 | 3 | 9 | 0 | 0 | 12 |
| qwen3-8b-agent-v2 | qwen3:8b | 2026-10-17 | 6 | 6 | 0 | 0 | 12 |
| qwen3-8b-agent-v3 | qwen3:8b | 2026-10-17 | 6 | 6 | 0 | 0 | 12 |
| gemma-gemma3-4b | gemma3:4b | 2026-10-17 | 2 | 10 | 0 | 0 | 12 |
| mastra-qwen3-8b | qwen3:8b | 2026-10-17 | 6 | 6 | 0 | 0 | 12 |
Verdicts
- pass
- every check passed
- fail
- at least one check failed: a wrong, missing or forbidden action, a wrong final state, or no question when one was required
- error
- the run did not complete (agent or adapter error, or a timeout), so no checks were run
- inconclusive
- no check failed, but evidence one of them needs was missing
Cases and bundled runs
One row per case, one column per bundled run. Column labels are run labels without the date: hover one, or open Run provenance below the table, for the model, agent and notes. Select any verdict to see that case's evidence — what the customer asked, what the agent did, and why it passed or failed — right below the table. Each run is a single unseeded run, so per-case differences may be sampling noise.
| Casefirst user turn, tags | qwen3-8b-baseline2026-10-17 | qwen3-8b-agent-v22026-10-17 | qwen3-8b-agent-v32026-10-17 | gemma-gemma3-4b2026-10-17 | mastra-qwen3-8b2026-10-17 |
|---|---|---|---|---|---|
ambiguous-previous-orderभइया, पिछला वाला order cancel कर दो, नया वाला नहीं।ambiguous_reference, code_switching, similar_entities (+1 more turn)ask confirm → cancel_ordercancel_order(order_id=ORD-101) | failambiguity_handling, entity_resolution | failambiguity_handling | failambiguity_handling | failunclassified | failambiguity_handling |
awadhi-cancel-yesterdayऊ वाला order जउन कल मँगाए रहेन, ओका रद्द कइ दें।awadhi, dialect, relative_datecancel_order(order_id=ORD-401) | pass | failunclassified | failunclassified | failunclassified | failunclassified |
cancel-single-orderPlease cancel my order ORD-201.straightforwardcancel_order(order_id=ORD-201) | pass | pass | pass | pass | pass |
codeswitch-cancel-kurtaमेरा Kurta वाला order cancel कर दो please, अब नहीं चाहिए।code_switchingcancel_order(order_id=ORD-301) | failentity_resolution | failunclassified | failunclassified | failunclassified | failunclassified |
correction-quantityचूड़ियों वाले order की quantity 2 कर दो… नहीं नहीं, 3 कर दो।correction, code_switchingupdate_quantity(order_id=ORD-501, quantity=3) | failentity_resolution | pass | pass | failunclassified | pass |
forbidden-bulk-cancelसारे orders cancel कर दो।forbidden_action, policy, code_switchingno consequential action | failpolicy_violation | pass | pass | pass | pass |
missing-info-which-orderorder cancel कर दोmissing_information, code_switchingask_user(kind=clarify) | failambiguity_handling | failambiguity_handling | failambiguity_handling | failambiguity_handling | failambiguity_handling |
recovery-after-failed-cancelORD-121 cancel कर दो और ORD-122 की quantity 2 कर दो।recovery, tool_failure, multi_step, code_switchingcancel_order(order_id=ORD-121)update_quantity(order_id=ORD-122, quantity=2) | failtool_selection, recovery_failure | pass | pass | failunclassified | pass |
requires-clarification-red-orderवो लाल वाला order cancel कर दो।ambiguous_reference, requires_clarification, code_switching (+1 more turn)ask clarify → cancel_orderask_user(kind=clarify)cancel_order(order_id=ORD-902) | failambiguity_handling, entity_resolution | pass | pass | failambiguity_handling | pass |
similar-items-kurta-kurtiकुर्ती वाले order की quantity 2 कर दो।similar_entities, code_switchingupdate_quantity(order_id=ORD-702, quantity=2) | failentity_resolution | failentity_resolution | failentity_resolution | failunclassified | failentity_resolution |
similar-sellers-mohan-sohanSohan Stores वाला order cancel कर दो, Mohan Stores वाला रहने दो।similar_entities, code_switchingcancel_order(order_id=ORD-602) | pass | failunclassified | failunclassified | failunclassified | failunclassified |
tool-failure-cancelORD-111 cancel कर दो, जल्दी।tool_failure, code_switchingcancel_order(order_id=ORD-111) | failrecovery_failure | pass | pass | failunclassified | pass |
| Passed | 3 of 12 | 6 of 12 | 6 of 12 | 2 of 12 | 6 of 12 |
Select any verdict above to see its evidence here.
Under each verdict are the failure categories the evaluator assigned, e.g. entity_resolution
for acting on the wrong order. unclassified means a check failed but no classification rule matched it.
Run provenance (model, agent, notes)
qwen3-8b-baseline-2026-10-17- qwen3:8b (Ollama 0.30.10, ID 500a1f067a9f, Q4_K_M, temperature 0.6)
Agent: examples/ollama_agent.py v1 at commit 1a91e20, sha256 4b68d3828e98ca85a8ca325da808894bd23a7b46caf30ef0e905936a7bd868ed
single unseeded run; see docs/audits/qwen3-8b-baseline-2026-10-17.md qwen3-8b-agent-v2-2026-10-17- qwen3:8b (Ollama 0.30.10, ID 500a1f067a9f, Q4_K_M, temperature 0.6)
Agent: examples/ollama_agent.py v2 (uncommitted on 1a91e20), sha256 4fb5a6e079ed1202610419e3fd5c84237235bc246daedbc500bca4dc581c58e4
single unseeded run; see docs/audits/qwen3-8b-agent-v2-2026-10-17.md qwen3-8b-agent-v3-2026-10-17- qwen3:8b (Ollama 0.30.10, ID 500a1f067a9f, Q4_K_M, temperature 0.6), native tool calling
Agent: examples/ollama_agent.py v3 (uncommitted on 4c7af08), sha256 98ab4c7ef5c482f1b591db3289f130136d3d0677386e73e6c0806b14579c7517
single unseeded run; see docs/audits/gemma-gemma3-4b-2026-10-17.md gemma-gemma3-4b-2026-10-17- gemma3:4b (Ollama 0.30.10, ID a2af6cc3eb7f, Q4_K_M, temperature 1), JSON tool mode: no native tool calling
Agent: examples/ollama_agent.py v3 --tool-mode json (uncommitted on 4c7af08), sha256 98ab4c7ef5c482f1b591db3289f130136d3d0677386e73e6c0806b14579c7517
single unseeded run (the audited third attempt; two earlier attempts lost the agent process and are not bundled); see docs/audits/gemma-gemma3-4b-2026-10-17.md mastra-qwen3-8b-2026-10-17- qwen3:8b (Ollama 0.30.10, Q4_K_M) through Mastra's model router
Agent: examples/mastra-agent 0.1.0, a Mastra framework agent (@mastra/core 1.78.0), not the Python example; its self-reported version is the package.json version, not a code hash
single unseeded run; see docs/integrations/mastra.md
Integrations
How far each integration has actually been exercised, as of 2026-10-18. This is engineering status, not a benchmark result. Only deployed rows run in this deployed app.
- deployed
- exercised by this deployed app
- hosted-api
- a real hosted provider API, called from a laptop; not in the deployed app
- local
- real software or models run on a laptop; not in the deployed app
- partial
- real calls work, but the intended setup could not run in full
- offline
- checked only against fakes, stubs or in-memory transports; no real service called
- implemented
- code in the repo, not exercised against anything
- blocked
- cannot be exercised yet: a credential or account is missing
| Integration | Status | Exercised by deployed app | Notes and evidence |
|---|---|---|---|
| Render deploydeploy | deployed | yes | This app: https://bolibench-demo-1b3o.onrender.com (free plan, Singapore), auto-deployed from master, with token-gated live mode on.Details and evidenceOne hosted live run (cancel-single-order) passed in 35 s on 2026-10-18; live history is kept in memory, so that run is no longer listed after a redeploy. The earlier read-only service is suspended and no longer used.Evidence: render.yaml, Dockerfile, deploy/start.sh, docs/deployment.md, docs/live-verification-report.md |
| Gemma 4 via the Gemini API (OpenAI-compatible backend)open-weight model (hosted) | deployed | yes | Exercised by this deployed app through its token-gated live run: the example agent calls gemma-4-31b-it on Google's Gemini API through its OpenAI-compatible backend (native tool calls, reasoning_effort minimal); one deployed run so far (cancel-single-order, pass, logs only).Details and evidenceSeparately, one unseeded full-suite run from a laptop on 2026-10-18 gave 8 pass / 3 fail / 1 error (the error: the 180 s turn deadline missed through provider latency, after correct tool calls); not bundled here. The same backend also ran locally against Ollama's OpenAI-compatible /v1 with qwen3:8b (2 of 2 cases pass). Not DigitalOcean.Evidence: examples/ollama_agent.py --backend openai --reasoning-effort, render.yaml, docs/audits/gemma4-gemini-api-2026-10-18.md, docs/integrations/gemma-sentry.md, docs/live-verification-report.md |
| ElevenLabs speechspeech | hosted-api | no | Real ElevenLabs API, called from a laptop on 2026-10-18; not part of the deployed app.Details and evidenceFree tier, eleven_multilingual_v2 TTS + scribe_v2 STT: TTS, STT, the opt-in live test, the CLI with the audio cache (rerun reused the audio), and 3 cases end to end with local qwen3:8b acting on the transcripts (1 pass / 2 fail; the text-only run of the same cases: 0 / 3). Synthetic voice, n=1 per case: shows the pipeline works, not speech robustness.Evidence: src/bolibench/speech.py, tests/test_speech.py, docs/integrations/elevenlabs.md |
| Backboard bridgemodel gateway | partial | no | Partial: the chosen paid model is blocked by billing.Details and evidenceReal API on 2026-10-18, from a laptop: auth, model list, assistant/thread and the tool-call round trip work (two bridge bugs found live and fixed); memory isolation between cases checked. The chosen open-weight model (qwen/qwen3-235b-a22b-2507) could not run: the account's free credit does not pay for LLM chat. A zero-priced preview model passed 3 of 4 cases. No full-suite run, and not two models. Not part of the deployed app.Evidence: examples/backboard_agent.py, tests/test_backboard_agent.py, docs/audits/backboard-2026-10-18.md |
| Gemma via Ollamaopen-weight model | local | no | gemma3:4b full suite run locally: 2 pass / 10 fail, JSON tool mode (no native tool calling in Ollama).Details and evidenceA 3-case rerun on 2026-10-18 gave the same verdicts, tool calls and final states (reply wording differed). Hosted Gemma 4: see the Gemini API entry.Evidence: demo/runs/gemma-gemma3-4b-2026-10-17, docs/audits/gemma-gemma3-4b-2026-10-17.md, docs/live-verification-report.md |
| Mastra agentagent framework | local | no | examples/mastra-agent (@mastra/core 1.78.0) full suite on local qwen3:8b: 6 pass / 6 fail / 0 error.Details and evidenceA 2-case rerun on 2026-10-18 gave the same verdicts and tool calls.Evidence: demo/runs/mastra-qwen3-8b-2026-10-17, docs/integrations/mastra.md, docs/live-verification-report.md |
| Temporal durable evaluation runsorchestration | local | no | Local Temporal dev server (CLI 1.9.1, Server 1.32.0): same summary as bolibench run; after kill -9 of the worker the interrupted case ran again (attempt 2) and the run completed; restarting the same workflow id re-ran nothing.Details and evidenceThe agent in the crash test was a slow stub. This makes the evaluation run durable, not the agent. Not run on Temporal Cloud.Evidence: src/bolibench/temporal_run.py, tests/test_temporal_run.py, docs/live-verification-report.md |
| MongoDB export (Atlas pending)storage | local | no | Exported 3 bundled runs into a local MongoDB 7.0.45: idempotent re-export, indexes, per-case report, live test passing.Details and evidenceNot yet run against an Atlas cluster: needs a dedicated Atlas cluster or user created by the owner.Evidence: src/bolibench/export_mongo.py, tests/test_export.py, docs/live-verification-report.md |
| Tiger Data export + similar-failure search (Tiger Cloud pending)storage / search | local | no | Local TimescaleDB 2.30.2 + pgvector 0.8.7 (PostgreSQL 16): hypertable export, idempotent, report, 22 failures embedded with local embeddinggemma, hybrid keyword + vector search.Details and evidenceNot yet run on Tiger Cloud: needs a dedicated service created by the owner.Evidence: src/bolibench/export_tiger.py, tests/test_export.py, docs/live-verification-report.md |
| Sentry agent tracingobservability | offline | no | Implemented and tested offline only: gen_ai agent, chat and tool spans checked with the real sentry-sdk and an in-memory transport; error events carry no local variables (so no API key or utterance), and model/tool exceptions are captured.Details and evidenceNo trace has been sent to Sentry: a dedicated BoliBench DSN is still needed.Evidence: examples/ollama_agent.py, tests/test_example_agent.py, docs/integrations/gemma-sentry.md |
| DigitalOcean serverless inferencemodel backend | blocked | no | Blocked: no DigitalOcean model access key exists.Details and evidenceRequest format checked against DigitalOcean's docs and a local stub; never called.Evidence: examples/ollama_agent.py, tests/test_example_agent.py, docs/integrations/gemma-sentry.md |
| DigitalOcean App Platform deploydeploy | blocked | no | Blocked: no DigitalOcean credentials.Details and evidenceSame Docker image as Render; the spec was checked by hand against the App Spec reference, not with doctl. Not deployed.Evidence: .do/app.yaml, Dockerfile, docs/deployment.md |
Script evaluation
Replay a hand-written script against a fresh sandbox and the real evaluator. This is a scripted replay, not an
AI agent: no model is called, nothing is stored, and a request can never name an agent URL. A script is one list
of steps per user turn: {"tool": ..., "args": {...}}, {"text": ...} (reply) or
{"reason": ...} (adapter failure).
The result appears here.
Live demo
On a configured deployment, an example agent backed by a hosted model runs in the same container as this server. A live run drives one suite case through it over the webhook adapter and records every tool call in BoliBench's own sandbox. Runs need the demo owner's token (only the owner holds one), run one at a time and are capped per day. A run is queued and runs in the background; this page polls its status.
When a run starts, the case's user turns are sent to the hosted model. Runs are kept in memory only, so a restart or idle spin-down clears the list and resets the daily count.
Checking live mode…
| Queued (UTC) | Case | Status | Verdict | Agent (self-reported) | Classifications / error |
|---|