TORNEO
Menu

Case studytranscriptionobserved

Faithfully transcribe audio into text (speech-to-text via API)

Operated by Tempo AI LLC, accountable for the calculation, publication and correction of every figure published here. No vendor pays to be measured; TORNEO buys what it tests.

done · Evidence coverage: OK. This is not an execution-progress indicator.

Compare measurementsShare this dated resultExport canonical JSON, including limits

What we bought

gemini-3-5-transcribe
Google · total cost USD 0.043
gemini-2-5-flash
Google · total cost USD 0.046
gemini-2-5-pro
Google · total cost USD 0.21
gemini-3-5-flash
Google · total cost USD 0.28

What we asked

Task, as frozen in the protocol
Batch transcription of FR/DE/EN clips of 5 to 30 s, language hint provided, plain-text output
Protocol
version 1.0, lock 7056f0e2

What happened

Replay of a recorded result

  1. gemini-3-5-transcribe0.520.420.62
  2. gemini-2-5-flash0.570.460.66
  3. gemini-2-5-pro0.610.510.71
  4. gemini-3-5-flash0.640.540.74

Measured with intervals across 2 replicated blocks; the intervals overlap, no tool is ranked first.

Exact intervals
Interval bands, transcription, observed 2026-09-02T18:42:22.745775Z: gemini-3-5-transcribe interval from 0.420246 to 0.622379, estimate 0.522222; gemini-2-5-flash interval from 0.463641 to 0.664234, estimate 0.566667; gemini-2-5-pro interval from 0.507825 to 0.705301, estimate 0.611111; gemini-3-5-flash interval from 0.541502 to 0.735561, estimate 0.644444.overlapgemini-3-5-transcribe0.42 to 0.62gemini-2-5-flash0.46 to 0.66gemini-2-5-pro0.51 to 0.71gemini-3-5-flash0.54 to 0.740.30.40.50.60.70.8Interval bands, transcription, observed 2026-09-02T18:42:22.745775Z: gemini-3-5-transcribe interval from 0.420246 to 0.622379, estimate 0.522222; gemini-2-5-flash interval from 0.463641 to 0.664234, estimate 0.566667; gemini-2-5-pro interval from 0.507825 to 0.705301, estimate 0.611111; gemini-3-5-flash interval from 0.541502 to 0.735561, estimate 0.644444.overlapgemini-3-5-transcribe0.42 to 0.62gemini-2-5-flash0.46 to 0.66gemini-2-5-pro0.51 to 0.71gemini-3-5-flash0.54 to 0.740.30.40.50.60.70.8
Share of recordings transcribed with no error at all, estimate and interval per tool; the tinted zone is where every interval overlaps. Population n = 90, 2 replicated blocks, observed , valid until .

This is our experience, on our corpus, at this date. It is not buying advice.

Do not name a winner: the confidence intervals overlap. Report the range, the n and the observation date.

Per-tool figures
ToolEstimateShare of recordings transcribed with no error at allnCost per unit (USD)Latency p50 (ms)
gemini-3-5-transcribe0.520.42 to 0.62900.000931500
gemini-2-5-flash0.570.46 to 0.66900.000901500
gemini-2-5-pro0.610.51 to 0.71900.00383400
gemini-3-5-flash0.640.54 to 0.74900.00482300

4 participants measured with intervals across 2 replicated blocks, intervals overlap, no tool ranked first (as of 2026-09-02)

Scope, reply and reproduction

What we did not test, and why

Not specified in this bundle.

Legal preflight, verdict per tool

Two verdicts per tool, read from the archived terms before the first call: may we run the test, may we publish the result. A tool that was not run keeps its recorded reason. The clause notes stay in the served file (never quoted on a page): LEGAL_PREFLIGHT.json.

ToolVerdictNot run becauseReviewedSource
gemini-2-5-flashPASSranlegal/gemini-2-5-flash/
gemini-2-5-proPASSranlegal/gemini-2-5-pro/
gemini-3-5-flashPASSranlegal/gemini-3-5-flash/
gemini-3-5-transcribePASSranlegal/gemini-3-5-transcribe/
openai-whisper-1PASSBLOCKED NO CREDITSlegal/openai-whisper/LEGAL_PREFLIGHT.json
openai-gpt-transcribePASSBLOCKED NO CREDITSlegal/openai-whisper/LEGAL_PREFLIGHT.json
openai-gpt-4o-transcribePASSBLOCKED NO CREDITSlegal/openai-whisper/LEGAL_PREFLIGHT.json
openai-gpt-4o-mini-transcribePASSBLOCKED NO CREDITSlegal/openai-whisper/LEGAL_PREFLIGHT.json
deepgramBLOCKEDBLOCKED TOSlegal/deepgram/LEGAL_PREFLIGHT.json
assemblyaiBLOCKEDBLOCKED TOSlegal/assemblyai/LEGAL_PREFLIGHT.json
elevenlabs-scribePASSBLOCKED NO CREDENTIALlegal/elevenlabs-scribe/LEGAL_PREFLIGHT.json
speechmaticsPASSBLOCKED NO CREDENTIALlegal/speechmatics/LEGAL_PREFLIGHT.json
gladiaPASSBLOCKED NO CREDENTIALlegal/gladia/LEGAL_PREFLIGHT.json

Right of reply

Every tool measured here, and every vendor named in the market state, can reply. A reply is published verbatim, dated, on a separate page, never mixed into this result.

  • gemini-2-5-flash: notified , window closes , no reply received
  • gemini-2-5-pro: notified , window closes , no reply received
  • gemini-3-5-flash: notified , window closes , no reply received
  • gemini-3-5-transcribe: notified , window closes , no reply received

How to reproduce

torneo reproduce runs/TRANSCRIPTION-001

Canonical JSON: latest result of the category, this run. Evidence path in the public repository: runs/TRANSCRIPTION-001.

Reproduction files served byte-identical to the repository: events.ndjson (hash-chained journal), PROTOCOL.lock, contest.yaml, LEGAL_PREFLIGHT.json; schema result.v1.json. Install the command line with python -m pip install ./projects/torneo (or from the repository), then torneo verify and torneo reproduce on the downloaded run directory recompute result_sha256.

Audit in five minutes: the record, the draft and the verdict, side by side, for a sample of this run.

Machine surfaces (MCP server, canonical JSON, llms.txt): For agents.

What this does not prove

Errata

Published corrections of this run, each chained to the served result hash; the earlier bytes stay in the history. Full record: ERRATA.md.

  • ERR-001, : erratum ERR-001: limits LIM-2 nommait OpenAI comme vendeur des 4 solutions alors que les 4 participants sont vendor Google; amendement A2 (contest.yaml gelé). Estimations, intervalles, rangs et statuts inchangés. (served result ac21d556 replaced by 1658e291)
  • ERR-002, : ERR-002: conflicts served as [] while the vendor of all four measured arms is a tool devlo pays per call (served result 1658e291 replaced by 082730f5)
  • ERR-003, : ERR-003: promise, task, context, dataset description and the five limits were served in French on an English-only surface; amendment A4 translates the nine frozen strings (contest.yaml), French originals preserved verbatim in PROTOCOL_AMENDMENTS.md. Estimates, intervals, ranks, status and directive unchanged. (served result 082730f5 replaced by e6377e76)
  • Swiss accents and B2B register are not covered by this run (no open corpus available; phase 2).
  • 4 solutions from a single vendor (Google): market coverage is incomplete; the other candidate providers did not run (API credits exhausted, no credential, or an anti-benchmark clause in their terms), so they were blocked, not beaten; provider by provider in categories/transcription/PROVIDERS.json and legal/.
  • tnorm.v1 does not normalize numbers.
  • Wall-clock latency depends on the runner's network.
  • Canonical metric = share of clips transcribed perfectly (tnorm.v1, an API failure counts as an unsuccessful attempt), the only form the L1 scorer recomputes; the micro WER (the metric frozen in the protocol) and its bootstrap 95% CIs remain in RUN_REPORT.json, reproducible from events.l2stub.ndjson (amendment A1).

What no TORNEO result can tell you, for every category: Limits.

Conflicts, funding and per-tool details

Conflicts and funding

Run by devlo lane L2 (agent), funding: fonds propres devlo. Operated by Tempo AI LLC, accountable for the calculation, publication and correction of every figure published here.

Conflicts declared in this bundle:

  • gemini-3-5-transcribe: devlo is a paying customer of Google, the vendor of this arm, billed per call; devlo is a user of the tool, not a vendor; same conditions and same protocol as any customer, journaled; what this run cost is in FUNDING.json (CONFLICT_REGISTER.json: USER_OF_TOOLS_PAY_PER_USE)
  • gemini-2-5-flash: devlo is a paying customer of Google, the vendor of this arm, billed per call; devlo is a user of the tool, not a vendor; same conditions and same protocol as any customer, journaled; what this run cost is in FUNDING.json (CONFLICT_REGISTER.json: USER_OF_TOOLS_PAY_PER_USE)
  • gemini-2-5-pro: devlo is a paying customer of Google, the vendor of this arm, billed per call; devlo is a user of the tool, not a vendor; same conditions and same protocol as any customer, journaled; what this run cost is in FUNDING.json (CONFLICT_REGISTER.json: USER_OF_TOOLS_PAY_PER_USE)
  • gemini-3-5-flash: devlo is a paying customer of Google, the vendor of this arm, billed per call; devlo is a user of the tool, not a vendor; same conditions and same protocol as any customer, journaled; what this run cost is in FUNDING.json (CONFLICT_REGISTER.json: USER_OF_TOOLS_PAY_PER_USE)

Details

Per tool

ToolRank rangeLatency p95 (ms)Error rateIncidentsBlocks observedAccess
gemini-3-5-transcribe1 to 419000.16142API (api_key)
gemini-2-5-flash1 to 42200002API (api_key)
gemini-2-5-pro1 to 46100002API (api_key)
gemini-3-5-flash1 to 44200002API (api_key)

What agents receive for this category: EVIDENCE.

This bundle declares no pairwise design: intervals only.

Unblinding

No alias table is published for this run: tool names are the names of the public journal. A blinded arm keeps its alias until the unblinding file exists.

Population and context

Population
FLEURS dev fr_fr/de_de/en_us, read speech, encyclopedic domain; France/Germany/USA accents (n = 90, frozen , personal data: no)
Context
FLEURS dev fr_fr/de_de/en_us, read speech, encyclopedic domain; France/Germany/USA accents; NO Swiss accents and no B2B register in this run
Status
OK, 2 replicated blocks
run
TRANSCRIPTION-001
protocol
v1.0 · lock 7056f0e2
result
32cbd00a
raw data
9e870988
observed
2026-09-02T18:42:22.745775Z
evidence
runs/TRANSCRIPTION-001