Voice AI Evaluation Harness
Measure the contract, not the vendor
A one-day harness for choosing a voice provider, and the three things it told us that no leaderboard could.
12 October 2026. Every figure below is from runs made between 5 and 9 October 2026; vendors change their models and prices under you, so read the numbers as a dated snapshot and the method as the point.
The app we work on has a microphone button. A user asks it, in one breath, to add two items to a list and set a reminder for tomorrow; the app turns that into two entries and a task, and reads back one short confirmation. It works in nine languages, mostly Hebrew and English, and people mix them in the same sentence. In early October we had to move it off the provider it had been built on, for a contractual reason that had nothing to do with quality. We had a week, one developer, and no appetite for building three integrations to find out which one to keep.
This is the harness we built instead, what it found, and the three things it told us that no leaderboard could have.
Why the leaderboards did not help
The two big independent speech-to-text leaderboards are English-only by their own methodology. The one independent Hebrew leaderboard, run by the ivrit.ai non-profit, measures transcription on podcasts, read sentences and news, not on a short command with a day and a time in it, spoken into a phone in a noisy room. Every other Hebrew number on the internet is a vendor's own. And our feature is not transcription: it is a contract. The server sends the recording, a system instruction, four function declarations and a forced function call, and gets back exactly one call — add entries, complete an entry, speak a summary, or decline — whose first argument is the language the person spoke. A provider can transcribe Hebrew beautifully and still put an appointment on the wrong day, attach it to the wrong person, or answer an Arabic question in Hebrew.
So the question was never "which provider is best". It was "which provider does our contract at least as well as today's", and that is measurable in a day if you refuse to build anything inside the app.
The one rule: the harness sends what production sends
The usual provider bake-off measures the prompts, not the providers. Each candidate gets a prompt someone wrote that morning, slightly different from the others and all of them different from what ships, and the comparison reports the differences between the prompts.
Our server's request builder is a pure function: it takes the user's data and the audio and returns the request body. The harness's first script is ten lines of JavaScript that import that function from the server's TypeScript, feed it a frozen test dataset and a placeholder audio part, and write the result to a file. Every adapter in the harness starts from that file. The control — the provider we were leaving — gets it byte for byte, which a hash proves. Audio-native candidates get the same body with the host changed. Transcriber-plus-text-model candidates get the same system instruction and the same four declarations translated mechanically into the other vendor's tool schema (the JSON Schema is identical; one vendor-specific format field is dropped), with the audio part replaced by one sentence: "The user said (transcribed; the transcriber heard language he): «…»". Nothing is rephrased, nothing is "improved for this vendor". When the swap was built, the same script read the new server code live and the re-run measured the request the server actually makes.
The control matters more than it looks. Today's provider runs through the same harness, on the same clips, and its score defines "good enough". It also loses on things, which is useful to know before you leave it: ours answered five of ten clips in the seven less common languages in the wrong language, and once invited a person who does not exist — the example name from its own prompt's transliteration rule, which the model had read as a person. Both had been in production for weeks. Nobody had measured.
Containment: a harness you can delete without a trace
Keep the harness out of the product. Ours is one folder at the repository root, outside the app's and the server's source, with its own ignore file for recordings, results, the virtual environment and the generated request template, which is regenerated from source rather than committed so it cannot drift. Keys live in the shell's environment and in no file at all. The test data is invented — people, tasks, events, a list of items — because several vendors keep uploads by default and the clips go to all of them.
Three gates in the repository will catch you anyway, and it is cheaper to know in advance. The TypeScript compiler, inheriting allowJs from the framework's base config and given no include list, compiled our one .mjs script and then a build check noticed that the file would not exist on the build server. The linter walks the whole tree. The build tool's own ignore file, not .gitignore, decides what is uploaded. The rule that came out of it: a top-level folder that is not the app joins all three ignore lists in one commit, with a comment saying why. Create the vendor trial accounts yourself, write down what you created, and revoke every key the day you are done.
The test set, and the two things that went wrong with it
The manifest is 105 lines: 45 Hebrew, 35 English, 10 code-switched, 10 across the other seven languages, and 5 edge cases — silence, a one-word clip, a 25-second ramble, two people talking at once, a television in the background. Across them, every shape the contract takes: items with quantities and a category, tasks with an assignee and a relative date, events with a day, a time and a duration, mixed requests that must land in one call, a task finished by part of its title, a question about the user's day, and ten lines that must be declined — the weather, a joke, a maths question, and three attempts to change the model's instructions. Each line carries the verbatim transcript, the expected function and the slots a user would accept, against one frozen "today" so that "next Thursday" resolves the same for every provider. Half a day of one person's time, and the asset outlives the exercise.
Then we took a shortcut, and it is the most useful part of this article.
Text-to-speech test sets mislead, unevenly. Recording 105 clips takes a day; voicing them with two vendors' text-to-speech takes an hour, so we did that first. Three things happened. One of the candidates was also one of the TTS vendors, and it scored 93% on clips spoken by its own voices and 76% on the other vendor's — the largest gap in the whole study. One TTS voice refused to read a harassment line from the safety set and read its apology instead; the clip measured nothing, and nothing in the pipeline noticed until a human listened. And on the ten code-switched lines — the hard ones, Hebrew sentences with English brand names and English sentences with Hebrew names — the two leading candidates scored 9 and 7 on the TTS clips and then swapped places, 8 and 9, the moment fifteen real recordings replaced them. The rule we now keep: TTS measures the contract, real voices rank the candidates, and a generated set is transcribed once and read before anyone trusts it.
Noise is larger than most of the differences people argue about. The same text model, given the same transcripts two days apart, changed four verdicts in 95. One transcriber gave identical transcripts across two runs; another changed four of 95. With ten code-switched clips, one clip is ten points. So everything runs twice, the flip rate is reported as a result, and any difference smaller than it is weather.
What to score, in the order it decides
The scorecard is gates first — any miss removes the candidate — then a ranking. The gates are the contract's promises: the language identified correctly on clean clips and on every code-switched one; the right function name; every off-topic and override line declined and no real command declined; the spoken line written entirely in the declared language's script, which is a Unicode-block test, not a model's opinion. The ranking is what a user would notice: slot accuracy on the hard language, then content-word recall, then flip rate, latency at the 95th percentile end to end, cost per thousand commands from the vendors' usage fields, and the engineering cost of the swap read off the adapter.
Two of those deserve a word. Content-word recall — the share of the manifest's item names, people, days and numbers that survive in the transcript — is a more honest number for this kind of app than word error rate. A Hebrew transcript can have a 20% WER on prefixes and particles and still carry every word the app needs; and one transcriber wrote "2" where the speaker said "שתי", which WER counts as an error and recall counts as right. And "all slots correct" needs a second axis: whether a miss wrote something wrong. We will come back to that.
The gate table for the first full round, 5 October, with the control on the left: the text-to-speech set, 105 lines, two runs each.
Hebrew: two candidates matched or beat the control. Code-switched speech: every candidate lost to it. That gap is the next section.
Three findings we would not have had without the harness
Names and loanwords are the whole code-switching problem, and the vendors already have the fix. We pulled every miss on the English and code-switched lines, and almost all of them were the same failure: a proper noun reaching the text model already wrong. "Ido" became Idaho and Ito, "Gil" became Jill and Deal's, English brand names inside Hebrew sentences came through as words that mean nothing, and the text model, finding nobody called Idaho in the user's data, assigned the task to nobody. Given correct transcripts, the text model got all ten code-switched lines right; the loss was in transcription, and no prompt downstream can mend it.
Every transcriber has a vocabulary feature for exactly this — a context list, a keywords field, a phrase list — and no bake-off we have read uses one. We primed each with the four names in the test data, eight terms in two scripts, and separately with a 38-word lexicon of brand names and everyday nouns. On the 95 Hebrew, English and code-switched clips:
The names fixed every misspelt name that had cost a clip, on every transcriber, inserted none, and cost nothing — the names already travel to the provider in the user's data. The lexicon closed the loanword gap too, to 10 of 10 code-switched on one candidate, and then invented two brands: two ordinary nouns came back as two brand names that were in the lexicon and not in the audio, and the pipeline put both on the user's list. A list that contains the very words in your test set is a ceiling, not a result. Prime with names; be careful with everything else.
The async trap, which only a whole-pipeline clock shows. The most accurate transcriber on Hebrew lost on wall-clock, and not because of the model. Its batch API is upload, create, poll: a flat two to three seconds whatever the clip's length, so a five-second command waited on the job rather than on the audio. Its streaming API was slower still for a finished file — it works through a recording at about twice real time plus a connection — and only wins if the app streams while the person is still speaking, which is a different product. Model latency tables do not contain any of this. Time the pipeline end to end, from where your server will stand, and split the asynchronous steps so you can see which one you are paying for.
The wrong-write failure, which "all correct" cannot see. One line in the set is "Mark [a Hebrew task title] as done". Every transcriber heard the English "Mark" as a Hebrew noun that sounds the same. From that transcript one pipeline completed the right task and replied in the wrong language; another wrote wrong data — that noun added to the user's list three times, a duplicate of the task once — in four of its six real-voice runs. A scorer that counts "all slots correct" sees two equal misses. One that classifies a wrong write separately — something added or changed that the person did not ask for — sees the difference a user would see, and that classification decided the guard we shipped: a repeated task fails the whole command out loud rather than quietly adding to the list.
A fourth, briefly, because it belongs to a later article on safety: when we added a vendor's inline moderation to the text call, planting one flaggable sentence in the system message flagged the request, while the same Hebrew harm lines in the user turn flagged nothing. The inline input side was scoring the whole prompt, the user's data included, and Hebrew drowned in 2,000 tokens of English instructions. Moderating the transcript alone flagged the same Hebrew lines at 0.65–0.80. You find that with a clean control and one planted sentence. You do not find it in the documentation.
What the harness cannot tell you
Latency from a laptop is not latency from the edge; the order between providers held, the seconds did not. Rate limits and daily caps do not show at two hundred calls — read the account's own limits page, not the docs page, before you deploy. Prices are the day's; one vendor's pricing page disagreed with its own model page by a factor of two during the week. The vendors' terms, which started this, are not something a benchmark reads; a person does, at the source, with the date. And three of the models in our tables were marked deprecated on their vendors' pages within the same week — which is the argument for the last section.
The swap, and the harness as a regression suite
We chose on speed, because accuracy was within noise: on 41 real recordings by two speakers, run twice, the control scored 79 of 82, the chosen pipeline 77 and the runner-up 76. Medians from the laptop were 2.9–3.1 s against 5.2–5.3 s for the runner-up and 2.1–2.2 s for the control. The adapter was built in half a day. Then the harness read the new server code live and ran the server's own request against a bar written in advance: no loss against the primed results, the two known failures fixed, no wrong write on the "Mark … as done" line, the invented guest gone. It found one regression — a rule added to the production prompt after the evaluation branch was cut, which the new text model read differently — and that was fixed and re-measured before anything deployed.
The harness is now the thing we run when a vendor deprecates a model, when a line in the prompt changes, or when the next contractual surprise arrives. The whole exercise cost half a day for the test set, an hour in the vendors' consoles, one to two developer-days of scripting, and less than ten dollars of API spend. Nothing in the app changed.
The harness is published as voice-provider-harness under the MIT licence: the dump script, the runner with adapters for the providers we tested, the scorer with its gates and ranking, the normaliser, a manifest template with example lines, the TTS helper and the containment rules from the section above. The examples use an invented domain and invented people; the repository contains no keys, no recordings of anyone, and none of our production data, and run.py --check will tell you which adapters your own keys unlock before you spend anything. The numbers in this article will not reproduce exactly — vendors have moved on since October 2026 — but the method will, and that is what the repository is for.
Figures are from runs made 5–9 October 2026. Vendor names belong to their owners; nothing here is an endorsement. Methods, not numbers, are the part that keeps.
Comments