Evaluating speech systems without a leaderboard
Speech quality is a perceptual property, and perceptual properties resist the single number that comparison tables want.
The question people want answered is which system sounds better, and the honest answer is that it depends on which failure the listener is sensitive to. Speech quality decomposes into at least intelligibility, naturalness, speaker similarity and appropriateness of expression, and a system can be strong on one while weak on another.
That is not a reason to avoid measurement. It is a reason to measure the axes separately and to be sceptical of any protocol that collapses them.
Subjective protocols are the ground truth
Listening tests remain the reference method because the property being measured is perceptual. The standard design asks listeners to rate samples on a scale, usually of five points, and averages the result. The value of the resulting number depends almost entirely on protocol discipline: samples randomised, the same text across conditions, listeners screened for headphone use and attention, and enough listeners that the confidence interval is worth quoting.
Comparative designs, where a listener hears two samples and picks one, are generally more sensitive than absolute rating for small differences, because they remove the listener's personal calibration of the scale.
Any single number for speech quality is a compression of several unrelated failures into one axis.
Objective measures are triage
Objective measures are fast, cheap and only loosely correlated with perception, which makes them excellent for catching regressions and poor for ranking good systems against each other. Running a speech recogniser over generated audio and measuring word error against the input text catches gross intelligibility failures and pronunciation collapse. Speaker embedding similarity against the reference audio catches identity drift.
Both are useful as gates in a build pipeline: they answer the question of whether something broke, not the question of whether something is good.
Build a fixed listening set
The most useful evaluation asset a project can own is a small, fixed set of test texts chosen to stress specific behaviours: a long compound sentence, a passage with unusual proper nouns, a question followed by a short answer, a numeric list, an exchange between two speakers, and a paragraph long enough to expose identity drift. Twenty items is enough.
Because the set is fixed, results are comparable across versions, and because it was designed around known failure modes, a regression shows up as a specific item getting worse rather than as a slightly lower average.
Report what was measured
The reporting standard that makes results reusable is unglamorous: state the protocol, the number of listeners, the comparison condition and the text set alongside the number. A score without that context cannot be compared with any other score, which is the usual reason comparison tables disagree with each other and with the reader's own ears.