Preparing audio datasets for speech models
Most of the quality difference between two speech systems was decided during data preparation, long before either was trained.
Corpus preparation is the least discussed and most consequential stage of building a speech system. The choices made here are inherited wholesale: a corpus assembled from clean read speech produces a model that reads, and a corpus with inconsistent loudness produces a model that has learned loudness is random.
What follows is an ordering that tends to cause the fewest rewrites, because each step depends on the one before it.
1. Decide the segment boundary before anything else
The unit of training is a segment, and the definition of a segment determines what the model can learn about structure. Segments cut at silences of a fixed length will cut mid-clause whenever a speaker pauses to think, teaching the model that clauses end there. Segments cut at punctuation in the transcript preserve syntactic units but require the transcript to be trustworthy first.
For dialogue material, the boundary question also decides whether turn-taking survives at all. Cutting each speaker into isolated segments discards exactly the timing information that makes conversation sound like conversation.
A corpus is a set of decisions about what counts as an example, and every one of them is inherited by the model.
2. Normalise loudness, not peaks
Peak normalisation makes the loudest sample in each file identical and leaves the perceived loudness all over the place, because perceived loudness depends on the whole signal rather than its maximum. Integrated loudness normalisation across the corpus is the version that helps, and it should be applied per segment only if the segments are long enough that the measurement is stable.
Resample once, to a single rate, and record the rate in the corpus manifest. Mixed sampling rates that are silently resampled at load time are a recurring source of quiet quality loss.
3. Align transcripts and then check the alignment
Forced alignment produces time boundaries for words given audio and a transcript. It also produces confident nonsense when the transcript is wrong, and the wrong transcripts are exactly the ones nobody spot-checks. Sorting segments by alignment confidence and reviewing the worst few hundred by ear is the highest return activity in the whole pipeline.
Decide explicitly how to represent non-verbal events. Breaths, laughter and hesitation sounds either get transcript tokens or they get discarded, and the choice determines whether the model can ever produce them deliberately.
4. Label speakers conservatively
Automatic speaker labelling is reliable within a recording and unreliable across recordings. Treating each recording session as its own speaker identity, and only merging identities where there is external evidence, avoids the failure where two people are trained as one voice and the model learns to average them.
5. Keep a manifest and keep it honest
A manifest that records, for every segment, its source recording, time offsets, speaker identity, sampling rate and transcript provenance makes every later question answerable. Without one, the only way to investigate a suspected data problem is to rebuild the corpus, which is why the problem usually goes uninvestigated.