Diatts

Notes on open-source speech synthesis

Prosody

Prosody is the harder half of speech

Getting the phonemes right is a solved-enough problem. Deciding which syllable carries the weight of a sentence is where synthesis still shows its seams.

Published June 12, 2026 · Updated June 14, 2026 · Filed under Prosody

Prosody is the collective name for the properties of speech that survive when you remove the words: the melody of pitch, the rhythm of syllable durations, the distribution of loudness, and the placement of pauses. It is the layer that turns a correct pronunciation into a plausible utterance, and it is the layer that most reliably reveals a synthetic voice.

The core difficulty is that prosody is underdetermined. A written sentence maps onto many acceptable spoken realisations, and the choice among them depends on context the text does not carry: what was said before, what the speaker assumes the listener already knows, and what is being contrasted with what.

Three variables, heavily entangled

Pitch, duration and intensity are usually described separately and are never independent. An emphasised syllable is typically higher, longer and louder at the same time, and the relative weighting of those three cues varies across languages and speaking styles. A model that learns emphasis as a pitch excursion alone will produce something that sounds strained rather than emphatic.

Duration in particular does more work than its reputation suggests. Final lengthening at phrase boundaries is one of the strongest cues to syntactic structure in speech, and a system that keeps syllable durations too even will sound like it is reading a list even when the sentence is not one.

The text of a sentence does not contain enough information to say it correctly.

Focus is a semantic decision

Which word carries the main accent of a sentence depends on what the sentence is answering. The same clause spoken as a correction places its accent differently than the same clause spoken as new information. Because the accent placement follows the information structure, a synthesis system with no access to the discourse has to guess, and its safest guess is a default late accent that is right often enough to be unremarkable and wrong often enough to be noticed.

This is why passage-level conditioning helps more than it seems it should. Knowing the previous sentence narrows the plausible focus positions in the current one, sometimes to a single choice.

Where control interfaces help and where they mislead

Many systems expose a small set of prosodic controls, usually some form of speed, pitch range and an emotion or style selector. These are useful for coarse adjustment and misleading as a mental model, because they are global while the problem is local. Slowing an entire sentence does not fix an accent placed on the wrong word; it produces a slow sentence with the accent still in the wrong place.

The more effective interfaces operate on spans: mark the word that carries focus, mark the phrase boundary, and let the model realise those marks with its own learned combination of pitch, duration and intensity. That keeps the human decision at the level where the ambiguity actually lives.

Listening for it

A practical exercise: take a synthesised paragraph and transcribe, by ear, which syllable in each sentence carries the main accent. Then read the paragraph aloud and mark your own choices. The disagreements are the system's prosodic error profile, and they are far more informative than a single overall quality score, because they point at a specific and fixable class of mistake.