Question
A character tries to complete a sentence while holding back tears. The breath catches on a particular word. They recover, try again, and cannot finish.
Can text-to-speech (TTS) handle this sequence as something the character is doing, rather than simply produce a sad voice? Can a director reproduce, revise, or reject the breakdown at the intended point?
Edited from Report 19, dated 7 September 2026. This is a production and evaluation question, not a comprehensive benchmark of models available in October 2026.
Context
The research considered here explores control over speaker characteristics, prosody, emotion, style, and phenomena found in spontaneous speech. Prosody includes the patterns of pitch, stress, rhythm, and timing through which speech is organized.
But a performed struggle is not necessarily the sum of those features. Applying a crying sound uniformly to a line differs from trying not to cry, speaking through that effort, losing control briefly, recovering, and breaking again. The sequence carries both an objective and a resistance to it.
I use performed breakdown for a local disruption of speech while a character is trying to accomplish an action. It is not a mispronunciation, degraded audio, or a clinical classification of a speaker. Ordinary disfluency is not automatically failure: a pause can be a deliberate means of managing an interaction.
Evidence
Control is becoming more differentiated
Xie and colleagues' survey separates control over attributes including prosody, timbre, emotion, style, language, and environment. It also treats evaluation as more than a naturalness judgment. A voice may sound plausible while failing to follow a timing or emphasis instruction. [1]
Spontaneous speech can be modeled in parts
Li and colleagues' Interspeech 2024 work models 19 spontaneous behaviors through behavior labels, syntactic information, and fine-grained prosody. Its Chinese-language material includes repetitions, prolongations, filled pauses, and non-speech sounds. This is evidence of controllable components, not a demonstration of every kind of dramatic action across languages. [2]
Chaudhury and colleagues use a language model to place emotion and disfluency cues in text, then convert them through a rule-based synthesis and audio assembly pipeline. Context-sensitive cue placement matters, but that particular construction should not be mistaken for one model continuously generating an embodied dramatic performance. [3]
The text does not determine a unique performance
TEMOTTS connects text representations with an emotional style space without requiring explicit emotion labels. It provides a way to infer expressive delivery from text. [4] My production question is different: the same line can conceal an emotion, test another person, sound ironic, or maintain a fragile appearance of calm. Those alternatives require a choice about the scene.
Naturalness is not the whole evaluation
Cooper and colleagues' evaluation guidance questions under-specified naturalness judgments and isolated-sentence mean opinion score (MOS) tests, and emphasizes evaluation suited to the claimed use case. [5] For this proposed use case, a smooth line is not necessarily the desired outcome.
Findings
A sound event and its function are different objects
A filler can express uncertainty, buy time, or soften a reply. A repetition can register revised thought, agitation, concealment, or failed persuasion. Breath, laughter, and vocal tremor need to fit the changing relation between the character's objective and bodily state. Recovery matters too: returning to the sentence, replacing a word, falling silent, or breaking again can change the scene.
Disfluency generation and performed breakdown overlap, but they are not interchangeable.
Three layers of the problem
Acoustic generation: produce speech sounds, voice quality, pitch, duration, intensity, and non-verbal events.
Utterance design: connect hesitation, repair, breath, or a failing phrase with meaning and situation.
Production judgment: decide which result belongs to the character, which representation it produces, and how it can be revised and approved.
This is an editorial framework, not a standardized taxonomy. Research can support the first two layers without settling the third.
An intermediate representation for performance?
A direction such as “sadly” does not specify what fails, when, or why. A production team may need to share the character's objective, bodily condition, effort to suppress an emotion, point of breakdown, possibility of recovery, and relation to the listener.
Translating that account into breath, silence, syllable timing, pace, and vocal texture could become an intermediate control layer. That is a proposal, not a claim that an established system already solves it.
Performed breakdown is not simply added noise. It concerns what a person tries to maintain, and the sequence through which they cease to maintain it.
Structural implications
Translation includes the word that cannot be finished
The source language's point of breakdown may not retain its function at the same position in another language. Word order, sentence endings, politeness, and vowel length can move the expressive weight. The adaptation needs to ask which unfinished word makes the scene work, not only how many seconds the sentence occupies.
Direction can move from an emotion to an action
Instead of “sadly,” a director might specify: the character tries to sound composed, loses the breath at the final verb, pauses, and escapes into another formulation. This invented direction makes an action and sequence discussable before they become editing and synthesis instructions.
Optimization for smoothness could remove a hesitation, rough edge, or missing word that a scene needs. Evaluation must distinguish an intended disruption from an unintended generation failure.
Separate dramatic success from production control
Performance: do the objective, breakdown point, bodily cues, recovery, and surrounding scene make sense together?
Basic quality: are the words, pronunciation, speaker continuity, and audio quality acceptable?
Production control: can the team reproduce the result, change its location or intensity, and record the conditions and approval?
A lucky generation can work dramatically but be difficult to revise. A reproducible effect can still be meaningless in the scene. Neither test replaces the other.
Reference performances, consent, reuse, and final approval remain part of the workflow. Expressive material involving crying, pain, intoxication, or stuttering also requires attention to the person or group being represented; a convincing effect alone is not authorization to use it.
Uncertainties
The cited work examines components of expressivity, spontaneous speech, control, and evaluation. It is not a set of comprehensive experiments on the specific dramatic problem proposed here. Its scope does not establish that every current model fails to handle such a sequence.
The Chinese-language study does not establish cross-language or cross-cultural generality. The cue-and-assembly pipeline does not demonstrate continuous bodily causation. Nor does human acting supply a single correct version: evaluations depend on scene, audience, language, and performance conventions.
The useful next step is to test the sequence, the meaning it carries, and its revisability separately. Depending on the intended audience, native listeners, actors, directors, and audio practitioners may need to evaluate different aspects.
For the conceptual background, see From Script to Utterance and Dubbing as Re-embodiment.
Sources
- [1] Xie et al. — Towards Controllable Speech Synthesis in the Era of Large Language Models ↗ — Systematic survey; version 3, August 2025.
- [2] Li et al. — Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models ↗ — Interspeech 2024; 19 behaviors. The authors' readable preprint is also linked below.
- Li et al. — Author preprint ↗ — July 2024. Accessible companion to the conference paper.
- [3] Chaudhury et al. — Humane Speech Synthesis through Zero-Shot Emotion and Disfluency Generation ↗ — 2024; language-model cues and rule-based audio conversion.
- [4] Chandra et al. — Emotional TTS without Emotion Labels ↗ — Odyssey 2024; TEMOTTS.
- [5] Cooper et al. — Good Practices for Evaluation of Synthesized Speech ↗ — March 2025; evaluation guidance, not a benchmark of acting.
- Further reading: Triantafyllopoulos & Schuller — Expressivity and Speech Synthesis ↗ — Retained from the Japanese report; 2025 revision.
- Further reading: Suni et al. — Style and Prosody Control for Zero-shot Speech Synthesis ↗ — SSW 2025; retained from the Japanese report.
The English edition clarifies the September observation as a research and production question, rather than a categorical limit of every TTS model. Core references [1]–[5], including the author preprint for [2], were consulted for this edition on 11 October 2026. The performance framework and invented scene directions are the editor's analysis.