I Could Hear Every Word. Why Didn't It Feel Like Someone Was Speaking?

Hearing a voice, seeing the text

While watching and listening to AI dubbing demos in different languages, I found myself caught on a Japanese voice.

It was clear. I could understand the words. Yet it sounded less like a character speaking than a character reading the text. I was listening to a voice, but the impression that stayed with me was of words on a page.

I work on the performance side, not in mixing. What caught my attention was less the finish of the sound than the choices within the utterance.

Would they take a breath there?

Is that the word they want this person to notice, right now?

The placement of breaths and the words that stood out did not quite connect with the character's intention as I understood the scene.

That does not make my reading the only valid one. I did not know the input, model, or production workflow, so I could not identify a cause. But I wanted to investigate the gap between understanding the words and hearing them as something this character would say.

What are we calling a good voice?

The literature offered a useful distinction: evaluating an isolated sentence is not the same as evaluating speech in the situation where it will be used.

Lameris and colleagues studied a robot explaining the rules of a game, examining how situational information and preceding speech affected suitability ratings. Including context influenced ratings, although the type of context did not produce a significant difference. This was not a study of film acting. It did, however, give me a reason to ask what listeners are being shown and what, exactly, they are being asked to evaluate.[7]

I began separating several questions:

Can I understand the words?
Are the pronunciation and lexical accent appropriate for the intended variety of the language?
Does the speech sound natural?
Does it sound like a reply, correction, or contrast in this conversation?
Does it fit the character's purpose, relationships, and changing situation?
Can an unwanted detail be revised without changing the parts that should remain?

This is my way of organising the questions, not a standardised assessment framework. A high naturalness rating should not silently become evidence that a performance fits a scene.

Perhaps there was no contradiction in finding the speech clear and still feeling that something was off.

Why make that word stand out?

Consider this invented line—not an actual AI output:

私が、昨日、渡したんです。
Watashi ga, kinō, watashita n desu.
“I handed it over yesterday.”

If someone thinks another person did it, the speaker might make “I” stand out. If the misunderstanding concerns the date, “yesterday” could become prominent. If someone thinks the speaker still has the object, the act of handing it over might be the point of the correction.

The written line stays the same, but what it answers changes what the listener needs to notice. This is an illustration of possible readings, not a claim that English and Japanese realise them in the same way. Nor does making a word louder complete a performance.

Here I use prominence for a part of speech standing out perceptually, and focus for the information to which attention is particularly directed in context. I do not assume a one-to-one correspondence between the two.

Research on Japanese showed me that lexical pitch accent and focus-related intonation interact. Lee and Xu reported that post-focus compression—a narrowing of the fundamental-frequency range after focus—depends on lexical prosodic conditions.[1]

A perception study by Lee, Chiu, and Xu also found that focus identification varied with lexical accent and focus location. Its main experiment used resynthesised speech varying only in fundamental frequency. It was not a test of everyday conversation, in which listeners can also use context, particles, word order, duration, and voice quality.[2]

My practical takeaway was that checking a word's lexical accent and checking whether the intended information stands out are different tasks.

Choosing what should stand out is not merely a pronunciation problem either. Stephenson and colleagues investigated predicting contrastive focus on personal pronouns, including information from preceding utterances, and controlling prominence in TTS. Their work treats prediction as a problem involving semantic, discourse, and pragmatic knowledge.[11]

Choosing where to place prominence and producing the intended prominence: at least two kinds of work were folded into what I had heard as one impression.

Unpacking “I wouldn't breathe there”

Breathing was the other thing that caught my attention.

But “the breathing sounds wrong” is not a very precise description. A silent pause, an audible breath-like sound, a prosodic boundary signalled by pitch or duration, and physiological breathing activity need to be distinguished.

Fujisaki and colleagues analysed pauses in Japanese text reading in relation to syntax and respiratory constraints. Sugahara investigated how syntactic boundaries and lexical pitch accent condition post-focus prosodic phrasing in Tokyo Japanese. In the latter study, a boundary was indexed by an F0 rise—not by silence or an inhalation.[6][3]

A small pilot study recording speech alongside chest and abdominal movement in Japanese and Swedish reading also found that some pitch peaks and pauses coincided with respiratory-movement peaks, while others did not. It cannot establish a general rule, but it reinforces the need not to treat phrasing, pausing, and breathing as interchangeable.[5]

Watanabe and colleagues, studying Japanese monologues, likewise distinguished silent pauses, filled pauses, and conjunctions. Boundary strength corresponded with silent-pause duration and some occurrence rates, but not significantly with the durations of filled pauses or conjunctions. Thinking of timing simply as an amount of empty space seemed too coarse.[4]

So what had bothered me?

A character might be trying to cut someone off and get the whole line out in one go. They might be searching for a word. They might be trying to sound composed while their breathing gives them away. A breath in the same position can make sense for different reasons—or not make sense in the reading I have of the scene.

I do not want to turn that response into a rule that no Japanese speaker would breathe there. My question is whether I can understand why this character would breathe there, in this situation.

An inhalation-like sound in synthetic speech is not necessarily a recording of someone taking a breath at that moment. Yet a listener may still hear it as a cue to a body and its state. Perhaps the issue is less how many breath-like sounds are present than whether their placement makes sense as the character's utterance. That is my interpretation, not a result established by the studies above.

Is Japanese especially difficult?

I initially wondered whether shaping Japanese dialogue might involve particular difficulties. I still want to investigate that. But having language-specific conditions is not the same as being harder overall than another language.

Kurihara and colleagues evaluated a method for controlling Japanese TTS through phonetic and prosodic symbols, including accent, pauses, accent-phrase boundaries, and sentence endings. Prosody is not simply something that must be left to generation. The study does not, however, establish that a system can infer which choices a scene calls for or reproduce a character's breathing.[8]

Conditioning TTS on dialogue history is already a research topic, including in DailyTalk. PromptTTS 2 addresses a one-to-many problem: a text description does not capture every detail of voice variation. Neither establishes that arbitrary film scripts or Japanese dramatic performance have been solved.[10][9]

There are several possible layers to investigate: the language's prosodic conditions, the context provided, the model and its training data, synchronisation constraints, and what the tools allow someone to revise. There is also the listener—me. I may notice finer distinctions in Japanese than in other languages.

For a comparison, I would want native listeners in each language to judge speech with comparable communicative purposes and preceding context. Equal character counts or identical prompts would not, by themselves, make the conditions equivalent.

Along the way, I also became interested in the difference between reading visually and hearing speech unfold in time. But this research does not establish that Japanese is inherently optimised for silent reading, or that a reading culture causes problems in synthetic speech. That belongs to another investigation.

I did not need to settle the nature of Japanese as a whole to make my original question more concrete.

Turning an impression into something I can test

When I say the character seemed to be “reading the text,” I am not treating reading aloud as a lesser activity. A reading can bring a character, a situation, and a relationship with listeners into being.

What I mean is closer to this:

I could hear the text moving forward, but the arrangement of the voice gave me little sense of why those words were being spoken at that moment.

Rather than end with “AI has no feelings,” I want to separate what was supplied, what came out, and how I heard it.

The next step would be to secure the audio and the available production information. I would then record audible breaths, silent pauses, prominent words, timing, and lexical accent separately. I would offer my preferred reading as one possibility, while leaving room for other readings to work.

For revisions, I would specify what should change and what should remain. Rewording a line, changing a pause, adding an inhalation, and changing pitch are different operations. A single tool may not offer all of them.

An issue being audible does not mean it can be solved through sound processing alone. It may require a return to the script, direction, generation, editing, or synchronisation. I also want to assess performance fit separately from the ability to revise a particular detail while preserving the rest.

This is a proposal, not an experiment I have conducted. Recorded performances or voice clones would also require permission appropriate to the purpose and scope of use. Agreement to take part in an evaluation should not be read as agreement to model training or voice replication.

The question changed as I investigated

I was not trying to find a limit that AI could never cross. I wanted to understand why something had caught my attention when the words themselves were clear.

The literature already contained tools for thinking about it. What I have done here is connect my particular listening experience with work from several areas.

I still do not know what caused the result in that demo. I have not established that Japanese is the hardest language. But I now have a more specific question.

What does this character want this person to take from these words, at this moment? What should be held together, and what should stand out?

Once the text became a voice, I could hear choices the written line had not fully settled. That, I think, was what had caught me. How can those choices be communicated, checked, and revised when necessary? That is where I want to investigate next.

Sources and scope

This is a literature-based essay beginning with a personal listening experience, not a direct analysis of the demo audio, a product comparison, or a generation experiment. Translation accuracy has not been evaluated. The cited studies, published between 1998 and 2023, provide foundations and methods; they are not a survey of product capabilities in 2026. Findings about human speech, findings about TTS, and my own interpretations and proposals should be kept distinct.

The underlying research includes controlled stimuli, small samples, and single-speaker studies. Some sources were considered primarily through their public abstracts. These limits should not be generalised into a ranking of languages or a verdict on all current products. The numbered references match the Japanese edition.

First published 23 October 2026. English edition of the Japanese column, retaining the author’s first-person perspective and distinctions between personal observation, research and proposals. Written and edited by the LAYER SHIFT editor; published by Ausreißer Verlag.

Continue reading: Is Acting a Name for What AI Cannot Do? →