I cloned my voice for the podcast, and found out TTS engines swallow whole sentences

For twenty episodes, the voice of my podcast wasn’t mine. It was an off-the-shelf synthetic voice, rented on a monthly plan. It worked fine and I had no complaints.

Then one day I read there were much cheaper alternatives and went to have a look. That trip has three parts: a voice catalogue I wasn’t expecting, some fine print that killed the whole reason for the trip, and a technical failure I don’t think many people talk about, which is the most useful part of all this.

Spoiler: I ended up back where I started, but with a different voice.

Part 1: the Spanish catalogue is a minefield #

The candidate was Fish Audio: same kind of service, API pricing well below what I pay. The first thing I did was look at its Spanish voices, because the recurring problem is the accent: I need European Spanish, not Latin American.

I filtered their library and this is what came out, sorted by usage:

• a very well-known TV presenter
• the narrator of the nature documentaries
• dubbing characters: SpongeBob, Homer, Rambo, Goku
• a footballer, listed by name

Voices cloned from real people, uploaded by users, ready for anyone to generate audio with. The API serves them like any other.

Them being there doesn’t mean you can use them. If I publish a podcast in the voice of someone recognisable, the problem isn’t a technical one. And this isn’t the grey area of “sounds a bit like”: these are voices labelled with the person’s full name.

Discarding all that, what’s left are generic voices with no apparent owner. I tried four. They all read the same text at about 165 words per minute, while my podcast runs at 245. That’s audiobook pace: a five-and-a-half-minute episode would stretch to eight.

Part 2: the fine print that killed the reason for the trip #

I was doing all this for the price, so I ran the numbers.

My scripts weigh between 4,000 and 7,700 bytes. At fifteen dollars per million bytes, each episode costs about eight cents. Usage is irrelevant either way. What decides is the subscription.

And there was the catch:

The API I had run every test with is only available from the Pro plan up. So migrating would have cost me more than triple what I was already paying.

The reason for the trip had just evaporated. But by then I had already found the interesting part.

Part 3: cloning my own voice #

If the catalogue is unusable, the obvious option is my own voice.

I recorded two minutes inside my clothes wardrobe, which is the best recording booth in an ordinary house: clothes eat the echo, and echo can’t be removed afterwards. On an iPhone, in Voice Memos, with quality set to lossless.

Before spending the clone, I measured the take:

1ffmpeg -i sample.m4a -af volumedetect -f null -   # peaks: -6.3 dB, no clipping
2ffmpeg -i sample.m4a -af astats -f null -          # noise floor: -60 dB

Noise floor at −60 dB and no clipping: good enough. If you clone from a bad take, every episode you ever make carries that flaw.

And here came the pleasant surprise: a cloned voice copies your pace. Reading the same text, my cloned voice took 40.2 seconds and the catalogue voice I’d used for twenty episodes took 40.0. Identical. The audiobook-pace problem solved itself.

And now the important bit: they swallow sentences #

Listening to a test I noticed something odd and assumed the model had switched languages for a couple of seconds. I went to check properly, and that wasn’t it.

What it does is worse. It skips an entire sentence from the script and fills the gap with sounds that mean nothing. This is what whisper transcribed from a six-second stretch:

diompreinse quintun tianos den timpreinse hana santo de pugenia

It sounds like another language because they’re loose phonemes with sentence intonation. And where it should have said “And an official till in the middle has its convenience, but also its risk”, it said nothing of the sort. It carried on as if nothing had happened, no odd pauses, no cuts.

I measured it properly, generating the same text several times and comparing word by word:

Parameters fix nothing. The trigger is sending several paragraphs at once: the model loses the thread and invents that stretch. Split by paragraph, it nails the text.

And the one I already used isn’t immune either, which is the part that actually matters. With my cloned voice, a 227-word block dropped sentences on four consecutive attempts, a different one each time. Its numbers were better —95.4% and 98.7% against 82.2% and 88.4%— but it’s the same failure, just rarer.

I fixed it in two boring ways:

1. Splitting long sentences. The ones it swallowed had sub-clauses: “Adding, counting and measuring is done by an ordinary program, the old-fashioned kind, the one where two plus two always makes four”. Turned into four short sentences, it came out clean on the first try.
2. One call per paragraph on the long blocks.

The method: let something without ears check it #

This is my takeaway. I would never have caught it by ear. Two seconds of gibberish in the middle of a sentence goes unnoticed when you’ve spent half an hour listening to your own text and you know what comes next. Your ear fills the gap.

So the validation is done by a machine: generate the audio, transcribe it with whisper, compare it to the script word by word.

1said = transcribe(mp3)                        # whisper-cli, local
2a, b = normalise(script), normalise(said)
3ratio = difflib.SequenceMatcher(None, a.split(), b.split()).ratio()

With that, every block of the episode tells you whether it’s publishable before you assemble it. And if it fails, you regenerate just that piece.

One important detail if you build something like this: you must ignore numeric differences, because whisper writes “12,345” where your script says “twelve thousand three hundred and forty-five”. Without that filter the validator throws false positives constantly.

The punchline: the verifier lies too #

With the episode assembled, the validator flagged two missing sentences in one block. But those same sentences, generated separately, had come out clean.

I compared durations: the sum of the pieces was 72.05 seconds and the assembled file, 72.05 seconds. Not a tenth missing. No audio was missing: whisper was the one skipping bits while transcribing the long file.

So the tool I use to detect silent failures has exactly the same silent failure. That’s why validation runs on short blocks and gets cross-checked against durations, which don’t have opinions.

What I take away #

• In a catalogue of cloned voices, check whose voice it is before publishing it. The API serving it means nothing.
• Check which plan unlocks the API before counting your savings. Mine was three tiers above the one I’d looked at.
• No TTS is reliable word for word. They all skip things, some more than others. Short sentences, one call per paragraph.
• If you publish generated audio, validate it with something that has no ears. And distrust the validator too.

I ended up staying where I was, paying the same. But the voice reading my scripts now comes from two minutes I recorded inside a wardrobe, and that feels like a bigger change than switching provider.

I tell the story in that same voice in episode 21 of the podcast (in Spanish), where the previous voice also says goodbye.