Why stripping 'um' from audio is hard — and how a local CLI cracked it

A developer built erm, a command-line tool that automatically removes filler sounds (um, uh, er) from voice recordings, and the writeup is a case study in why the obvious solution fails. The naive pipeline — transcribe with Whisper, find filler tokens, cut them with ffmpeg — only catches about 60% of disfluencies and produces audible artifacts. Whisper silently omits many fillers from transcripts because its training data is clean prose, cuts at arbitrary waveform points create audible clicks, and mismatched background noise makes every splice perceptible.

The tool layers three audio-analysis passes on top of Whisper’s transcript: detecting voiced sound inside long ‘pauses’ where Whisper dropped a filler entirely, splitting fillers that got glued onto adjacent words into a single token, and flagging words that last implausibly long for their text — with a pitch test to distinguish a held vowel from a slow talker. Cut points then slide to the quietest nearby spot and snap to zero-crossings to eliminate clicks, splices use crossfades scaled to cut length, and a loop of the room’s own ambient tone runs under the whole output to mask residual background mismatches. Denoising order matters too: detection runs on raw audio (where the acoustic cues survive) while cuts are rendered from a denoised copy.

Two design choices stand out. Everything runs locally via faster-whisper, so recordings never leave the machine. And the tool deliberately won’t touch ‘like,’ ‘you know,’ repeated words, or false starts — its rule is to remove only sound, never language, since those carry meaning and cutting them would change what the speaker said. A validate subcommand re-transcribes the output to confirm no fillers survived, giving an end-to-end correctness check.