Questions about speech-truncation-detection-12M for audiobook/TTS QC
Hi @brthor — I'm work on an open-source audio QC tool called FinalPass (on my github at https://github.com/alexeymohr/FinalPass) and came across speech-truncation-detection-12M. Looks like it could really help with a problem I want to solve: detecting TTS utterances where the final word remains intelligible, but the final phoneme/decay has been audibly chopped off.
I have a real labeled test case from an AI-generated audiobook with four independently identified examples of this defect, plus roughly 110 clean utterance endings, so I'd love to evaluate your model against it.
Before doing that, a few quick questions:
- What kind of truncations was the model trained to detect? Especially whether it includes subtle terminal-phoneme truncation versus more obvious mid-word cuts? My problem audio is intelligible, but the very ends of words get chopped.
- Were the positive examples naturally occurring, synthetically truncated, or both?
- What license do you intend for the model code and weights? I couldn't find one declared in the repository. FinalPass is an open-source GitHub project, so I want to be careful and not integrate anything that isn't allowed.
Happy to share the results of my evaluation back with you. Your model is the first purpose-built system I've found that appears to target this exact class of problem.
Hello @alexeymohr
In my testing the detection is fairly sensitive. It won't detect if a word itself is incomplete (not a language model), but if the audio ends abruptly in a word or as a word is beginning it will mark that.
IIRC it was trained using forced alignment timings with something like 20-30ms of sensitivity.
apache 2.0 license has been added
I have made wide use of this model. Any AI agent should be able to make short work of an eval given the examples. I'd be happy to hear if you end up using it.
Thanks for taking a look.
EDIT:
My problem audio is intelligible, but the very ends of words get chopped.
This is the problem the model was built to detect, for the purpose of improving TTS training datasets.
Hi @brthor , following up as promised: I did end up using it. Your model now powers the “word ends abruptly” check in FinalPass AudioBook, an offline QC tool for AI-narrated audiobook chapters (and really any AI-generated single-speaker voice work). It’s a sibling project to FinalPass and lives at https://github.com/alexeymohr/FinalPassAudioBook.
Thank you for adding the Apache-2.0 licence so quickly. It’s what made this possible, since the project is MIT and I didn’t want to include anything without clear terms. I wanted to make certain I wasn't passing along anything that was not allowed.
How it’s used:
It runs fully on the user’s own machine. The model code ships unmodified with its licence, pinned to a specific revision, and the weights are checked against a recorded checksum. For speed and a smaller install it runs as a numpy port of the inference path, which is tested to match your PyTorch code.
It’s credited by name, with a link back to this page, in the README and in the licence notices shipped with the Mac and Windows apps.
What I found in evaluation, as promised:
When I scored every pause it gave many false alarms. So the check only scores the points where a generated clip ends and digital silence begins, which is where these chops happen in my tested output. This keeps it focused on the errors that I was experiencing most frequently.
On my main labelled title, it flags 10 of 118 clip endings: 9 were real chops, and 1 was actually a clean ending.
On a second title it flagged 0 of 27, which was correct. None were chopped, and it never falsely tagged any.
On two more titles it flagged 8 endings. All of them were genuinely abrupt by ear, though none was actually missing part of a word. But I still rated that as quite useful, since it called attention to elements that could be deemed creatively unacceptable.
So it does what you said it would: it catches audio that stops while speech is still going. That turns out to be exactly what an audiobook QC listener notices. Thanks again for building it and sharing it!
@alexeymohr
Very cool. I am glad to see the model put to good use.
When I scored every pause it gave many false alarms.
Its a tough problem to nail exactly, especially due to needing some really clean markers to train against. I was conservative with positive/negative labels trying to use a sparse supervision to encourage generalization but it may have muddied some boundary areas.
Perhaps it doesn't handle between-words as well as end of utterance as far as chop detection goes.
So, there is definitely room for improvement.
On two more titles it flagged 8 endings. All of them were genuinely abrupt by ear, though none was actually missing part of a word.
I noticed this behavior as well, including flagging clicks or the start of some other speech at the end of an audio.
From the perspective of getting clean TTS endings i think this is probably desirable.
Thanks for reporting the findings!