Talky

Basically a high-precision, phoneme-level pronunciation scoring engine using the CTC decoding, Wav2Vec 2.0, and Viterbi forced alignment to give per-phoneme feedback on the sentences you say. Turned this engine into an open-source gamified speech therapy app with React and Tailwind CSS. Think of this project as a “Duolingo” for speech therapy. Currently leading a small team of open source contributors.

I think this project was quite interesting as we essentially built a high-performing version of Azure Speech or the Duolingo speaking exercises. This was definitely the most difficult part of the project, and I tried many many options to make this core functionality work.

Option 1: This was the first option that we tried (the version we implemented in the hackathon). The pronounciation scoring engine was quite trivial to implement, where we simply used OpenAI Whisper to transcribe speech and then basically matched how close the transcribed sentence was to the target sentence. However, this was evidently very bad, because 1) Whisper is known to be notoriously slow, and 2) the transcription model would often auto-correct your words if your pronounciation was close enough.

Option 2: Following the hackathon, we decided to improve our product, since speech therapy is a very real and expensive problem for many people. My wonderful co-worker made a function that used a ASR model as well as the Goodness of Pronounciation algorithm, taking the logits of the expected word tokens to calculate the score (this accounted for not only how many words were correct but also the model’s confidence level that the pronounced word was correct, indicating better pronunciation).

Option 3: I found that still, option 2 didn’t give precise enough feedback – it still only gave word-level feedback. So, naturally I searched on everyone’s favourite website (Huggingface) for a better model, and found a phoneme-level ASR model, Wav2Vec-base-960h. I matched the phoneme-level tokens transcribed from the model to the target list of phonemes converted from the target sentence. However, here, there was also a problem: I had used a separate words-to-phonemes library, and the phoneme format and standards were completely different. Though I tried normalizing them, some were following IPA and some were in ARPABET, TIMIT, etc., and the phonemes were too different to match. This was solved in the 4th version.

Option 4: In this version, I set out to fix all the issues mentioned above, which were:

1. latency of the scoring engine being too high
2. accuracy of the scoring engine was unreliable
3. phonemes had many different standards which didn’t match

Latency. I completely replaced the previous scoring engine and built from scratch. I prototyped a new function which records audio, and processes an audio chunk every 80 ms (inspired by Mistral’s Vixtral model). Then, I would use the phoneme-level ASR model to transcribe phonemes, and stream out the decoded phonemes. I then set up websockets between the frontend and the server, streaming audio chunks every 500 ms (500 ms, because from experience, using 80 ms affected decode accuracy).

Accuracy. There were several things I did with improving the accuracy. Previously, we relied on the argmax token to find the model’s confidence, and use that in the scoring system, but that would be misleading if the user speaks a completely wrong phoneme that leads the model to be “incorrectly confident.” Thus, I fixed this to take the logit of the actual expected phoneme. Additionally, from many testing trials, I found that the Wav2Vec model was actually hyper-sensitive to the microphone changes, so I relaxed the constraints a bit and allowed for a phoneme to be counted as correct as long as it was within the top-3 highest logits.

There was another huge problem with the accuracy. The CTC decoder can decode what it hears, but has no idea what the user is trying to say, or what the target phoneme at any particular point is. To fix this, I kept a pointer that pointed to where the target phoneme was, and advanced the pointer as the user spoke words correctly.

However, this led to another problem. If the user skips a word, or is incorrect on a particular word, the pointer would never advance. For example, for the sentence “the quick brown fox jumps over the lazy dog”, the user may have pronounced “the” incorrectly, but goes on to subsequently speak the word “quick”. In this scenario, the model would still believe that the target phoneme is the first phoneme in “the”, since it was never registered as correct.

To solve this, I used a lookahead mechanism, where for every phoneme, the algorithm scans for phonemes up to 3 positions ahead of the current target phoneme. So, if a phoneme has been skipped over or pronounced incorrectly, it would be marked as either an [omission] or [mispronounced].

Unmatching standards: I was super happy with this fix as it’s a pretty clever idea. I basically used the Wav2Vec model’s own tokenizer to parse the target phonemes from the target sentence, so it will always be consistent with the ASR-decoded outputs. Here’s an example.

reference_phonemes = [p for p in processor.tokenizer.tokenize(ipa.convert(reference_sentence)]

There were a few last edge cases here and there, like normalization, but those fixes were quite small compared to the issues I described above. Yayy!

Tags: React, Tailwind CSS, Python, Wav2Vec 2.0, Forced Alignment

Links: Code