Subtleties of Edit Distance

Normally ASR is measure by edit distance measured in words or charecters (also refered to as levenstien distance). It measures how many insertions, deletions, substitutions.

This is good because it captures the intuitive notion of closeness and also because it resembles the proccess of making the needed correction. It is also good because it resembles the number of discrete errors that are made.

The need for normalization:

Ground Truth (GT)

Hi, I’m John. I’ll meet you at 5 p.m. on December 3rd at McDonald’s — okay?

ASR Transcript A (Pre-Normalization)

hi i'm john i'll meet you at five pm on december third at mcdonalds okay

• Differences (raw text):
  • Missing capitalization and punctuation
  • “5 p.m.” → “five pm”
  • Apostrophes and case all dropped

But this is of course a correct transcript, just a poorly formatted one

However it is not without problems of its' own consider the following case:

Ground Truth (GT)

After the meeting Dr Kowalski reviewed the data and Kowalski approved the plan for Kowalski and his team

ASR-1 (repeated proper-name misspelling)

After the meeting Dr Kowalsky reviewed the data and Kowalsky approved the plan for Kowalsky and his team
Errors vs GT: S=3 (each “Kowalski”→“Kowalsky”), D=0, I=0 → WER = 3/18 = 1/6 ≈ 16.7%

ASR-2

After the reading Dr Kowalski previewed data and Kowalski approved the plan for Kowalski and his team

Differences vs GT: - S=2: “meeting”→“reading”, "reviewed" -> "previewed" - D=1: dropped “the” before “data” Total = 3 → WER = 3/18 = 1/6 ≈ 16.7%

I think there is a reasonable basis to say that ASR-1 is more correct than ASR-2 but they get the same score. This is several fold: 1) In some sense one error was made three times in ASR-1 2) The error that was made was a proper name, for which no unambiguous mapping from audio to spelling can be made -- some people are named Kowalsky some Kowalski with the exact same pronouciation.

Issue 2 is related with normalization -- perhaps all proper names with common phonetic spellings should be normalized to a canonical case.

Issue 1 is more intrinsic to using edit distance in the first place, we look at the errors in isolation, and unless locally there are some errors that cancel out we don't consider errors in light of any others.

However this runs afoul of both our common sense notion of how many errors were made and the analogy to making corrections, after all in the case of receiving transcript ASR-1 above and correcting it we would naturally just search and replace "Kowalsky" -> "Kowalski" which would take a similar amount of time and effort as doing it once.

Likewise if we imagine telling the system that this error was made, a smart system would see that the other errors are obvious in light of that and make all needed corrections at once.

The more subtle case of semantic information and correction

Ground Truth (GT)

Book the 7:10 a.m. nonstop Southwest flight to Austin (AUS) — WN 2317, arriving 9:05 a.m. Dinner at Uchi in SoCo at 7:30 p.m.. After landing, use the rideshare pickup at “Lower Level G.”

ASR Transcript (seed error → all other fixes follow; all substitutions sound alike)

Book the 7:10 a.m. nonstop Southwest flight to Boston (BOS) — WN 231, arriving 9:15 a.m. Dinner at Uni in SoWa at 7:30 p.m.. After landing, use the rideshare pickup at “Lower Level C.”

this has the property of having various distinct errors however if a human we given the correction "Boston" -> "Austin", be equipped with a webbrowser and do a bit of reserach the other errors could be corrected. Likewise a sophisticated enough ASR system could do the same. This is 7 distinct errors but clearly a very different situation.