US6076059A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method for aligning text with audio signals
Assignee: Hewlett Packard Development Co LP
•
Filing: 1997-08-29
•
Granted: 2000-06-13
•
Expires: 2017-08-29
Abstract
Abstract In a computerized method, text segments of a text file are aligned with audio segments of an audio file. The text file includes written words, and the audio file includes spoken words. A vocabulary and language model are generated from the text segment. A word list is recognized from the audio segment using the vocabulary and language model. The word list is aligned with the text segment, and corresponding anchors are chosen in the word list and text segment. Using the anchors, the text segment and the audio segment are partitioned into unaligned and aligned segments according to the anchors. These steps are repeated for any unaligned segments until a termination condition is reached.
Inventors
- 👤 Oren Glickman
- 👤 Christopher Frank Joerg
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-03 07:29
CPC Classifications
G; G10; G10L; G10L15/00; G10L15/26
Claims Analysis
Total Claims: 24
Claim 1
Independent Claim
219 words
A computerized method for aligning text segments of a text file with audio segments of an audio file, comprising the steps of: generating a vocabulary and language model from the text file, generation of said model involving determination of relative probabilities of all one, two, and three word sequences in all unaligned text segments of the text file based upon frequencies of occurrences of said sequences in said unaligned text segments, all of said text segments being initially classified as unaligned text segments; recognizing a word list from the audio segments using the vocabulary and language model but without considering the text file; aligning the word list with the text segments based upon respective scores for all possible alignments of words in the word list with the text segments, each respective score being weighted to increase each respective score by a relatively greater amount if a respective alignment associated with the respective score involves relatively longer sequences of correctly aligned words; choosing corresponding anchors in the word list and text segments in accordance with the respective scores; partitioning the text and the audio segments into unaligned and aligned text and audio segments according to the anchors; and repeating the generating, recognizing, aligning, choosing, and partitioning steps with the unaligned text and audio segments until a termination condition is reached.
Claim 2
Dependent Claim
219 words
A computerized method for aligning text segments of a text file with audio segments of an audio file, comprising the steps of: generating a vocabulary and language model from the text file, generation of said model involving determination of relative probabilities of all one, two, and three word sequences in all unaligned text segments of the text file based upon frequencies of occurrences of said sequences in said unaligned text segments, all of said text segments being initially classified as unaligned text segments; recognizing a word list from the audio segments using the vocabulary and language model but without considering the text file; aligning the word list with the text segments based upon respective scores for all possible alignments of words in the word list with the text segments, each respective score being weighted to increase each respective score by a relatively greater amount if a respective alignment associated with the respective score involves relatively longer sequences of correctly aligned words; choosing corresponding anchors in the word list and text segments in accordance with the respective scores; partitioning the text and the audio segments into unaligned and aligned text and audio segments according to the anchors; and repeating the generating, recognizing, aligning, choosing, and partitioning steps with the unaligned text and audio segments until a termination condition is reached.
Claim 3
Dependent Claim
25 words
The method of claim 1 wherein the text segments correspond to the entire text file, and the audio segments correspond to the entire audio file.
Claim 4
Dependent Claim
15 words
The method of claim 1 wherein the language model is in the form of trigrams.
Claim 5
Dependent Claim
21 words
The method of claim 1 wherein the word list includes a sequential list of recognized spoken words of the audio segments.
Claim 6
Dependent Claim
17 words
The method of claim 1 further including the step of annotating the word list with timing information.
Claim 7
Dependent Claim
20 words
The method of claim 4 wherein the timing information includes the starting time and duration of each recognized spoken word.
Claim 8
Dependent Claim
18 words
The method of claim 1 wherein the word list is recognized using acoustic-phonetic models of a speech recognizer.
Claim 9
Dependent Claim
28 words
The method of claim 1 further including the steps of determining a plurality of possible alignments, scoring each possible alignment, and selecting a best alignment using dynamic programming.
Claim 10
Dependent Claim
32 words
The method of claim 8 further including the step of increasing the score of a particular possible alignment when the particular possible alignment includes a relatively long sequence of correctly aligned words.
Claim 11
Dependent Claim
26 words
The method of claim 5 further including the step of annotating the text segments with the timing information for correctly aligned words of the audio segments.
Claim 12
Dependent Claim
21 words
The method of claim 1 wherein the termination condition occurs when all of the text and audio segments are fully aligned.
Claim 13
Dependent Claim
21 words
The method of claim 1 wherein the termination condition is a detection of all anchors in the text and audio segments.
Claim 14
Dependent Claim
29 words
The method of claim 1 wherein the termination condition is reached for a particular unaligned segment if the particular unaligned segment has a duration less than a predetermined threshold.
Claim 15
Dependent Claim
15 words
The method of claim 1 wherein the unaligned segments include portions of adjacent aligned segments.
Claim 16
Dependent Claim
24 words
The method of claim 1 wherein the vocabulary and language model is rebuilt from the unaligned segments during repeated iterations of the generating step.
Claim 17
Dependent Claim
17 words
The method of claim 1 wherein the text segments correspond to a portion of the audio segments.
Claim 18
Dependent Claim
20 words
The method of claim 7 wherein the acoustic-phonetic models are dynamically adjusted during repeated iterations according to the aligned segments.
Claim 19
Dependent Claim
21 words
The method of claim 7 wherein the acoustic-phonetic models are dynamically adjusted during repeated iterations to take into consideration channel characteristics.
Claim 20
Dependent Claim
26 words
The method of claim 17 wherein the acoustic-phonetic models initially are speaker independent, and the acoustic-phonetic models are adjusted to be speaker dependent during the iterations.
Claim 21
Dependent Claim
38 words
The method of claim 19 wherein the text segments are annotated with speaker identification information for a plurality of speakers, and further including the step of adjusting the acoustic-phonetic models to include a plurality of speaker dependent models.
Claim 22
Dependent Claim
22 words
The method of claim 20 wherein corresponding text segments are generated from particular audio segments using the plurality of speaker dependent models.
Claim 23
Independent Claim
237 words
An apparatus for aligning text segments of a text file with audio segments of an audio file, comprising: an analyzer for analyzing the text segments to generate a vocabulary and language mode, generation of said model involving determination of relative probabilities of all one, two, and three word sequences in all unaligned text segments of the text file based upon frequencies of occurrences of said sequences in said unaligned text segments, all of the text segments of the text file being initially classified as unaligned text segments; a speech recognizer for generating a word list from the audio segments using the vocabulary and language model but without considering the text file; an aligner for aligning the word list with the text segments based upon respective scores for all possible alignments of words in the word list with the text segments, each respective score being weighted to increase each respective score by a relatively greater amount if a respective alignment associated with the respective score involves relatively longer sequences of correctly aligned words: an anchor choosing mechanism for choosing corresponding anchors in the word list and text segments in accordance with the respective scores; a partitioning mechanism for partitioning the text and the audio segments into unaligned and aligned segments according to the anchors; and a repetition mechanism for repeating the generating, recognizing, aligning, choosing, and partitioning steps with the unaligned segments until a termination condition is reached.
Claim 24
Independent Claim
237 words
An apparatus for aligning text segments of a text file with audio segments of an audio file, comprising: an analyzer for analyzing the text segments to generate a vocabulary and language mode, generation of said model involving determination of relative probabilities of all one, two, and three word sequences in all unaligned text segments of the text file based upon frequencies of occurrences of said sequences in said unaligned text segments, all of the text segments of the text file being initially classified as unaligned text segments; a speech recognizer for generating a word list from the audio segments using the vocabulary and language model but without considering the text file; an aligner for aligning the word list with the text segments based upon respective scores for all possible alignments of words in the word list with the text segments, each respective score being weighted to increase each respective score by a relatively greater amount if a respective alignment associated with the respective score involves relatively longer sequences of correctly aligned words: an anchor choosing mechanism for choosing corresponding anchors in the word list and text segments in accordance with the respective scores; a partitioning mechanism for partitioning the text and the audio segments into unaligned and aligned segments according to the anchors; and a repetition mechanism for repeating the generating, recognizing, aligning, choosing, and partitioning steps with the unaligned segments until a termination condition is reached.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2003-06-13 | 2003-12-13 | 2004-06-13 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2007-06-13 | 2007-12-13 | 2008-06-13 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2011-06-13 | 2011-12-13 | 2012-06-13 | $7700.00 | Window Closed / Lapsed |