EP0673170A2
Expired / Lapsed
EP
⏱️ 0.0 years remaining
Video signal processing systems and methods utilizing automated speech analysis
Assignee: AT&T Inc
•
Filing: 1995-03-08
•
Granted: —
•
Expires: 2015-03-08
Abstract
Abstract A method of increasing the frame rate of an image of a speaking person comprises monitoring an audio signal indicative of utterances by the speaking person and the associated video signal. The audio signal corresponds to one or more fields or frames to be reconstructed, and individual portions of the audio signal are associated with facial feature information. The facial information includes mouth formation and position information derived from phonemes or other speech-based criteria from which the position of a speaker's mouth may be reliably predicted. A field or frame of the image is reconstructed using image features extracted from the existing frame and by utilizing the facial feature information associated with a detected phoneme.
Inventors
- 👤 Tsuhan Chen
Bibliographic Details
Application Number
—
Jurisdiction
EP
Data Source
google_patents
Last Synchronized
2026-10-24 10:55
CPC Classifications
H; H04; H04N; H04N21/00; H04N21/40; H04N21/43; H04N21/434; H04N21/4341; G; G11; G11B; G11B20/00; G11B20/10; G10; G10L; G10L21/00; G10L21/02; G10L21/0316; G10L21/0356; G10L25/00; G10L25/48; G10L25/51; G10L25/57; H04N19/00; H04N19/10; H04N19/102; H04N19/132; H04N19/50; H04N19/587; H04N21/439; H04N21/4394; H04N21/44; H04N21/44008; H04N21/4402; H04N21/440281; G10L21/06; G10L21/10; G10L2021/105; H04N19/20
Claims Analysis
Total Claims: 13
Claim 1
Independent Claim
147 words
Apparatus for increasing the frame rate of a received video signal representative of a sequence of images of a speaking person and having a first frame rate, characterized by: monitoring means for monitoring an audio signal temporally corresponding to the sequence of images to detect speech segments uttered by the speaking person, at least some detected speech segments corresponding to images of the image sequence that are not represented by the received video signal; means for associating detected speech segments with stored parametric mouth formation data; and frame generating means, responsive to said monitoring means and said associating means, for generating at least one synthesized frame representative of an image of the speaking person and for inserting said at least one synthesized frame between adjacent frames of the received video signal to thereby provide a video signal having a frame rate higher than said first frame rate.
Claim 2
Independent Claim
149 words
A method of increasing the frame rate of a received video signal representative of a sequence of images of a speaking person and having a first frame rate, characterized by the steps of: monitoring an audio signal temporally corresponding to the sequence of images to detect speech segments uttered by the speaking person, at least some detected speech segments corresponding to imagesof the image sequence that are not represented by the received video signal; associating a detected speech segment with stored parametric mouth formation data; synthesizing, from at least one frame of the received video signal and parametric mouth formation data identified during said associating step, a frame representative of an image of the speaker; and inserting a frame obtained during said synthesizing step between adjacent frames of the received video signal to thereby obtain a second video signal having a frame rate higher than said first frame rate.
Claim 3
Dependent Claim
27 words
The apparatus according to claim 1, characterized in that each said detected speech segment corresponds to a respective mouth formation represented by said parametric mouth formation data.
Claim 4
Dependent Claim
27 words
The apparatus according to claim 1 or the method according to claim 2, characterized in that each said detected speech segment is a phoneme, or a homophene.
Claim 5
Dependent Claim
17 words
The apparatus according to claim 3, characterized in that each said speaker mouth formation is a viseme.
Claim 6
Dependent Claim
92 words
The apparatus according to claim 3, characterized in that said associating means includes either a memory having stored therein a table of mouth formation parameters, said mouth formation parameters representing mouth shapes of a generic human speaker uttering predetermined speech segments, or a feature extraction means responsive to said monitoring means for extracting mouth formation parameters from frames of said received video signal as speech segments are detected by said monitoring means, each stored mouth formation parameter being representative of the mouth shape of the speaking person uttering a predetermined speech segment.
Claim 7
Dependent Claim
31 words
The apparatus according to claim 1, characterized in that said frame generating means is operative to synthesize a plurality of frames for insertion between adjacent frames of the received video signal.
Claim 8
Dependent Claim
41 words
The apparatus according to claim 1, characterized in that said associating means is operative to associate detected speech segments uttered by the speaking person with stored shapes of at least one of the jaw, tongue, and teeth of the speaking person.
Claim 9
Dependent Claim
32 words
The method of claim 2, characterized in that a plurality of frames are sequentially synthesized during said synthesizing step, said synthesized frames representing images of the speaking person while making said utterances.
Claim 10
Dependent Claim
41 words
The method according to claim 2, characterized in that said associating step comprises addressing a stored table of parametric facial feature data containing mouth formation parameters, said mouth formation parameters representing mouth shapes of a generic speaker uttering predetermined speech segments.
Claim 11
Dependent Claim
148 words
A video teleconferencing apparatus for transmitting and receiving digital audio and video data signals, characterized by: means for receiving very low frame rate video and audio signals representative of a sequence of images of a speaking person and of temporally corresponding utterances of the speaking person, respectively; frame generating means responsive to an audio signal portion temporally corresponding to an image of the sequence not represented by the low frame rate video signal and to a video signal portion indicative of at least one frame for synthesizing a frame representative of an image of the speaking person; means for associating respective portions of said audio signal with stored facial feature information; means for inserting a frame synthesized by said generating means between consecutive received frames of said low frame rate video signal to form a higher frame rate sequence; and means for displaying said higher frame rate sequence.
Claim 12
Dependent Claim
22 words
The apparatus according to claim 1 or 11, characterized in that said first frame rate is less than ten frames per second.
Claim 13
Dependent Claim
29 words
The apparatus according to claim 1, further comprising feature extraction means responsive to the received video signal for extracting parametric mouth formation data from a plurality of frames thereof.
References Cited (Backward Citations)
No citation records indexed.
USPTO Annuity & Maintenance Windows
Maintenance schedule is calculated for granted US utility patents.