US6397181B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for voice annotation and retrieval of multimedia data
Assignee: National University of Singapore
•
Filing: 1999-01-27
•
Granted: 2002-05-28
•
Expires: 2019-01-27
Abstract
Abstract A method, an apparatus, a computer program product and a system for voice annotating and retrieving digital media content are disclosed. An annotation module (420) post annotates digital media data (410), including audio, image and/or video data, with speech. A word lattice (222) can be created from speech annotation (210) dependent upon acoustic and/or linguistic knowledge. An indexing module (430) then indexes the speech-annotated data (422). The word lattice (222) is reverse indexed (230), and content addressing (240) is applied to produce the indexed data (432, 242). A speech query (474) can be generated as input to a retrieval module (480) for retrieving a segment of the indexed digital media data (432). The speech query (474, 310) is converted into a word lattice (322), and a shortlist (344) is produced from it (322) by confidence filtering (330). The shortlist (344) is input to a lattice search engine (350) to search the indexed content (342) to obtain the search result (352).
Inventors
- 👤 Haizhou Li
- 👤 Jiankang Wu
- 👤 Arcot Desai Narasimhalu
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-04 18:09
CPC Classifications
G; G10; G10L; G10L15/00; G10L15/08; G10L15/18; G10L15/183; G10L15/19; G10L15/197
Claims Analysis
Total Claims: 81
Claim 1
Independent Claim
66 words
A method of voice annotating source digital media data, said method including the steps of: speech annotating one or more portions of said source digital media data with a speech annotation that is independent of the source digital media data thereby providing a speech annotated digital media data; and indexing said speech annotated digital media data by said speech annotation to provide an indexed media content.
Claim 2
Independent Claim
66 words
A method of voice annotating source digital media data, said method including the steps of: speech annotating one or more portions of said source digital media data with a speech annotation that is independent of the source digital media data thereby providing a speech annotated digital media data; and indexing said speech annotated digital media data by said speech annotation to provide an indexed media content.
Claim 3
Dependent Claim
23 words
The method according to claim 1 , wherein said indexing step includes the step of creating a word lattice from said speech annotation.
Claim 4
Dependent Claim
26 words
The method according to claim 2 , wherein said step of creating said word lattice is dependent upon at least one of acoustic and linguistic knowledge.
Claim 5
Dependent Claim
23 words
The method according to claim 2 , further including the step of reverse indexing said word lattice to provide a reverse index table.
Claim 6
Dependent Claim
18 words
The method according to claim 4 , further including the step of content addressing said reverse index table.
Claim 7
Dependent Claim
25 words
The method according to claim 1 , wherein said annotating step is dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 8
Dependent Claim
79 words
The method according to claim 1 , wherein said indexing step further includes the steps of: applying said speech annotation to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and applying said reversed index table to an addressing engine module to create the indexed media content.
Claim 9
Dependent Claim
18 words
The method according to claim 7 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 10
Dependent Claim
17 words
The method according to claim 7 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 11
Independent Claim
67 words
An apparatus for voice annotating source digital media data, said apparatus including: means for speech annotating one or more portions of said source digital media data with a speech annotation that is independent of the source digital media data to provide a speech annotated digital media data; and means for indexing said speech annotated digital media data by said speech annotation to provide an indexed media content.
Claim 12
Independent Claim
67 words
An apparatus for voice annotating source digital media data, said apparatus including: means for speech annotating one or more portions of said source digital media data with a speech annotation that is independent of the source digital media data to provide a speech annotated digital media data; and means for indexing said speech annotated digital media data by said speech annotation to provide an indexed media content.
Claim 13
Dependent Claim
22 words
The apparatus according to claim 10 , wherein said indexing means includes means for creating a word lattice from said speech annotation.
Claim 14
Dependent Claim
26 words
The apparatus according to claim 11 , wherein said means for creating said word lattice is dependent upon at least one of acoustic and linguistic knowledge.
Claim 15
Dependent Claim
21 words
The apparatus according to claim 11 , further including means for reverse indexing said-word lattice to provide a reverse index table.
Claim 16
Dependent Claim
17 words
The apparatus according to claim 13 , further including means for content addressing said reverse index table.
Claim 17
Dependent Claim
25 words
The apparatus according to claim 10 , wherein said annotating means is dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 18
Dependent Claim
82 words
The apparatus according to claim 10 , wherein said means for indexing further includes: means for applying said speech annotation to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; means applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and means for applying said reversed index table to an addressing engine module to create the indexed media content.
Claim 19
Dependent Claim
18 words
The apparatus according to claim 16 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 20
Dependent Claim
17 words
The apparatus according to claim 16 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 21
Dependent Claim
82 words
A computer program product having a computer readable medium having a computer program recorded therein for voice annotating source digital media data, said computer program product including: means for speech annotating one or more portions of said source digital media data with a speech annotation that is independent of the source digital media data to provide a speech annotated digital media data; and means for indexing said speech annotated digital media data by said speech annotation to provide an indexed media content.
Claim 22
Dependent Claim
82 words
A computer program product having a computer readable medium having a computer program recorded therein for voice annotating source digital media data, said computer program product including: means for speech annotating one or more portions of said source digital media data with a speech annotation that is independent of the source digital media data to provide a speech annotated digital media data; and means for indexing said speech annotated digital media data by said speech annotation to provide an indexed media content.
Claim 23
Dependent Claim
24 words
The computer program product according to claim 19 , wherein said indexing means includes means for creating a word lattice from said speech annotation.
Claim 24
Dependent Claim
28 words
The computer program product according to claim 19 , wherein said means for creating said word lattice is dependent upon at least one of acoustic and linguistic knowledge.
Claim 25
Dependent Claim
19 words
The computer program product according to claim 20 , further including means for content addressing said reverse index table.
Claim 26
Dependent Claim
24 words
The computer program product according to claim 20 , further including means for reverse indexing said word lattice to provide a reverse index table.
Claim 27
Dependent Claim
27 words
The computer program product according to claim 19 , wherein said annotating means is dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 28
Dependent Claim
84 words
The computer program product according to claim 19 , wherein said means for indexing further includes: means for applying said speech annotation to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; means applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and means for applying said reversed index table to an addressing engine module to create the indexed media content.
Claim 29
Dependent Claim
20 words
The computer program product according to claim 25 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 30
Dependent Claim
19 words
The computer program product according to claim 26 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 31
Independent Claim
76 words
A method of voice retrieving digital media data annotated with speech, said method including the steps of: providing an indexed digital media data, said indexed digital media data derived from a word lattice created from a speech annotation of said digital media data, wherein said speech annotation is independent of a source digital media data; generating a speech query; and retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 32
Independent Claim
76 words
A method of voice retrieving digital media data annotated with speech, said method including the steps of: providing an indexed digital media data, said indexed digital media data derived from a word lattice created from a speech annotation of said digital media data, wherein said speech annotation is independent of a source digital media data; generating a speech query; and retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 33
Dependent Claim
20 words
The method according to claim 28 , further including the step of creating a word lattice from said speech query.
Claim 34
Dependent Claim
46 words
The method according to claim 29 , further including the step of searching said indexed digital media data dependent upon said speech query by matching said word lattice created from said speech query with word lattices of said speech annotation of said indexed digital media data.
Claim 35
Dependent Claim
29 words
The method according to claim 30 , further including the step of confidence filtering said lattice created from said speech query to produce a short-list for said searching step.
Claim 36
Dependent Claim
23 words
The method according to claim 29 , wherein said word lattice is created dependent upon at least one of acoustic and linguistic knowledge.
Claim 37
Dependent Claim
18 words
The method according to claim 32 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 38
Dependent Claim
17 words
The method according to claim 32 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 39
Dependent Claim
23 words
The method according to claim 28 , further including the step of searching said indexed digital media data dependent upon a text query.
Claim 40
Dependent Claim
26 words
The method according to claim 28 , wherein said speech query is generated dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 41
Independent Claim
79 words
An apparatus for voice retrieving digital media data annotated with speech, said apparatus including: means for providing an indexed digital media data, said indexed digital media data derived from a word lattice created from a speech annotation of said digital media data, wherein said speech annotation is independent of a source digital media data; means for generating a speech query; and means for retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 42
Independent Claim
79 words
An apparatus for voice retrieving digital media data annotated with speech, said apparatus including: means for providing an indexed digital media data, said indexed digital media data derived from a word lattice created from a speech annotation of said digital media data, wherein said speech annotation is independent of a source digital media data; means for generating a speech query; and means for retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 43
Dependent Claim
19 words
The apparatus according to claim 37 , further including means for creating a word lattice from said speech query.
Claim 44
Dependent Claim
45 words
The apparatus according to claim 38 , further including means for searching said indexed digital media data dependent upon said speech query by matching said word lattice created from said speech query with word lattices of said speech annotation of said indexed digital media data.
Claim 45
Dependent Claim
28 words
The apparatus according to claim 39 , further including means for confidence filtering said lattice created from said speech query to produce a short-list for said searching means.
Claim 46
Dependent Claim
23 words
The apparatus according to claim 38 , wherein said word lattice is created dependent upon at least one of acoustic and linguistic knowledge.
Claim 47
Dependent Claim
18 words
The apparatus according to claim 41 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 48
Dependent Claim
17 words
The apparatus according to claim 41 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 49
Dependent Claim
22 words
The apparatus according to claim 37 , further including means for searching said indexed digital media data dependent upon a text query.
Claim 50
Dependent Claim
26 words
The apparatus according to claim 37 , wherein said speech query is generated dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 51
Dependent Claim
94 words
An computer program product having a computer readable medium having a computer program recorded therein for voice retrieving digital media data annotated with speech, said computer program product including: means for providing an indexed digital media data, said indexed digital media data derived from a word lattice created from a speech annotation of said digital media data, wherein said speech annotation is independent of a source digital media data; means for generating a speech query; and means for retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 52
Dependent Claim
94 words
An computer program product having a computer readable medium having a computer program recorded therein for voice retrieving digital media data annotated with speech, said computer program product including: means for providing an indexed digital media data, said indexed digital media data derived from a word lattice created from a speech annotation of said digital media data, wherein said speech annotation is independent of a source digital media data; means for generating a speech query; and means for retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 53
Dependent Claim
21 words
The computer program product according to claim 46 , further including means for creating a word lattice from said speech query.
Claim 54
Dependent Claim
47 words
The computer program product according to claim 47 , further including means for searching said indexed digital media data dependent upon said speech query by matching said word lattice created from said speech query with word lattices of said speech annotation of said indexed digital media data.
Claim 55
Dependent Claim
30 words
The computer program product according to claim 48 , further including means for confidence filtering said lattice created from said speech query to produce a short-list for said searching means.
Claim 56
Dependent Claim
25 words
The computer program product according to claim 47 , wherein said word lattice is created dependent upon at least one of acoustic and linguistic knowledge.
Claim 57
Dependent Claim
20 words
The computer program product according to claim 50 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 58
Dependent Claim
19 words
The computer program product according to claim 50 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 59
Dependent Claim
24 words
The computer program product according to claim 46 , further including means for searching said indexed digital media data dependent upon a text query.
Claim 60
Dependent Claim
28 words
The computer program product according to claim 46 , wherein said speech query is generated dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 61
Independent Claim
95 words
A system for voice annotating and retrieving source digital media data, said system including: means for speech annotating at least one segment of said source digital media data with a speech annotation that is independent of the source digital media data to provide a speech annotated digital media data; means for indexing said speech-annotated digital media data by said speech annotation to provide an indexed digital media data; means for generating a speech or voice query; and means for retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 62
Independent Claim
95 words
A system for voice annotating and retrieving source digital media data, said system including: means for speech annotating at least one segment of said source digital media data with a speech annotation that is independent of the source digital media data to provide a speech annotated digital media data; means for indexing said speech-annotated digital media data by said speech annotation to provide an indexed digital media data; means for generating a speech or voice query; and means for retrieving one or more portions of said indexed digital media data dependent upon said speech query.
Claim 63
Dependent Claim
19 words
The system according to claim 55 , further including means for creating a lattice structure from said speech annotation.
Claim 64
Dependent Claim
26 words
The system according to claim 56 , wherein said means for creating said lattice structure is dependent upon at least one of acoustic and linguistic knowledge.
Claim 65
Dependent Claim
18 words
The system according to claim 57 , wherein said acoustic knowledge is based on a hidden Markov model.
Claim 66
Dependent Claim
17 words
The system according to claim 57 , wherein said linguistic knowledge is an N-gram statistical linguistic model.
Claim 67
Dependent Claim
22 words
The system according to claim 56 , further including means for reverse indexing said lattice structure to provide a reverse index table.
Claim 68
Dependent Claim
17 words
The system according to claim 60 , further including means for content addressing said reverse index table.
Claim 69
Dependent Claim
17 words
The system according to claim 55 , wherein said speech-annotating means post-annotates said source digital media data.
Claim 70
Dependent Claim
19 words
The system according to claim 55 , further including means for creating a lattice structure from said speech query.
Claim 71
Dependent Claim
45 words
The system according to claim 63 , further including means for searching said indexed digital media data dependent upon said speech query by matching said lattice structure created from said speech query with lattice structures of said speech annotation of said indexed digital media data.
Claim 72
Dependent Claim
29 words
The system according to claim 64 , further including means for confidence filtering said lattice structure created from said speech query to produce a short-list for said searching means.
Claim 73
Dependent Claim
23 words
The system according to claim 63 , wherein said lattice structure is created dependent upon at least one of acoustic and linguistic knowledge.
Claim 74
Dependent Claim
22 words
The system according to claim 55 , further including means for searching said indexed digital media data dependent upon a text query.
Claim 75
Dependent Claim
33 words
The system according to claim 55 , wherein at least one of said annotating means and said speech query is dependent upon at least one of a customised vocabulary and Backus-Naur Form grammar.
Claim 76
Independent Claim
99 words
A method of voice annotating source digital media data, said method comprising the steps of: speech annotating, independently of the source media data, one or more portions of said source media data using a formal spoken language; applying said annotated speech to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and applying said reversed index table to an addressing engine module to create indexed media content.
Claim 77
Independent Claim
99 words
A method of voice annotating source digital media data, said method comprising the steps of: speech annotating, independently of the source media data, one or more portions of said source media data using a formal spoken language; applying said annotated speech to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and applying said reversed index table to an addressing engine module to create indexed media content.
Claim 78
Independent Claim
102 words
An apparatus for voice annotating source digital media data, comprising: means for speech annotating, independently of the source media data, one or more portions of said source media data using a formal spoken language; means for applying said annotated speech to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; means for applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and means for applying said reversed index table to an addressing engine module to create indexed media content.
Claim 79
Independent Claim
102 words
An apparatus for voice annotating source digital media data, comprising: means for speech annotating, independently of the source media data, one or more portions of said source media data using a formal spoken language; means for applying said annotated speech to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; means for applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and means for applying said reversed index table to an addressing engine module to create indexed media content.
Claim 80
Dependent Claim
119 words
A computer program product having a computer readable medium having a computer program recorded therein for voice annotating source digital media data, said computer program product including: means for speech annotating, independently of the source media data, one or more portions of said source media data using a formal spoken language; means for applying said annotated speech to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; means for applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and means for applying said reversed index table to an addressing engine module to create indexed media content.
Claim 81
Dependent Claim
119 words
A computer program product having a computer readable medium having a computer program recorded therein for voice annotating source digital media data, said computer program product including: means for speech annotating, independently of the source media data, one or more portions of said source media data using a formal spoken language; means for applying said annotated speech to a lattice engine and further applying at least one of acoustic and linguistic knowledge to said lattice engine to generate a word lattice; means for applying said word lattice to a reverse index engine to build a reversed index table for said lattice; and means for applying said reversed index table to an addressing engine module to create indexed media content.
References Cited (Backward Citations)
US5835667A
Cited
US6185527B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2005-05-27 | 2005-11-26 | 2006-05-28 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2009-05-27 | 2009-11-26 | 2010-05-28 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2013-05-27 | 2013-11-26 | 2014-05-28 | $7700.00 | Window Closed / Lapsed |