US6892193B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for inducing classifiers for multimedia based on unified representation of features reflecting disparate modalities
Assignee: International Business Machines Corp
•
Filing: 2001-05-10
•
Granted: 2005-05-10
•
Expires: 2021-05-10
Abstract
Abstract This invention is a system and method to perform categorization (classification) of multimedia items. These items are comprised of a multitude of disparate information sources, in particular, visual information and textual information. Classifiers are induced based on combining textual and visual feature vectors. Textual features are the traditional ones, such as, word count vectors. Visual features include, but are not limited to, color properties of key intervals and motion properties of key intervals. The visual feature vectors are determined in such a fashion that the vectors are sparse. The vector components are features such as the absence or presence of the color green in spatial regions and the absence or the amount of visual flow in spatial regions of the media items. The text and the visual representation vectors are combined in a systematic and coherent fashion. This vector representation of a media item lends itself to well-established learning techniques. The resulting system, subject of this invention, categorizes (or classifies) media items based both on textual features and visual features.
Inventors
- 👤 Rudolf M. Bolle
- 👤 Norman Haas
- 👤 Frank J. Oles
- 👤 Tong Zhang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-25 00:38
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/70; G06F16/78; G06F16/783; G06F16/7844; G06V; G06V20/00; G06V20/40; G06F16/30; G06F16/35; G06F16/353; G06F16/50; G06F16/58; G06F16/583; G06F16/5846; G06F16/60; G06F16/68; G06F16/683; G06F16/685; G06F16/7834; G06F16/7847; G06F16/785; G06F16/786
Claims Analysis
Total Claims: 29
Claim 1
Independent Claim
106 words
A computer system having one or more memories and one or more central processing units (CPUs), the system comprising: one or more multimedia items, stored in the memories, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; and a combining process that creates a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities, and concatenates for each of the one or more multimedia items the visual feature vectors and the textual feature vectors into a unified feature vector.
Claim 2
Dependent Claim
106 words
A computer system having one or more memories and one or more central processing units (CPUs), the system comprising: one or more multimedia items, stored in the memories, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; and a combining process that creates a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities, and concatenates for each of the one or more multimedia items the visual feature vectors and the textual feature vectors into a unified feature vector.
Claim 3
Dependent Claim
22 words
A system, as in claim 1 , further comprising a classifier induction process that induces a classifier from the unified feature vectors.
Claim 4
Dependent Claim
30 words
A system, as in claim 2 , where the classifiers include any one or more of the following: a hyperplane classifier, a rule-based classifier, a Bayesian classifier, maximum likelihood classifier.
Claim 5
Dependent Claim
73 words
A system, as in claim 1 , further comprises: one or more classifiers having one or more classes; and an application process that for each of the multimedia items, uses the classifiers to predict zero or more of the classes to which the respective multimedia items belong, the multimedia items being unprocessed multimedia items, and where in the case that zero categories are predicted the multimedia item does not belong to any class.
Claim 6
Dependent Claim
58 words
A system, as in claim 1 , further comprising a transformation process that transforms one or more feature vectors in the set of visual feature vectors and textual feature vectors in order to make one or of more the visual feature vectors compatible with one or more of the textual feature vectors for the all the multimedia items.
Claim 7
Dependent Claim
32 words
A system, as in claim 5 , where the visual feature vectors and textual feature vectors are made compatible by limiting the component values in the respective visual and textual feature vectors.
Claim 8
Dependent Claim
43 words
A system, as in claim 6 , where the component values include: a binary value; a one bit binary value; a 0, 1, 2 or many value; a value in a range; a discrete value; and a 0, 1, 2, or 3 value.
Claim 9
Dependent Claim
32 words
A system, as in claim 5 , where the visual feature vectors and textual feature vectors are made compatible by limiting a difference between magnitudes of the visual and textual feature vectors.
Claim 10
Dependent Claim
22 words
A system, as in claim 8 , where the difference in magnitudes is limited by normalizing the visual and textual feature vectors.
Claim 11
Dependent Claim
32 words
A system, as in claim 5 , where the visual feature vectors and textual feature vectors are made compatible by limiting the difference between the number of components in the respective vectors.
Claim 12
Dependent Claim
80 words
A system, as in claim 1 , where the visual feature vectors comprise one or more of the following: a set of ordered components, a set of unordered components, a set of only temporally ordered components, a set of only spatially ordered components, a set of temporally and spatially ordered components, a set of visual features extracted from ordered key intervals, a set of visual features extracted from ordered key intervals divided into regions, and a set of semantic features.
Claim 13
Dependent Claim
27 words
A system, as in claim 1 , where the visual feature vectors have a fixed length, the fixed length being independent of length of the multimedia items.
Claim 14
Dependent Claim
28 words
A system, as in claim 1 , where the visual feature vectors comprise one or more components that are selected so that the visual feature vectors are sparse.
Claim 15
Dependent Claim
54 words
A system, as in claim 1 , where the visual feature vectors represent any one or more of the following: a color, a motion, a visual texture, an optical flow, a semantic meaning, semantic meanings derived from one or more video streams, an edge density, a hue, an amplitude, a frequency, and a brightness.
Claim 16
Dependent Claim
54 words
A system, as in claim 1 , where the textual feature vectors are derived from any one or more of the following: close captions, open captions, captions, speech recognition applied to one or more audio inputs, semantic meanings derived from one or more audio streams, and global text information associated with a multimedia item.
Claim 17
Dependent Claim
199 words
A computer system having one or more memories and one or more central processing units (CPUs), the system comprising: one or more multimedia items, stored in the memories, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; a block process that divides the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; a combining process that creates a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities, and concatenates for each of the blocks the visual feature vectors and the textual feature vectors into a unified feature vector; one or more classifiers having one or more classes; an application process that for each of the blocks, uses the classifiers to determine zero or more of the classes to which the respective blocks belong; and a segmentation process that finds temporally contiguous groups of the blocks and combines the contiguous groups into media segments where all the blocks in the media segment have one or more of the same classes.
Claim 18
Dependent Claim
199 words
A computer system having one or more memories and one or more central processing units (CPUs), the system comprising: one or more multimedia items, stored in the memories, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; a block process that divides the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; a combining process that creates a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities, and concatenates for each of the blocks the visual feature vectors and the textual feature vectors into a unified feature vector; one or more classifiers having one or more classes; an application process that for each of the blocks, uses the classifiers to determine zero or more of the classes to which the respective blocks belong; and a segmentation process that finds temporally contiguous groups of the blocks and combines the contiguous groups into media segments where all the blocks in the media segment have one or more of the same classes.
Claim 19
Dependent Claim
43 words
A system, as in claim 16 , further comprising an aggregation process that aggregates two or more of the media segments belonging to the same class with one or more media segments of a different class according to one or more aggregation rules.
Claim 20
Dependent Claim
37 words
A system, as in claim 17 , where the aggregation rules include any one or more of the following rule types: segment region rules, segment boundary indicator rules, and learned rules that are derived from training data.
Claim 21
Dependent Claim
37 words
A system, as in claim 18 , where the segment region rule has a minimum segment length constraint and a plurality of rules that change small sequences of blocks of varying categorization into blocks of equal category.
Claim 22
Dependent Claim
49 words
A system, as in claim 18 , where the segment boundary indicator rules are multimedia cues and these multimedia cues are one or more of the following: a shot transition, an audio silence, a speaker change, an end-of-sentence in speech transcript, and a topic change indicator in the closed-caption.
Claim 23
Dependent Claim
53 words
A system, as in claim 18 , where the learned rules are the costs of transitions and the aggregations process aggregates two or more of the media segments belonging to the same class with one or more media segments of a different class by minimizing the overall cost of the sequence of segments.
Claim 24
Independent Claim
139 words
A method for segmenting multimedia streams comprising the steps of: storing one or more multimedia items in one or more memories of computer, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; dividing the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; for each block, creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities; for each block, concatenating the visual feature vectors and the textual feature vectors into a unified feature vector; categorizing each of the blocks by categorizing the respective unified feature vector; and assembling two or more of the categorized blocks into a segment.
Claim 25
Independent Claim
139 words
A method for segmenting multimedia streams comprising the steps of: storing one or more multimedia items in one or more memories of computer, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; dividing the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; for each block, creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities; for each block, concatenating the visual feature vectors and the textual feature vectors into a unified feature vector; categorizing each of the blocks by categorizing the respective unified feature vector; and assembling two or more of the categorized blocks into a segment.
Claim 26
Dependent Claim
140 words
A memory storing a program, the program comprising the steps of: storing one or more multimedia items in one or more memories of computer, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; dividing the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; for each block, creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities; for each block, concatenating the visual feature vectors and the textual feature vectors into a unified feature vector; categorizing each of the blocks by categorizing the respective unified feature vector; and assembling two or more of the categorized blocks into a segment.
Claim 27
Dependent Claim
140 words
A memory storing a program, the program comprising the steps of: storing one or more multimedia items in one or more memories of computer, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; dividing the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; for each block, creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities; for each block, concatenating the visual feature vectors and the textual feature vectors into a unified feature vector; categorizing each of the blocks by categorizing the respective unified feature vector; and assembling two or more of the categorized blocks into a segment.
Claim 28
Independent Claim
148 words
A system for segmenting multimedia streams comprising: means for storing one or more multimedia items in one or more memories of computer, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; means for dividing the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; means for creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities, block by block; means for concatenating the visual feature vectors and the textual feature vectors into a unified feature vector, block by block; means for categorizing each of the blocks by categorizing the respective unified feature vector; and means for assembling two or more of the categorized blocks into a segment.
Claim 29
Independent Claim
148 words
A system for segmenting multimedia streams comprising: means for storing one or more multimedia items in one or more memories of computer, each multimedia item having two or more disparate modalities, the disparate modalities being at least one or more visual modalities and one or more textual modalities; means for dividing the multimedia items into blocks of one or more key intervals, each key interval having one more frames of the multimedia items; means for creating a visual feature vector for each of the visual modalities and a textual feature vector for each of the textual modalities, block by block; means for concatenating the visual feature vectors and the textual feature vectors into a unified feature vector, block by block; means for categorizing each of the blocks by categorizing the respective unified feature vector; and means for assembling two or more of the categorized blocks into a segment.
References Cited (Backward Citations)
US5151788A
Cited
US5428733A
Cited
US5434965A
Cited
US5696866A
Cited
US5343251A
Cited
US6173275B1
Cited
US5794178A
Cited
US5708477A
Cited
US6138130A
Cited
US5924077A
Cited
US5983237A
Cited
US6055548A
Cited
US6055043A
Cited
US6363380B1
Cited
US6611842B1
Cited
US20020004793A1
Cited
US20020161747A1
Cited
US20020147782A1
Cited
US20030033347A1
Cited
US20030004966A1
Cited
US20030028534A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2008-05-09 | 2008-11-08 | 2009-05-10 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2012-05-09 | 2012-11-08 | 2013-05-10 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2016-05-09 | 2016-11-08 | 2017-05-10 | $7700.00 | Window Closed / Lapsed |