US6704029B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for specifying scene information in a moving picture
Assignee: Canon Inc
•
Filing: 2000-04-11
•
Granted: 2004-03-09
•
Expires: 2020-04-11
Abstract
Abstract In an image processing apparatus, a moving picture storage module stores moving picture data. A scene designation module designates at least one scene in the moving picture data, and sets a key frame and importance level of the designated scene. A frame feature extraction module acquires an image feature amount of the designated key frame. A scene feature information storage module generates scene feature information including the duration, key frame, importance level, and image feature amount of the key frame of the scene designated by the scene designation module, and stores the generated information in a memory in correspondence with the moving picture data. Based on the scene feature information, digest playback for playing back only principal part of the designated moving picture data is executed, or desired moving picture data is found by a similar image search using the key frame.
Inventors
- 👤 Kazuyo Ikeda
- 👤 Hirotaka Shiiyama
- 👤 Hiroshi Tojo
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-24 15:54
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/70; G06F16/78; G06F16/783; G06F16/7847; G06F16/785; G06F16/73; G06F16/738; G06F16/739
Claims Analysis
Total Claims: 60
Claim 1
Independent Claim
89 words
An image processing apparatus comprising: specifying means for specifying at least one desired scene in moving picture data, and a frame serving as a key frame in the desired scene; setting means for setting an importance level of the desired scene specified by said specifying means, with respect to the entire moving picture; and storage means for storing information indicating the scene and key frame specified by said specifying means and the importance level set by said setting means as scene information in association with the moving picture data.
Claim 2
Dependent Claim
89 words
An image processing apparatus comprising: specifying means for specifying at least one desired scene in moving picture data, and a frame serving as a key frame in the desired scene; setting means for setting an importance level of the desired scene specified by said specifying means, with respect to the entire moving picture; and storage means for storing information indicating the scene and key frame specified by said specifying means and the importance level set by said setting means as scene information in association with the moving picture data.
Claim 3
Dependent Claim
44 words
The apparatus according to claim 1 , wherein the information indicating the scene specified by said specifying means includes frame identifiers of start and end frames of the scene, and the information indicating the key frame includes a frame identifier of the key frame.
Claim 4
Dependent Claim
26 words
The apparatus according to claim 2 , wherein the frame identifier is a frame number assigned in turn from the beginning of the moving picture data.
Claim 5
Dependent Claim
25 words
The apparatus according to claim 2 , wherein the frame identifier is a time from the beginning to that frame of the moving picture data.
Claim 6
Dependent Claim
45 words
The apparatus according to claim 1 , further comprising: acquisition means for acquiring feature amount information of the key frame, and wherein the scene information stored in said storage means includes the feature amount information of the key frame image acquired by said acquisition means.
Claim 7
Dependent Claim
56 words
The apparatus according to claim 5 , wherein the feature amount information of the key frame is a label sequence obtained by segmenting an image of the key frame into a plurality of blocks, assigning labels in accordance with feature amounts acquired from the individual segmented blocks, and arranging the assigned labels in a predetermined order.
Claim 8
Dependent Claim
30 words
The apparatus according to claim 6 , wherein the image is segmented into a plurality of vertical and horizontal blocks, the numbers of which are given by powers of 2.
Claim 9
Dependent Claim
38 words
The apparatus according to claim 6 , further comprising means for storing information that pertains to the number of segmented blocks, a feature amount extraction method, and a color expression of an image together with the label sequence.
Claim 10
Dependent Claim
86 words
The apparatus according to claim 5 , further comprising: computation means for computing a similarity value between images of a designated key frame and another key frame on the basis of image feature amounts of the key frames when one key frame is designated; presentation means for presenting key frames on the basis of similarity values computed by said computation means; and playback means for acquiring and playing back moving picture data including the key frame selected from the key frames presented by said presentation means.
Claim 11
Dependent Claim
38 words
The apparatus according to claim 1 , further comprising generation means for generating extracted moving picture data, which can be played back as a moving picture, on the basis of the scene information stored in said storage means.
Claim 12
Dependent Claim
31 words
The apparatus according to claim 10 , wherein said generation means generates the extracted moving picture data by time-serially merging scenes indicated by the scene information stored in said storage means.
Claim 13
Dependent Claim
44 words
The apparatus according to claim 10 , wherein said generation means generates the extracted moving picture data by time-serially merging scenes indicated by the scene information stored in said storage means, and opening and ending scenes of the moving picture for a predetermined time.
Claim 14
Dependent Claim
50 words
The apparatus according to claim 10 , further comprising: designation means for designating a playback time of the extracted moving picture data; and adjustment means for adjusting durations of scenes to be merged by s aid generation means on the basis of the playback time designated by said designation means.
Claim 15
Dependent Claim
106 words
The apparatus according to claim 13 , wherein said adjustment means comprises: frame extraction means for determining importance levels of frames in a scene on the basis of an importance level set in the scene, extracting frames having importance levels that exceed a threshold value from the scene, and providing the extracted frames as a scene to be merged to said generation means; and threshold value control means for controlling the threshold value used in said frame extraction means to make the playback time of the extracted moving picture data generated by said generation means fall within a predetermined error range of the designated playback time.
Claim 16
Dependent Claim
64 words
The apparatus according to claim 14 , further comprising extension adjustment means for extending the playback time as needed using the key frame of each scene as the center to fall within a predetermined error range of the designated playback time when a total time of all scenes designated by the scene information is shorter than the playback time designated by said designation means.
Claim 17
Dependent Claim
35 words
The apparatus according to claim 10 , further comprising removing means for removing a scene, having a playback time which is shorter than a predetermined time, from scenes that form the extracted moving picture data.
Claim 18
Dependent Claim
31 words
The apparatus according to claim 16 , further comprising extending means for extending another scene by a time corresponding to the playback time of the scene removed by said removing means.
Claim 19
Dependent Claim
43 words
The apparatus according to claim 17 , wherein said extending means extends a time of a scene set with a higher importance level of previous and next scenes which neighbor the scene removed by said removing means in the extracted moving picture data.
Claim 20
Dependent Claim
97 words
The apparatus according to claim 10 , further comprising: designation means for designating a playback time of the extracted moving picture data; and extraction means for extracting scenes for a predetermined time duration by selecting scenes indicated by the scene information stored in said storage means in descending order of importance levels, and wherein said extraction means executes a process until a total time of the extracted scenes exceeds the playback time designated by said designation means, and said generation means generates the extracted moving picture data by time-serially merging the scenes extracted by said extraction means.
Claim 21
Dependent Claim
64 words
The apparatus according to claim 19 , further comprising extension adjustment means for extending the playback time as needed using the key frame of each scene as the center to fall within a predetermined error range of the designated playback time when a total time of all scenes designated by the scene information is shorter than the playback time designated by said designation means.
Claim 22
Dependent Claim
118 words
The apparatus according to claim 10 , wherein the scene information includes an image feature amount of an image of each key frame, and said apparatus further comprises: computation means for computing a similarity value between images of a designated key frame and another key frame on the basis of image feature amounts of the key frames when one key frame is designated; presentation means for presenting key frames on the basis of similarity values computed by said computation means; and playback means for playing back the extracted moving picture data generated by said generation means in association with the moving picture data which includes the key frame selected from the key frames presented by said presentation means.
Claim 23
Dependent Claim
56 words
The apparatus according to claim 21 , wherein the feature amount information of the key frame is a label sequence obtained by segmenting an image of the key frame into a plurality of blocks, assigning labels in accordance with feature amounts acquired from the individual segmented blocks, and arranging the assigned labels in a predetermined order.
Claim 24
Dependent Claim
115 words
The apparatus according to claim 10 , wherein the scene information includes an image feature amount of an image of each key frame, said apparatus further comprises: first means for randomly displaying key frames; second means for playing back the extracted moving picture data generated by said generation means; third means for mal king a similarity search by computing a similarity value between images of the designated key frame and another key frame on the basis of image feature amounts of the key frames; and fourth means for playing back a moving picture on the basis of the designated key frame, and said first to fourth means can operate to function in an arbitrary combination.
Claim 25
Dependent Claim
21 words
The apparatus according to claim 1 , further comprising display means for displaying a list of thumbnails of key frame images.
Claim 26
Dependent Claim
95 words
An image processing method comprising: the specifying step of specifying at least one desired scene in moving picture data, and a frame serving as a key frame in the desired scene; the setting step of setting an importance level of the desired scene specified in the specifying step, with respect to the entire moving picture; and the storage step of storing information indicating the scene and key frame specified in the specifying step and the importance level set in the setting step as scene information in storage means in association with the moving picture data.
Claim 27
Dependent Claim
95 words
An image processing method comprising: the specifying step of specifying at least one desired scene in moving picture data, and a frame serving as a key frame in the desired scene; the setting step of setting an importance level of the desired scene specified in the specifying step, with respect to the entire moving picture; and the storage step of storing information indicating the scene and key frame specified in the specifying step and the importance level set in the setting step as scene information in storage means in association with the moving picture data.
Claim 28
Dependent Claim
44 words
The method according to claim 25 , wherein the information indicating the scene specified in the specifying step includes frame identifiers of start and end frames of the scene, and the information indicating the key frame includes a frame number of the key frame.
Claim 29
Dependent Claim
26 words
The method according to claim 26 , wherein the frame identifier is a frame number assigned in turn from the beginning of the moving picture data.
Claim 30
Dependent Claim
25 words
The method according to claim 26 , wherein the frame identifier is a time from the beginning to that frame of the moving picture data.
Claim 31
Dependent Claim
46 words
The method according to claim 25 , further comprising: the acquisition step of acquiring feature amount information of the key frame, and wherein the scene information stored in the storage step includes the feature amount information of the key frame image acquired in the acquisition step.
Claim 32
Dependent Claim
56 words
The method according to claim 29 , wherein the feature amount information of the key frame is a label sequence obtained by segmenting an image of the key frame into a plurality of blocks, assigning labels in accordance with feature amounts acquired from the individual segmented blocks, and arranging the assigned labels in a predetermined order.
Claim 33
Dependent Claim
30 words
The method according to claim 30 , wherein the image is segmented into a plurality of vertical and horizontal blocks, the numbers of which are given by powers of 2.
Claim 34
Dependent Claim
39 words
The method according to claim 30 , further comprising the step of storing information that pertains to the number of segmented blocks, a feature amount extraction method, and a color expression of an image together with the label sequence.
Claim 35
Dependent Claim
89 words
The method according to claim 29 , further comprising: the computation step of computing a similarity value between images of a designated key frame and another key frame on the basis of image feature amounts of the key frames when one key frame is designated; the presentation step of presenting key frames on the basis of similarity values computed in the computation step; and the playback step of acquiring and playing back moving picture data including the key frame selected from the key frames presented in the presentation step.
Claim 36
Dependent Claim
39 words
The method according to claim 25 , further comprising the generation step of generating extracted moving picture data, which can be played back as a moving picture, on the basis of the scene information stored in the storage means.
Claim 37
Dependent Claim
35 words
The method according to claim 34 , wherein the generation step includes the step of generating the extracted moving picture data by time-serially merging scenes indicated by the scene information stored in the storage means.
Claim 38
Dependent Claim
48 words
The method according to claim 34 , wherein the generation step includes the step of generating the extracted moving picture data by time-serially merging scenes indicated by the scene information stored in the storage means, and opening and ending scenes of the moving picture for a predetermined time.
Claim 39
Dependent Claim
51 words
The method according to claim 34 , further comprising: the designation step of designating a playback time of the extracted moving picture data; and the adjustment step of adjusting durations of scenes to be merged in the generation step on the basis of the playback time designated in the designation step.
Claim 40
Dependent Claim
108 words
The method according to claim 37 , wherein the adjustment step comprises: the frame extraction step of determining importance levels of frames in a scene on the basis of an importance level set in the scene, extracting frames having importance levels that exceed a threshold value from the scene, and providing the extracted frames as a scene to be merged to the generation step; and the threshold value control step of controlling the threshold value used in the frame extraction step to make the playback time of the extracted moving picture data generated in the generation step fall within a predetermined error range of the designated playback time.
Claim 41
Dependent Claim
65 words
The method according to claim 38 , further comprising the extension adjustment step of extending the playback time as needed using the key frame of each scene as the center to fall within a predetermined error range of the designated playback time when a total time of all scenes designated by the scene information is shorter than the playback time designated in the designation step.
Claim 42
Dependent Claim
36 words
The method according to claim 34 , further comprising the removing step of removing a scene, having a playback time which is shorter than a predetermined time, from scenes that form the extracted moving picture data.
Claim 43
Dependent Claim
32 words
The method according to claim 40 , further comprising the extending step of extending another scene by a time corresponding to the playback time of the scene removed in the removing step.
Claim 44
Dependent Claim
47 words
The method according to claim 41 , wherein the extending step includes the step of extending a time of a scene set with a higher importance level of previous and next scenes which neighbor the scene removed in the removing step in the extracted moving picture data.
Claim 45
Dependent Claim
107 words
The method according to claim 34 , further comprising: the designation step of designating a playback time of the extracted moving picture data; and the extraction step of extracting scenes for a predetermined time duration by selecting scenes indicated by the scene information stored in the storage means in descending order of importance levels, and wherein the extraction step includes the step of executing a process until a total time of the extracted scenes exceeds the playback time designated in the designation step, and the generation step includes the step of generating the extracted moving picture data by time-serially merging the scenes extracted in the extraction step.
Claim 46
Dependent Claim
65 words
The method according to claim 43 , further comprising the extension adjustment step of extending the playback time as needed using the key frame of each scene as the center to fall within a predetermined error range of the designated playback time when a total time of all scenes designated by the scene information is shorter than the playback time designated in the designation step.
Claim 47
Dependent Claim
121 words
The method according to claim 34 , wherein the scene information includes an image feature amount of an image of each key frame, and said method further comprises: the computation step of computing a similarity value between images of a designated key frame and another key frame on the basis of image feature amounts of the key frames when one key frame is designated; the presentation step of presenting key frames on the basis of similarity values computed in the computation step; and the playback step of playing back the extracted moving picture data generated in the generation step in association with the moving picture data which includes the key frame selected from the key frames presented in the presentation step.
Claim 48
Dependent Claim
56 words
The method according to claim 45 , wherein the feature amount information of the key frame is a label sequence obtained by segmenting an image of the key frame into a plurality of blocks, assigning labels in accordance with feature amounts acquired from the individual segmented blocks, and arranging the assigned labels in a predetermined order.
Claim 49
Dependent Claim
119 words
The method according to claim 34 , wherein the scene information includes an image feature amount of an image of each key frame, and said method further comprises: the first step of randomly displaying key frames; the second step of playing back the extracted moving picture data generated in the generation step; the third step of making a similarity search by computing a similarity value between images of the designated key frame and another key frame on the basis of image feature amounts of the key frames; and the fourth step of playing back a moving picture on the basis of the designated key frame, and the first to fourth steps can operate to function in an arbitrary combination.
Claim 50
Dependent Claim
22 words
The method according to claim 25 , further comprising the display step of displaying a list of thumbnails of key frame images.
Claim 51
Dependent Claim
122 words
A storage medium for storing a control program that makes a computer implement a moving picture data storage process, said control program comprising: a code of the specifying step of specifying at least one desired scene in moving picture data, and a frame serving as a key frame in the desired scene; a code of the setting step of setting an importance level of the desired scene specified in the specifying step, with respect to the entire moving picture; and a code of the storage step of storing information indicating the scene and key frame specified in the specifying step and the importance level set in the setting step as scene information in storage means in association with the moving picture data.
Claim 52
Dependent Claim
122 words
A storage medium for storing a control program that makes a computer implement a moving picture data storage process, said control program comprising: a code of the specifying step of specifying at least one desired scene in moving picture data, and a frame serving as a key frame in the desired scene; a code of the setting step of setting an importance level of the desired scene specified in the specifying step, with respect to the entire moving picture; and a code of the storage step of storing information indicating the scene and key frame specified in the specifying step and the importance level set in the setting step as scene information in storage means in association with the moving picture data.
Claim 53
Dependent Claim
53 words
The medium according to claim 49 , wherein said control program further comprises: a code of the acquisition step of acquiring feature amount information of the key frame, and wherein the scene information stored in the storage step includes the feature amount information of the key frame image acquired in the acquisition step.
Claim 54
Dependent Claim
102 words
The medium according to claim 50 , wherein said control program further comprises: a code of the computation step of computing a similarity value between images of a designated key frame and another key frame on the basis of image feature amounts of the key frames when one key frame is designated; a code of the presentation step of presenting key frames on the basis of similarity values computed in the computation step; and a code of the playback step of acquiring and playing back moving picture data including the key frame selected from the key frames presented in the presentation step.
Claim 55
Dependent Claim
46 words
The medium according to claim 49 , wherein said control program further comprises a code of the generation step of generating extracted moving picture data, which can be played back as a moving picture, on the basis of the scene information stored in the storage means.
Claim 56
Dependent Claim
131 words
The medium according to claim 52 , wherein the scene information includes an image feature amount of an image of each key frame, and said control program further comprises: a code of the computation step of computing a similarity value between images of a designated key frame and another key frame on the basis of image feature amounts of the key frames when one key frame is designated; a code of the presentation step of presenting key frames on the basis of similarity values computed in the computation step; and a code of the playback step of playing back the extracted moving picture data generated in the generation step in association with the moving picture data which includes the key frame selected from the key frames presented in the presentation step.
Claim 57
Dependent Claim
75 words
A storage medium for storing moving picture data, said medium storing, in correspondence with the moving picture data, scene information, which includes: scene specifying information for specifying at least one desired scene from the stored moving picture data; key frame information representing a key frame of the scene specified by the scene specifying information; and an importance level set for each scene specified by the scene specifying information, with respect to the entire moving picture.
Claim 58
Dependent Claim
75 words
A storage medium for storing moving picture data, said medium storing, in correspondence with the moving picture data, scene information, which includes: scene specifying information for specifying at least one desired scene from the stored moving picture data; key frame information representing a key frame of the scene specified by the scene specifying information; and an importance level set for each scene specified by the scene specifying information, with respect to the entire moving picture.
Claim 59
Dependent Claim
36 words
The medium according to claim 54 , wherein the scene specifying information includes frame numbers of start and end frames of the scene, and the key frame information includes a frame number of the key frame.
Claim 60
Dependent Claim
25 words
The medium according to claim 54 , wherein the key frame information includes a feature amount of a key frame image in the scene information.
References Cited (Backward Citations)
US5537528A
Cited
US5606690A
Cited
JPH08241335A
Cited
US5553221A
Cited
EP0866409A1
Cited
JPH10260983A
Cited
US6400853B1
Cited
US6487360B1
Cited
US6526215B2
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-03-09 | 2007-09-08 | 2008-03-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-03-09 | 2011-09-08 | 2012-03-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-03-09 | 2015-09-08 | 2016-03-09 | $7700.00 | Window Closed / Lapsed |