Patent US5963664A successfully refreshed from Google Patents.
US5963664A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and system for image combination using a parallax-based technique
Assignee: Sarnoff Corp
•
Filing: 1995-06-22
•
Granted: 1999-10-05
•
Expires: 2015-06-22
Abstract
Abstract A system for generating three-dimensional mosaics from a plurality of input images representing an imaged scene. The plurality input images contain at least two images of a single scene, where at least two of the images have overlapping regions. The system combines the images using a parallax-based approach that generates a three-dimensional mosaic comprising an image mosaic representing a panoramic view of the scene and a shape mosaic representing the three dimensional geometry of the scene. Specifically, in one embodiment, the system registers the input images along a parametric surface within the imaged scene and derives translation vectors useful in aligning the images into a two-dimensional image mosaic. Once registered, the system generates a shape mosaic representing objects within the scene.
Inventors
- 👤 Rakesh Kumar
- 👤 Keith James Hanna
- 👤 James R. Bergen
- 👤 Padmanabhan Anandan
- 👤 Michal Irani
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-27 15:13
CPC Classifications
G; G06; G06T; G06T15/00; G06T15/10; G06T15/20; G06T7/00; G06T7/30; G06T7/33; G06T7/50; G06T7/55; G06T7/593; G06V; G06V10/00; G06V10/20; G06V10/24; H; H04; H04N; H04N13/00; H04N13/10; H04N13/106; H04N13/111; H04N19/00; H04N19/20; H04N19/23; G06T2207/00; G06T2207/10; G06T2207/10004; G06T2207/10012; G06V10/10; G06V10/16; H04N13/189; H04N13/20; H04N13/204; H04N13/239; H04N13/243; H04N2013/0074; H04N2013/0081; H04N2013/0088
Claims Analysis
Total Claims: 60
Claim 1
Independent Claim
75 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; and registering said images along a non-planar parametric surface to construct said three-dimensional mosaic containing an image mosaic of registered images and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 2
Independent Claim
75 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; and registering said images along a non-planar parametric surface to construct said three-dimensional mosaic containing an image mosaic of registered images and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 3
Dependent Claim
67 words
The method of claim 1 wherein said registering step further comprises the steps of: registering each image in said plurality of images along said non-planar parametric surface to produce registered images; determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the non-planar parametric surface; and generating a parallax field representing parallax of objects within the scene.
Claim 4
Dependent Claim
30 words
The method of claim 2 further comprising the step of converting said plurality of images into a plurality of multi-resolutional pyramids, where each image pyramid contains a plurality of levels.
Claim 5
Dependent Claim
35 words
The method of claim 3 wherein said registering and determining steps are iterated over each of said levels within said multi-resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.
Claim 6
Dependent Claim
32 words
The method of claim 4 wherein said predefined degree of accuracy is a sum of the squares difference measure integrated over selected regions within each of said levels of said multi-resolutional pyramids.
Claim 7
Dependent Claim
14 words
The method of claim 1 wherein said shape mosaic contains a parallax motion field.
Claim 8
Dependent Claim
16 words
The method of claim 1 wherein said image mosaic and said shape mosaic are multi-resolutional pyramids.
Claim 9
Dependent Claim
66 words
The method of claim 1 further comprising the steps of: converting said image mosaic and said shape mosaic into multi-resolutional pyramids; converting a new image into a multi-resolutional pyramid; and determining pose parameters for relating the new image with the image mosaic and the shape mosaic, where the pose parameters contain translation parameters, a planar motion field, and a parallax motion field for the new image.
Claim 10
Dependent Claim
29 words
The method of claim 8 further comprising the step of generating a updated image mosaic and an updated shape mosaic, each containing the new image and the pose parameters.
Claim 11
Dependent Claim
92 words
The method of claim 8 further comprising the steps of: providing an existing three-dimensional mosaic; determining pose parameters for a new image with respect to said existing three-dimensional mosaic; warping said existing three-dimensional mosaic to image coordinates of said new image to create a synthetic image, where said synthetic image represents a view of the three-dimensional mosaic from the coordinates of the new image; and merging said synthetic image into said new image to produce a new three-dimensional mosaic that is a combination of said new image and said existing three-dimensional mosaic.
Claim 12
Dependent Claim
52 words
The method of claim 10 further comprising the steps of: providing a next image that sequentially follows said new image; detecting changes between said new image, said existing three-dimensional mosaic, and said next image, where said changes represent motion within the scene without detecting parallax due to viewpoint change as said motion.
Claim 13
Dependent Claim
47 words
The method of claim 1 further comprising the steps of: detecting points within said three-dimensional mosaic that are occluded within the scene by objects in the scene; and image processing the detected occluded points such that said occluded points do not produce artifacts in said three-dimensional mosaic.
Claim 14
Dependent Claim
44 words
The method of claim 1 further comprising the step of: estimating a height of points within said three-dimensional mosaic relative to said parametric surface, where said height of said points form a height map that represents the height of object points within said scene.
Claim 15
Dependent Claim
110 words
The method of claim 1 further comprising the steps of: providing a plurality of three-dimensional mosaics representing a scene from different viewpoints, where a three-dimensional mosaic has been generated at each viewpoint; warping said plurality of three-dimensional mosaics to a reference coordinate system; merging said plurality of three-dimensional mosaics to form a composite three-dimensional mosaic; providing coordinates for a new viewpoint of said scene; determining parameters to relate said new viewpoint coordinates to said composite three-dimensional mosaic; and warping said composite three-dimensional mosaic to said viewpoint coordinates to create a synthetic image, where said synthetic image represents a new view of the composite three-dimensional mosaic taken from the new viewpoint.
Claim 16
Dependent Claim
100 words
The method of claim 1 further comprising the steps of: providing a plurality of three-dimensional mosaics representing a scene from different viewpoints, where a three-dimensional mosaic has been generated at each viewpoint; providing coordinates for a new viewpoint of said scene; determining parameters to relate said new viewpoint coordinates to a plurality of the three-dimensional mosaics; warping said plurality of three-dimensional mosaics to said viewpoint coordinates to create a synthetic image, where said synthetic image represents a new view of the three-dimensional mosaic taken from the new viewpoint; and merging said plurality of three-dimensional mosaics to form said synthetic image.
Claim 17
Dependent Claim
150 words
The method of claim 1 wherein said registering step further comprises the steps of: performing a plane-then-parallax process including the steps of registering each image in said plurality of images along a parametric surface to produce initially registered images; determining, in response to said initially registered images, initial translation parameters and a initial parametric motion field useful in initially aligning the images along the parametric surface; and generating an initial parallax field representing parallax of objects within the scene; and simultaneously registering, using said initial translation parameters, initial parametric motion field and initial parallax field, said images in said plurality of images along said parametric surface to produce final registered images, determining, in response to said final registered images, final translation parameters and a final parametric motion field useful in aligning the images along the parametric surface, and generating a final parallax field representing parallax of objects within the scene.
Claim 18
Dependent Claim
30 words
The method of claim 16 further comprising the step of converting said plurality of images into a plurality of multi-resolutional pyramids, where each multi-resolutional pyramid contains a plurality of levels.
Claim 19
Dependent Claim
37 words
The method of claim 17 wherein said registering, determining and simultaneously registering steps are iterated over each of said levels within said multi-resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.
Claim 20
Dependent Claim
32 words
The method of claim 18 wherein said predefined degree of accuracy is a sum of the squares difference measure integrated over selected regions within each of said levels of said multi-resolutional pyramids.
Claim 21
Dependent Claim
16 words
The method of claim 16 wherein said image mosaic and said shape mosaic are multi-resolutional pyramids.
Claim 22
Independent Claim
83 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; simultaneously registering said images in said plurality of images along a parametric surface to produce registered images, determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the parametric surface, and generating a parallax field representing parallax of objects not lying within said parametric surface.
Claim 23
Independent Claim
83 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; simultaneously registering said images in said plurality of images along a parametric surface to produce registered images, determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the parametric surface, and generating a parallax field representing parallax of objects not lying within said parametric surface.
Claim 24
Dependent Claim
30 words
The method of claim 21 further comprising the step of converting said plurality of images into a plurality of multi-resolutional pyramids, where each multi-resolutional pyramid contains a plurality of levels.
Claim 25
Dependent Claim
35 words
The method of claim 22 wherein said registering and determining steps are iterated over each of said levels within said multi-resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.
Claim 26
Dependent Claim
32 words
The method of claim 23 wherein said predefined degree of accuracy is a sum of the squares difference measure integrated over selected regions within each of said levels of said image pyramids.
Claim 27
Dependent Claim
66 words
The method of claim 21 further comprising the steps of: converting said image mosaic and said shape mosaic into multi-resolutional pyramids; converting a new image into a multi-resolutional pyramid; and determining pose parameters for relating the new image with the image mosaic and the shape mosaic, where the pose parameters contain translation parameters, a planar motion field, and a parallax motion field for the new image.
Claim 28
Dependent Claim
29 words
The method of claim 25 further comprising the step of generating a updated image mosaic and an updated shape mosaic, each containing the new image and the pose parameters.
Claim 29
Dependent Claim
92 words
The method of claim 25 further comprising the steps of: providing an existing three-dimensional mosaic; determining pose parameters for a new image with respect to said existing three-dimensional mosaic; warping said existing three-dimensional mosaic to image coordinates of said new image to create a synthetic image, where said synthetic image represents a view of the three-dimensional mosaic from the coordinates of the new image; and merging said synthetic image into said new image to produce a new three-dimensional mosaic that is a combination of said new image and said existing three-dimensional mosaic.
Claim 30
Dependent Claim
52 words
The method of claim 27 further comprising the steps of: providing a next image that sequentially follows said new image; detecting changes between said new image, said existing three-dimensional mosaic, and said next image, where said changes represent motion within the scene without detecting parallax due to viewpoint change as said motion.
Claim 31
Dependent Claim
47 words
The method of claim 21 further comprising the steps of: detecting points within said three-dimensional mosaic that are occluded within the scene by objects in the scene; and image processing the detected occluded points such that said occluded points do not produce artifacts in said three-dimensional mosaic.
Claim 32
Dependent Claim
44 words
The method of claim 21 further comprising the step of: estimating a height of points within said three-dimensional mosaic relative to said parametric surface, where said height of said points form a height map that represents the height of object points within said scene.
Claim 33
Dependent Claim
110 words
The method of claim 21 further comprising the steps of: providing a plurality of three-dimensional mosaics representing a scene from different viewpoints, where a three-dimensional mosaic has been generated at each viewpoint; warping said plurality of three-dimensional mosaics to a reference coordinate system; merging said plurality of three-dimensional mosaics to form a composite three-dimensional mosaic; providing coordinates for a new viewpoint of said scene; determining parameters to relate said new viewpoint coordinates to said composite three-dimensional mosaic; and warping said composite three-dimensional mosaic to said viewpoint coordinates to create a synthetic image, where said synthetic image represents a new view of the composite three-dimensional mosaic taken from the new viewpoint.
Claim 34
Dependent Claim
100 words
The method of claim 21 further comprising the steps of: providing a plurality of three-dimensional mosaics representing a scene from different viewpoints, where a three-dimensional mosaic has been generated at each viewpoint; providing coordinates for a new viewpoint of said scene; determining parameters to relate said new viewpoint coordinates to a plurality of the three-dimensional mosaics; warping said plurality of three-dimensional mosaics to said viewpoint coordinates to create a synthetic image, where said synthetic image represents a new view of the three-dimensional mosaic taken from the new viewpoint; and merging said plurality of three-dimensional mosaics to form said synthetic image.
Claim 35
Dependent Claim
150 words
The method of claim 21 wherein said registering step further comprises the steps of: performing a plane-then-parallax process including the steps of registering each image in said plurality of images along a parametric surface to produce initially registered images; determining, in response to said initially registered images, initial translation parameters and a initial parametric motion field useful in initially aligning the images along the parametric surface; and generating an initial parallax field representing parallax of objects within the scene; and simultaneously registering, using said initial translation parameters, initial parametric motion field and initial parallax field, said images in said plurality of images along said parametric surface to produce final registered images, determining, in response to said final registered images, final translation parameters and a final parametric motion field useful in aligning the images along the parametric surface, and generating a final parallax field representing parallax of objects within the scene.
Claim 36
Dependent Claim
30 words
The method of claim 33 further comprising the step of converting said plurality of images into a plurality of multi-resolutional pyramids, where each multi-resolutional pyramid contains a plurality of levels.
Claim 37
Dependent Claim
37 words
The method of claim 34 wherein said registering, determining and simultaneously registering steps are iterated over each of said levels within said multi-resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.
Claim 38
Dependent Claim
32 words
The method of claim 35 wherein said predefined degree of accuracy is a sum of the squares difference measure integrated over selected regions within each of said levels of said multi-resolutional pyramids.
Claim 39
Dependent Claim
16 words
The method of claim 34 wherein said image mosaic and said shape mosaic are multi-resolutional pyramids.
Claim 40
Independent Claim
129 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; registering each image in said plurality of images along a non-planar parametric surface to produce registered images; and determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the non-planar parametric surface; and generating a parallax field representing parallax of objects within the scene; constructing, in response to said translation parameters, parametric motion field, and said parallax field, said three-dimensional mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 41
Independent Claim
129 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; registering each image in said plurality of images along a non-planar parametric surface to produce registered images; and determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the non-planar parametric surface; and generating a parallax field representing parallax of objects within the scene; constructing, in response to said translation parameters, parametric motion field, and said parallax field, said three-dimensional mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 42
Dependent Claim
30 words
The method of claim 38 further comprising the step of converting said plurality of images into a plurality of multi-resolutional pyramids, where each multi-resolutional pyramid contains a plurality of levels.
Claim 43
Dependent Claim
35 words
The method of claim 39 wherein said registering and determining steps are iterated over each of said levels within said multi-resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.
Claim 44
Dependent Claim
32 words
The method of claim 40 wherein said predefined degree of accuracy is a sum of the squares difference measure integrated over selected regions within each of said levels of said multi-resolutional pyramids.
Claim 45
Dependent Claim
19 words
The method of claim 41 wherein said shape mosaic contains a parametric motion field and a parallax motion field.
Claim 46
Dependent Claim
16 words
The method of claim 38 wherein said image mosaic and said shape mosaic are multi-resolutional pyramids.
Claim 47
Independent Claim
128 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; simultaneously registering said images in said plurality of images along a parametric surface to produce registered images, determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the parametric surface, and generating a parallax field representing parallax of objects within the scene; and constructing, in response to said translation parameters, parametric motion field, and said parallax field, said three-dimensional mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 48
Independent Claim
128 words
A method of processing a plurality of images to generate a three-dimensional mosaic of a scene comprising the steps of: providing a plurality of images of the scene; simultaneously registering said images in said plurality of images along a parametric surface to produce registered images, determining, in response to said registered images, translation parameters and a parametric motion field useful in aligning the images along the parametric surface, and generating a parallax field representing parallax of objects within the scene; and constructing, in response to said translation parameters, parametric motion field, and said parallax field, said three-dimensional mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 49
Dependent Claim
30 words
The method of claim 44 further comprising the step of converting said plurality of images into a plurality of multi-resolutional pyramids, where each multi-resolutional pyramid contains a plurality of levels.
Claim 50
Dependent Claim
35 words
The method of claim 45 wherein said registering, and determining steps are iterated over each of said levels within said multi-resolutional pyramids until said plurality of images are registered to a predefined degree of accuracy.
Claim 51
Dependent Claim
32 words
The method of claim 46 wherein said predefined degree of accuracy is a sum of the squares difference measure integrated over selected regions within each of said levels of said multi-resolutionsal pyramids.
Claim 52
Dependent Claim
16 words
The method of claim 44 wherein said image mosaic and said shape mosaic are multi-resolutional pyramids.
Claim 53
Dependent Claim
82 words
An image processing system for generating a three-dimensional mosaic three-dimensional mosaic of a scene from a plurality of images of the scene, comprising: means for storing said plurality of images; a registration processor, connected to said storing means, for registering said images along a non-planar parametric surface to construct said three-dimensional mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 54
Dependent Claim
82 words
An image processing system for generating a three-dimensional mosaic three-dimensional mosaic of a scene from a plurality of images of the scene, comprising: means for storing said plurality of images; a registration processor, connected to said storing means, for registering said images along a non-planar parametric surface to construct said three-dimensional mosaic containing an image mosaic and a shape mosaic, where said image mosaic represents a panoramic view of the scene and said shape mosaic represents a three-dimensional geometry of the scene.
Claim 55
Dependent Claim
61 words
The system of claim 49 wherein said registration processor further comprises: a plane-then-parallax registration processor for aligning said images along said non-polar parametric surface that extends through the plurality of images to produce translation parameters and a parametric motion field used to align the images within the image mosaic Land then for determining a parallax field representing objects within the scene.
Claim 56
Dependent Claim
77 words
An image processing system for generating a three-dimensional mosaic of a scene from a plurality of images of the scene, comprising: means for storing said plurality of images; a plane-and-parallax registration processor for simultaneously aligning said images along a parametric surface that extends through the plurality of images to produce translation parameters and a parametric motion field used to align the images within the image mosaic and for determining a parallax field representing objects within the scene.
Claim 57
Dependent Claim
77 words
An image processing system for generating a three-dimensional mosaic of a scene from a plurality of images of the scene, comprising: means for storing said plurality of images; a plane-and-parallax registration processor for simultaneously aligning said images along a parametric surface that extends through the plurality of images to produce translation parameters and a parametric motion field used to align the images within the image mosaic and for determining a parallax field representing objects within the scene.
Claim 58
Dependent Claim
151 words
An image processing system for generating a three-dimensional mosaic of a scene from a plurality of images of the scene, comprising: means for storing said plurality of images; a plane-then-parallax registration processor for aligning said images along a parametric surface that extends through the plurality of images to produce initial translation parameters and an initial parametric motion field used to align the images within the image mosaic and then for determining an initial parallax field representing objects within the scene that do not lie in the parametric surface; and a plane-and-parallax registration processor, connected to an output of said plane-then-parallax registration processor, for simultaneously aligning said images along said parametric surface to produce final translation parameters and a final parametric motion field used to align the images within the image mosaic and for determining a final parallax field representing objects within the scene that do not lie in the parametric surface.
Claim 59
Dependent Claim
151 words
An image processing system for generating a three-dimensional mosaic of a scene from a plurality of images of the scene, comprising: means for storing said plurality of images; a plane-then-parallax registration processor for aligning said images along a parametric surface that extends through the plurality of images to produce initial translation parameters and an initial parametric motion field used to align the images within the image mosaic and then for determining an initial parallax field representing objects within the scene that do not lie in the parametric surface; and a plane-and-parallax registration processor, connected to an output of said plane-then-parallax registration processor, for simultaneously aligning said images along said parametric surface to produce final translation parameters and a final parametric motion field used to align the images within the image mosaic and for determining a final parallax field representing objects within the scene that do not lie in the parametric surface.
Claim 60
Dependent Claim
52 words
The system of claim 52 further comprising a three-dimensional mosaic generator, connected to said registration processor, for combining said images in said plurality of images using said final translation parameters and said final motion flow field to form said image mosaic and for generating said shape mosaic containing the final parallax field.
References Cited (Backward Citations)
US4797942A
Cited
US5202928A
Cited
US5187754A
Cited
US5568384A
Cited
US5550937A
Cited
US5682198A
Cited
US5530774A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-10-04 | 2003-04-05 | 2003-10-05 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-10-04 | 2007-04-05 | 2007-10-05 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-10-04 | 2011-04-05 | 2011-10-05 | $7700.00 | Window Closed / Lapsed |