Image processing method and apparatus

Claim 2

A method according to claim 1, wherein the actually taken image-based image generation step uses light space data as the actually taken image.

Claim 3

A method according to claim 1, wherein the model space rebuilding step includes the step of rebuilding the image using, as the first drawing specification, a perspective transformation derived from a position and direction of the viewpoint and an angle of view.

Claim 4

A method according to claim 1, wherein the model space rebuilding step includes the step of rebuilding the image using, as the first drawing specification, an orthogonal projection transformation derived from a position and direction of the viewpoint.

Claim 5

A method according to claim 1, wherein the image fusion step includes the step of fusing two drawn images by displaying an image drawn based on the shape model data on a screen first, and thereafter, overwriting an image drawn based on the actually taken image on the screen.

Claim 6

A method according to claim 1, wherein the shape model data in the model space rebuilding step is not data that expresses only a geometric shape but is data also including a surface characteristic and pattern of the geometric shape.

Claim 7

A method according to claim 1, wherein the actually taken image-based image generation step comprises the image reconstruction step of reconstructing an image from a plurality of images taken from a plurality of positions.

Claim 8

A method according to claim 7, wherein the actually taken image-based image generation step comprises the distortion correction step of correcting distortion of an image upon reconstructing the image in the image reconstruction step.

Claim 9

An image processing apparatus for generating and displaying a virtual environment, comprising: model space rebuilding means for generating a three-dimensional space using shape model data, and rebuilding the space as an image from the three-dimensional space in accordance with a first drawing specification at a given viewpoint; actually taken image-based image generation means for generating an image from an arbitrary viewpoint on the basis of an actually taken image in accordance with a second drawing specification that uses the same view condition as a view condition used by the first drawing specification; and image fusion means for fusing the image rebuilt by said model space rebuilding means and the image generated by said actually taken image-based image generation means.

Claim 10

An apparatus according to claim 9, wherein said actually taken image-based image generation means uses light space data as the actually taken image.

Claim 11

An apparatus according to claim 9, wherein said model space rebuilding means includes rebuilds the image using, as the first drawing specification, a perspective transformation derived from a position and direction of the viewpoint and an angle of view.

Claim 12

An apparatus according to claim 9, wherein said model space rebuilding means rebuilds the image using, as the first drawing specification, an orthogonal projection transformation derived from a position and direction of the viewpoint.

Claim 13

An apparatus according to claim 9, wherein said image fusion means fuses two drawn images by displaying an image drawn based on the shape model data on a screen first, and thereafter, overwriting an image drawn based on the actually taken image on the screen.

Claim 14

An apparatus according to claim 9, wherein the shape model data used by said model space rebuilding means is not data that expresses only a geometric shape but is data also including a surface characteristic and pattern of the geometric shape.

Claim 15

An apparatus according to claim 9, wherein said actually taken image-based image generation means comprises image reconstruction means for reconstructing an image from a plurality of images taken from a plurality of positions.

Claim 16

An apparatus according to claim 15, wherein said actually taken image-based image generation means comprises distortion correction means for correcting distortion of an image upon reconstructing the image by said image reconstruction means.

Claim 17

An image processing method of generating a virtual environment, comprising: the first description step of describing a shape of a three-dimensional object and a shape of a three-dimensional space on the basis of information including a three-dimensional position, a three-dimensional direction, and a surface color; the second description step of describing a three-dimensional object and a three-dimensional space on the basis of an actually taken image taken from an arbitrary position; and the generation step of generating the virtual environment by fusing the shapes described in the first description step and the three-dimensional object and the three-dimensional space described in the second description step.

Claim 18

A method according to claim 17, wherein the generation step includes the step of generating the virtual environment by fusing the shapes described in the first description step and the three-dimensional object and the three-dimensional space described in the second description step by a tree structure which includes the shapes and the objects as constituent elements.

Claim 19

A method according to claim 18, further comprising: the access step of searching and accessing all the constituent elements of the tree structure by a three search method; the drawing step of performing drawing suitable for each of the constituent elements accessed during the access step; and the changing step of changing a view condition of the virtual environment generated in the generation step.

Claim 21

A method according to claim 19, wherein when the constituent element to be accessed is a shape model, the drawing step comprises: the step of calculating a positional relationship between a viewpoint position and direction and the shape model; and the step of displaying an image on a display screen using a perspective transformation or an orthogonal projection transformation on the basis of the calculated relationship.

Claim 22

A method according to claim 19, wherein when the constituent element to be accessed is light space data, the drawing step comprises: the step of calculating a positional relationship between a viewpoint position and direction and the light space data; and the step of generating an image to be presented from the light space data on the basis of the calculated relationship.

Claim 23

A method according to claim 22, wherein when the constituent element to be accessed is light space data, the drawing step further comprises: the step of calculating a distance between an arrangement position of the light space data and the viewpoint position; and the step of discriminating depths of the constituent elements on the basis of the calculation result used as a depth value of the light space data.

Claim 24

A method according to claim 22, wherein the drawing step comprises: the step of preparing a depth value storage map having a size for one screen, when the constituent element to be accessed is a shape model, the drawing step further comprises: the step of generating a drawing image to be drawn on a display screen; the step of calculating distances between a viewpoint and pixels of the shape model corresponding to pixel positions on the display screen; and the step of displaying the drawing image by storing the distances in the depth value storage map in units of pixels when no depth values are stored in the depth value storage map, and displaying the drawing image on the display screen by storing the distances in the depth value storage map in units of pixels when the depth values are stored in the depth value storage map and when the calculated distances are compared with the corresponding depth values stored in the depth value storage map in units of pixels and the depth value is smaller than the corresponding distance, and when the constituent element to be accessed is light space data, the drawing step further comprises: the step of generating an image to be presented on the basis of the light space data; the depth value approximation step of calculating a distance between a viewpoint position and an arrangement position of the light space data and determining the calculation result as an approximate depth value of the light space data; and the step of displaying the generated image on the display screen by storing the approximate depth value in the depth value storage map in units of pixels when no depth value is stored in the depth value storage map, and displaying a pixel of the generated pixel on the display screen by storing the approximate depth value in the depth value storage map in units of pixels when the depth value is stored in the depth value storage map and when the approximate depth value is compared with the depth value in the depth value storage value in units of pixels and the approximate depth value is smaller than the stored depth value.

Claim 25

An image processing apparatus for generating a virtual environment, comprising: first description means for describing a shape of a three-dimensional object and a shape of a three-dimensional space on the basis of information including a three-dimensional position, a three-dimensional direction, and a surface color; second description means for describing a three-dimensional object and a three-dimensional space on the basis of an actually taken image taken from an arbitrary position; and generation means for generating the virtual environment by fusing the shapes described by said first description means and the three-dimensional object and the three-dimensional space described by said second description means.

Claim 26

An apparatus according to claim 25, wherein said generation means generates the virtual environment by fusing the shapes described by said first description means and the three-dimensional object and the three-dimensional space described by said second description means by a tree structure which includes the shapes and the objects as constituent elements.

Claim 27

An apparatus according to claim 26, further comprising: access means for searching and accessing all the constituent elements of the tree structure by a three search method; drawing means for performing drawing suitable for each of the constituent elements accessed during accessing by said access means; and changing means for changing a view condition of the virtual environment generated by said generation means.

Claim 29

An apparatus according to claim 27, wherein when the constituent element to be accessed is a shape model, said drawing means comprises: means for calculating a positional relationship between a viewpoint position and direction and the shape model; and means for displaying an image on a display screen using a perspective transformation or an orthogonal projection transformation on the basis of the calculated relationship.

Claim 30

An apparatus according to claim 27, wherein when the constituent element to be accessed is light space data, said drawing means comprises: means for calculating a positional relationship between a viewpoint position and direction and the light space data; and means for generating an image to be presented from the light space data on the basis of the calculated relationship.

Claim 31

An apparatus according to claim 29, wherein when the constituent element to be accessed is light space data, said drawing means further comprises: means for calculating a distance between an arrangement position of the light space data and the viewpoint position; and means for discriminating depths of the constituent elements on the basis of the calculation result used as a depth value of the light space data.

Claim 32

An apparatus according to claim 29, wherein said drawing means comprises: means for preparing a depth value storage map having a size for one screen, when the constituent element to be accessed is a shape model, said drawing means further comprises: means for generating a drawing image to be drawn on a display screen; means for calculating distances between a viewpoint and pixels of the shape model corresponding to pixel positions on the display screen; and means for displaying the drawing image by storing the distances in the depth value storage map in units of pixels when no depth values are stored in the depth value storage map, and displaying the drawing image on the display screen by storing the distances in the depth value storage map in units of pixels when the depth values are stored in the depth value storage map and when the calculated distances are compared with the corresponding depth values stored in the depth value storage map in units of pixels and the depth value is smaller than the corresponding distance, and when the constituent element to be accessed is light space data, said drawing means further comprises: means for generating an image to be presented on the basis of the light space data; depth value approximation means for calculating a distance between a viewpoint position and an arrangement position of the light space data and determining the calculation result as an approximate depth value of the light space data; and means for displaying the generated image on the display screen by storing the approximate depth value in the depth value storage map in units of pixels when no depth value is stored in the depth value storage map, and displaying a pixel of the generated pixel on the display screen by storing the approximate depth value in the depth value storage map in units of pixels when the depth value is stored in the depth value storage map and when the approximate depth value is compared with the depth value in the depth value storage value in units of pixels and the approximate depth value is smaller than the stored depth value.