System and method for measuring image similarity based on semantic meaning

Claim 2

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; for each semantic category, forming a set of the perceptual features comprising required features and frequently occurring features; comparing an image to said semantic categories; and classifying said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image.

Claim 3

A method as in claim 1 , where a database stores image information, where the image information comprises at least one of already classified images, network locations of already classified images and documents containing already classified images, and where said database is searched for images matching an input query.

Claim 4

A method as in claim 2 , where said input query comprises an input image, further comprising extracting features from said input image and applying a metric to determine the semantic category for said input image.

Claim 5

A method as in claim 2 , where said input query comprises an input image, further comprising processing said input image and comparing said input image to each semantic category using a metric that computes a similarity measure between the features used to describe the semantic category and corresponding features extracted from said input image; and assigning said input image to the semantic category that corresponds to a highest value of the similarity measure.

Claim 6

A method as in claim 2 , where said image information is obtained at least in part through the Internet.

Claim 7

A method as in claim 1 , where said images are medical images.

Claim 8

A method as in claim 5 , where said images are medical images.

Claim 9

A method as in claim 2 , further comprising searching said database for at least one image or image-related document that satisfies at least one threshold established by said search query.

Claim 10

A method as in claim 2 , where the database is located at a remote location and is searched by issuing a query to a database server through a data communications network.

Claim 11

A method as in claim 9 , and further comprising retrieving an image or image-related document from the remote database.

Claim 12

A method as in claim 2 , where the database is located at a remote location and is searched by issuing a query to a database server, through a data communications network, in conjunction with a text-based search algorithm executed by said server to retrieve a multi-media object from the database.

Claim 13

A method as in claim 2 , where the database comprises image information obtained from a plurality of autonomous searchers that execute in client computers and that return URLs corresponding to located image data and image-related documents.

Claim 14

A method as in claim 1 , further comprising segmenting an image by employing texture segmentation, color segmentation and foreground/background segmentation.

Claim 15

A method as in claim 13 , where texture segmentation includes forming a texture map.

Claim 16

A method as in claim 14 , where color segmentation includes forming a region map.

Claim 17

A method as in claim 15 , where foreground/background segmentation includes using the texture map and the region map to determine if there is at least one dominant object in the image, and forming an object map.

Claim 18

A method as in claim 16 , where the image, texture map, region map and object map are further processed to compute, for each region in the region map and for each object in the object map, a set of local features comprising information descriptive of size, boundary, neighbors, boundary curvature, texture, mean color, color name, and shape properties for that region/object.

Claim 19

A method as in claim 13 , where local features are analyzed to compute regional features for indicating a presence of semantic cues.

Claim 20

A method as in claim 18 , where said semantic cues comprise at least one of skin, sky, grass, snow, nature, water, man-made objects, man-made blobs, flowers and texture.

Claim 21

A method as in claim 13 , where local features are combined to compute global features that describe a global appearance of the image.

Claim 22

A method as in claim 20 , where said global features comprise at least one of number of details, histogram of color names, description of color composition, number of regions, number of blobs, number of objects, number of straight lines, number of regular curves, energy, spatial activity, symmetry and measures of contrast.

Claim 23

A data processing system comprising a data processor, a graphical user interface and a memory that stores a database containing image information, where the image information comprises at least one of already classified images, network locations of already classified images and documents containing already classified images, said data processor operating in accordance with a stored program for classifying images by deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; said data processor, for each semantic category, forming a set of the perceptual features comprising required features and frequently occurring features; comparing an image to said semantic categories; and classifying said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image, said data processor storing image-related classification data in said database.

Claim 24

A data processing system comprising a data processor, a graphical user interface and a memory that stores a database containing image information, where the image information comprises at least one of already classified images, network locations of already classified images and documents containing already classified images, said data processor operating in accordance with a stored program for classifying images by deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; said data processor, for each semantic category, forming a set of the perceptual features comprising required features and frequently occurring features; comparing an image to said semantic categories; and classifying said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image, said data processor storing image-related classification data in said database.

Claim 25

A data processing system as in claim 22 , where said database is searched for images matching an input query.

Claim 26

A data processing system as in claim 23 , where said input query comprises an input image, and where data processor extracts features from said input image and applies a metric to determine the semantic category for said input image.

Claim 27

A data processing system as in claim 23 , where said input query comprises an input image, and where said data processor processes said input image and compares said input image to each semantic category using a metric that computes a similarity measure between the features used to describe the semantic category and corresponding features extracted from said input image; and assigns said input image to the semantic category that corresponds to a highest value of the similarity measure.

Claim 28

A data processing system as in claim 22 , where said image information is obtained at least in part through the Internet.

Claim 29

A data processing system as in claim 22 , where said image information is comprised of medical image information.

Claim 30

A data processing system as in claim 22 , where said image information comprises data obtained from a plurality of autonomous searchers that execute in client computers and that return URLs corresponding to located images.

Claim 31

A data processing system as in claim 23 , where said data processor searches said database for locating at least one image that satisfies thresholds established by said search query.

Claim 32

A data processing system as in claim 23 , where the database is located at a remote location and is searched by issuing a query to a database server through a data communications network.

Claim 33

A data processing system as in claim 30 , and further comprising retrieving an image through said data communications network.

Claim 34

A data processing system as in claim 23 , where said database is located at a remote location and is searched by issuing a query to a database server, through a data communications network, in conjunction with a text-based search algorithm executed by said server to retrieve a multi-media object.

Claim 35

A data processing system as in claim 23 , where said image information comprises medical image information and is searched by issuing a query to a database server through the Internet.

Claim 36

A data processing system as in claim 23 , where said data processor segments an input image by employing texture segmentation, color segmentation and foreground/background segmentation.

Claim 37

A data processing system as in claim 34 , where texture segmentation includes forming a texture map.

Claim 38

A data processing system as in claim 35 , where color segmentation includes forming a region map.

Claim 39

A data processing system as in claim 36 , where foreground/background segmentation includes using the texture map and the region map to determine if there is at least one dominant object in the image, and to form an object map.

Claim 40

A data processing system as in claim 37 , where the input image, texture map, region map and object map are further processed to compute for each region in the region map and for each object in the object map, a set of local features comprising information concerning size, boundary, neighbors, boundary curvature, texture, mean color, color name, and shape properties for that region/object.

Claim 41

A data processing system as in claim 34 , where local features are analyzed to compute regional features, indicating a presence of semantic cues.

Claim 42

A data processing system as in claim 39 , where said semantic cues comprise at least one of skin, sky, grass, snow, nature, water, man-made objects, man-made blobs, flowers and texture.

Claim 43

A data processing system as in claim 39 , where local features are combined to compute global features that describe a global appearance of the image.

Claim 44

A data processing system as in claim 41 , where said global features comprise at least one of number of details, histogram of color names, description of color composition, number of regions, number of blobs, number of objects, number of straight lines, number of regular curves, energy, spatial activity, symmetry and measures of contrast.

Claim 45

A computer program embodied on a computer readable media for directing a computer to execute a method for processing digitally represented images, comprising program instructions for classifying images by deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories, where for each semantic category said program instructions form a set of the perceptual features comprising required features and frequently occurring features, compares an image to said semantic categories and classifies said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image.

Claim 46

A computer program embodied on a computer readable media for directing a computer to execute a method for processing digitally represented images, comprising program instructions for classifying images by deriving a plurality of semantic categories for representing important semantic cues in images, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories, where for each semantic category said program instructions form a set of the perceptual features comprising required features and frequently occurring features, compares an image to said semantic categories and classifies said image as belonging to one of said semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in said image.

Claim 47

A computer program as in claim 43 , where classified images are stored in a database, and where classified images or network locations of classified images are added to said database after being assigned to one of said semantic categories.

Claim 48

A computer program as in claim 43 , were said images are obtained at least in part through the Internet.

Claim 49

A computer program as in claim 44 , further comprising searching said database for at least one image or image-containing document that satisfies thresholds established by a search query.

Claim 50

A method as in claim 1 , where said perceptual features and their combinations are derived through subjective experiments performed with human observers.

Claim 51

A system as in claim 22 , where said perceptual features and their combinations are derived through subjective experiments performed with human observers.

Claim 52

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: for each semantic category of a plurality of semantic categories, forming a set of perceptual features comprising required features and frequently occurring features, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; determining features of an image; comparing features of the image to the sets of perceptual features of the semantic categories; and classifying the image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image.

Claim 53

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: for each semantic category of a plurality of semantic categories, forming a set of perceptual features comprising required features and frequently occurring features, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; determining features of an image; comparing features of the image to the sets of perceptual features of the semantic categories; and classifying the image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image.

Claim 54

A method as in claim 49 , where the perceptual features and their combinations are derived through subjective experiments performed with human observers.

Claim 55

A method as in claim 49 , where comparing further comprises computing a similarity measure between the features used to describe the semantic category and corresponding features determined from the image; and where classifying further comprises assigning the image to that semantic category that corresponds to a highest value of the similarity measure.

Claim 58

An apparatus as in claim 52 , where the perceptual features and their combinations are derived through subjective experiments performed with human observers.

Claim 59

An apparatus as in claim 52 , where the means for comparing further comprises means for computing a similarity measure between the features used to describe the semantic category and corresponding features determined from the image; and where the means for classifying further comprises means for assigning the image to that semantic category that corresponds to a highest value of the similarity measure.

Claim 60

A data processing system comprising a data processor and at least one memory comprising a stored program, the data processor operating in accordance with the stored program to form, for each semantic category of a plurality of semantic categories, a set of perceptual features comprising required features and frequently occurring features, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories, said data processor operating in accordance with the stored program to determine features of an image, to compare features of the image to the sets of perceptual features of the semantic categories, and to classify the image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image.

Claim 61

A data processing system comprising a data processor and at least one memory comprising a stored program, the data processor operating in accordance with the stored program to form, for each semantic category of a plurality of semantic categories, a set of perceptual features comprising required features and frequently occurring features, where each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories, said data processor operating in accordance with the stored program to determine features of an image, to compare features of the image to the sets of perceptual features of the semantic categories, and to classify the image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image.

Claim 62

An apparatus as in claim 55 , where the perceptual features and their combinations are derived through subjective experiments performed with human observers.

Claim 63

An apparatus as in claim 55 , where the data processor further operates when comparing to computer a similarity measure between the features used to describe the semantic category and corresponding features determined from the image; and where the data processor further operates when classifying to assign the image to that semantic category that corresponds to a highest value of the similarity measure.

Claim 64

A computer program as in claim 43 , where said perceptual features and their combinations are derived through subjective experiments performed with human observers.

Claim 65

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: performing subjective experiments with human observers to determine perceptual features of a plurality of images and to determine sets of semantic categories for the plurality of images, wherein each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; for each semantic category of a plurality of semantic categories, forming a set of perceptual features comprising required features and frequently occurring features; determining features of a query image; classifying the query image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image, wherein classifying further comprises computing a similarity measure between the perceptual features used to describe a semantic category and corresponding features determined from the query image; and assigning the query image to that semantic category that corresponds to a highest value of the similarity measure.

Claim 66

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: performing subjective experiments with human observers to determine perceptual features of a plurality of images and to determine sets of semantic categories for the plurality of images, wherein each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; for each semantic category of a plurality of semantic categories, forming a set of perceptual features comprising required features and frequently occurring features; determining features of a query image; classifying the query image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image, wherein classifying further comprises computing a similarity measure between the perceptual features used to describe a semantic category and corresponding features determined from the query image; and assigning the query image to that semantic category that corresponds to a highest value of the similarity measure.

Claim 67

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: performing subjective experiments with human observers to determine perceptual features of a plurality of images and to determine sets of semantic categories for the plurality of images, wherein each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; for each semantic category of a plurality of semantic categories, forming a set of perceptual features comprising required features and frequently occurring features; determining features of a query image; classifying the query image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image, wherein classifying further comprises computing a similarity measure between the perceptual features used to describe a semantic category and corresponding features determined from the query image, where the similarity measure comprises: sim ⁡ ( x , c i ) = 1 N i ⁢ ∏ j = 1 M i ⁢ ⁢ τ ( RF j ⁡ ( x ⁢  c i ) , RF j ⁡ ( c i ) ) · ∑ j = 1 N i ⁢ ⁢ τ ( FO j ⁡ ( x ⁢  c i ) , FO j ⁡ ( c i ) ) , where the query image is x, a semantic category is c i , {RF j (c i )|j=1, . . . , M i } is a set of M i required features, {FO j (c i )|j=1, . . . , N i } is a set of N i frequently occurring features, τ ⁡ ( a , B ) = { 1 , ( ∃ i ) 0 , ( ∀ i ) ⁢ a = b i a ≠ b i , and ⁢ ⁢ B = { b i } i = 1 , ⁢ … ⁢ , l ; and assigning the query image to that semantic category that corresponds to a highest value of the similarity measure.

Claim 68

A computer implemented method for characterizing images and determining image similarity based on semantic meaning of images, comprising: performing subjective experiments with human observers to determine perceptual features of a plurality of images and to determine sets of semantic categories for the plurality of images, wherein each semantic category is modeled through a combination of perceptual features that define the semantics of that category and that discriminate that category from other categories; for each semantic category of a plurality of semantic categories, forming a set of perceptual features comprising required features and frequently occurring features; determining features of a query image; classifying the query image as belonging to one of the semantic categories if all of the required features and at least one of the frequently occurring features for that semantic category are present in the features of the image, wherein classifying further comprises computing a similarity measure between the perceptual features used to describe a semantic category and corresponding features determined from the query image, where the similarity measure comprises: sim ⁡ ( x , c i ) = 1 N i ⁢ ∏ j = 1 M i ⁢ ⁢ τ ( RF j ⁡ ( x ⁢  c i ) , RF j ⁡ ( c i ) ) · ∑ j = 1 N i ⁢ ⁢ τ ( FO j ⁡ ( x ⁢  c i ) , FO j ⁡ ( c i ) ) , where the query image is x, a semantic category is c i , {RF j (c i )|j=1, . . . , M i } is a set of M i required features, {FO j (c i )|j=1, . . . , N i } is a set of N i frequently occurring features, τ ⁡ ( a , B ) = { 1 , ( ∃ i ) 0 , ( ∀ i ) ⁢ a = b i a ≠ b i , and ⁢ ⁢ B = { b i } i = 1 , ⁢ … ⁢ , l ; and assigning the query image to that semantic category that corresponds to a highest value of the similarity measure.