This paper presents an easy-to-use and highly precise image search system. The key technology of the system is an automatic weighting mechanism for selecting features based on combining color, shape and structure features. Generally, the users should consider and determine weights for features to represent their preferences for selecting the features of an image search. These conventional systems make difficult for the novice users because it needs technical consideration for retrieval system. This paper proposes a new mechanism of automatic weighting for selecting the features by analyzing the distribution of color information to determine representative features. The color moments of the image are extracted and manipulated to calculate the color distance, the texture density and the shape property to determine respectively color, structure and shape weights. The color distances are calculated from the first order color moment by applying the shape independent clustering in order to construct and calculate distances of color hierarchy. The texture density is calculated from the second order of color moment to be more sensitive to scene the structures of images. The shape property is obtained from the third order of color moments. This proposed image search system is evaluated with a image retrieval benchmark dataset from Wang image collections. The experimental results clarify effectiveness of the proposed image search system to ease the feature selection and to reach the highly-retrieval precision with designated weights from the automatic weighting mechanism proposed in this paper.
Keywords: Image search, CBIR, feature extraction, automatic weighting.
INTRODUCTION
The rapid growth of the internet technology accelerates inter-media exchanges, including image data. According to a recent study, there are 180 million images on the publicly indexable Web, a total amount of image data of about 3Tb [terabytes], and an astounding one million or more digital images are being produced every day [1]. An efficient image searching, browsing, and retrieval systems are widely developed in order to provide better ways and approaches for such kinds of activities. The image retrieval systems, known as image search engine, based on the contents are very attracting and challenging in research areas of image searching. The image retrieval systems based on the contents are very attracting and challenging in research areas of image searching. Many content-based image retrieval (CBIR) systems have been proposed and widely applied for both commercial purposes and research systems. The system analyzes the contents of an image by extracting primitive features such as color, shape, texture, etc. Most approaches have been introduced to explore the content of an image and identify the primary and dominant features inside the image.
QBIC [2] introduced an image retrieval system based on color information inside an image. VisualSeek [3] represented a system by diagramming spatial arrangements based on representation of color regions. NETRA [4] developed a CBIR system by extracting color and texture features. Virage [5] utilized color, texture, and shape features for the image retrieval engine. CoIRS [6] also introduced a cluster oriented image retrieval system based on color, shape, and texture features. Veltkamp and Tanase [7] and Liu et al [8] presented a survey to many image retrieval systems using diverse features.
Generally, the users should consider and determine weights for features to represent their preferences for selecting the features. These conventional systems make difficult for the novice users because it needs technical consideration for retrieval system. In this paper, we extend our previous work of image search system [13] by proposing a new mechanism to determine an automatic weighting for selecting the features. This proposed mechanism analyses the distribution of color information to determine representative features. Figure 1 shows an illustration of our novel image search system compared with the conventional image search systems.
Figure 1. Illustration of our image search system with an automatic weighting mechanism
DESIGN OF OUR IMAGE SEARCH SYSTEM
This section briefly describes our previous work of image search system presented in [13]. Our image search system realizes a retrieval engine for image retrieval based on combining color, shape and structure features. Figure 2 shows the system architecture of our image search with involving the proposed mechanism of automatic weighting for selecting features.
Figure 2. System architecture of our image search
Pre-processing
Our image search system performs pre-processing to enhance quality of an image before feature extraction. It involves noise removal, color system conversion and image segmentation. We apply adaptive noise removal filtering using the Wiener filter for noise removal. The Weiner filter can be considered as one of the most fundamental noise reduction approaches and widely used for solution for image restoration problems [15][16].
A new hybrid color system using HSL and CIELAB is used in our image search to improve the quality of image segmentation. The images are converted into HSL and CIELAB color systems. HSL is well-known as an improved color space of HSV because it represents brightness much better than saturation. Beside, since the hue component in the HSL color space integrates all chromatic information, it is more powerful and successful for segmentation of color images than the primary colors [17]. The CIELAB color system has the advantage of being approximately perceptually uniform, and it is better than the RGB color system based on the assumption of three statistically independent color attributes [18]. In order to improve the feature extraction, the image segmentation is needed to unify contiguous colors in the color vector space into representative colors [13]. We used our previous work Pillar-Kmeans algorithm [19] for image segmentation before extracting the image features.
Figure 3 shows the computational steps of the image segmentation with our Pillar-Kmeans algorithm.
Figure 3. Computational steps of the image segmentation with our Pillar-Kmeans algorithm
Color Feature Extraction
The noise removal and 4x4 image partitioning are applied before extracting color feature. Then, for each block we extract color information using our previous research work 3D Color Vector Quantization [11]. For our image search system, we use the 64x64x64 quantization size of the RGB color space so that it can be represented with 125 positions in the RGB color space. Figure 4 shows the color feature extraction of our image search system.
Figure 4. Color feature extraction of our image search system
Shape Feature Extraction
Our image search system converts a segmented RGB image into gray-scale. In order to improve precision of shape metadata, we make partitioning 4x4 of each image. Then, we apply edge detection using Canny and measure the image properties which are eccentricity, area, equivalent diameter, and convex area. We apply the summarization of each image property for each image partition [13]. Figure 5 shows steps of the shape feature extraction of our image search system.
Figure 5. Steps of shape feature extraction of our image search
Structure Feature Extraction
We convert a segmented RGB image into gray-scale. Then, after applying 4x4 of image partitioning, the structure feature is extracted. We utilize the Curvelet algorithm to extract structure of an image [13]. The Curvelet is two multiscale geometric transforms that have revealed themselves quite useful over the past few years in diverse fields [20]. In such image processing, it is used for multi-scale image representation where the image can be represented at different layers of image transformation. However, applying the Curvelet algorithm naively to the extended image will result in (at least) a fourfold increase in computational complexity and redundancy. Solving this matter, in our image search system, we use 2D forward mirror-extended Curvelet transform for identifying the structure of an image. The mirror-extended Curvelet can cut down redundant computations where it is possible. Figure shows steps of structure feature extraction of our image search.
Figure 6. Steps of structure feature extraction of our image search
Automatic Weighting Mechanism
In this paper, we propose a new mechanism of automatic weighting for selecting the features in our image search system. This mechanism analyses the distribution of color information to determine representative features. First, we transform the color space of images into hybrid color spaces with combining HSL and CIELAB color spaces. The image segmentation is then applied in our image search system using our Pillar-Kmeans algorithm [19].
The system extracts color moments of an image, and calculates the color distances for the color weight, the texture density for the structure weight and the shape property for the shape weight. The color moments have been successfully used in many retrieval systems and proved to be efficient and effective in representing color distributions of images [12]. The color moments gives three kinds of orders, which are the first order (mean µ), the second order (variance σ) and the third order (skewness s).
(1)
(2)
(3)
where fij is the value of the i-th color component of the image pixel j, and N is the number of pixels in the image.
To obtain the color weight, the color distances are calculated from the first order color moment by applying the shape independent clustering [21] in order to construct and calculate distances of color hierarchy. For measuring the structure weight, the texture density is calculated from the second order of color moment to be more sensitive to scene the structures of images. The segmented images from the Pillar-Kmeans algorithm are transformed into gray-scale images to reduce the variance of the second order. To calculate the shape weight, the shape property is obtained from the third order of color moments. In this case, the images are converted into binary images in order to sharpen the skewness. The edges detection is then applied before calculating the third order of the color moments for shape property. The edges detection is then applied before calculating the third order of the color moments for shape property. The design of the proposed automatic weighting mechanism for our image search is shown in Figure 6. For the normalization, we set more weighted consideration for the color weight because the color feature is essential and dominance to determine the structure and shape weights. In the case of our image search, the color feature is weighted twice rather than the structure and shape features.
Figure 6. Design of the proposed automatic weighting mechanism our image search system
EXPERIMENTS AND DISCUSSION
After features extraction, the metadata of shape, structure, and color are created. The metadata of image query are used to measure the similarity with metadata repository of the image database. The retrieval engine analyses the distribution of color information to determine dominant features and set automatic weighting for selecting the features. For similarity measurement between image query and image database, we use the Cosine distance for the color feature, and our own semantic distance [13] for the shape and structure features. The retrieved results are ranked based on highest similarity between image query and image database.
For the experimental study, our image search system uses the well known benchmark 1000 images SIMPLIcity dataset of Wang et al. [14] which consists of a-general purpose image database containing of 1000 JPEG images from COREL image collections. These images are manually divided into 10 categories which are African people, beaches, historian buildings, buses, dinosaurs, elephants, roses, horses, mountains, and foods. For the experimental study, the system uses one image for each category as an image query. The experiment determines 15 top correct retrieved images for the query.
For performance analysis, the experiment calculates the number or errors for category of each retrieved image in line with the category of image query [13].
(4)
where:
cri = category of retrieved images
cq = category of pre-classified image query
Figure 7 shows the retrieved images of “bus†image query. It performs 13 relevant images from 15 retrieved results. The system sets the automatic weighting for color weight= 0.3539, structure weight= 0.3627, and shape weight= 0.2834. The proposed automatic weighting mechanism determines the weight of the structure feature more higher rather than the weights of the color and shape features. Therefore, the bus object with different color can possibly be retrieved in the retrieval process.
Figure 7. Retrieved results of bus image query
Figure 8. Retrieved results of animal queries
Figure 9. Retrieved results of panoramic queries
Figure 10. Retrieved results of flower, people, and food image queries
Figure 8 shows different queries of animals. The results are all relevant retrieved images for dinosaur, 14 relevant images for horse image query, and 12 correct images for elephant image query. We also tried the other experiments of several image queries which are beach, mountain, building, flower, people, and food, as shown in Figure 9 and Figure 10.
Table 1 shows errors of the determined weights for color, shape and structure features which are generated by our proposed automatic weighting mechanism in the image search system and the manual weighting of the best performance presented in our previous work [13].
Table 1. Errors of the automatic weighting and the manual weighting of the best performance
Image Query ID
Automatic Weighting
Manual Weighting
wcolor,
wshape,
wstructure
Error
wcolor,
wshape,
wstructure
Error
57
(People)
0.2993,
0.319,
0.3817
3 / 15
0.5,
0.4,
0.1
2 / 15
102
(Beach)
0.5497,
0.1505,
0.2998
2 / 15
0.9,
0.1,
0
1 / 15
248
(Building)
0.5294,
0.2053,
0.2653
3 / 15
0.6,
0.4,
0
1 / 15
346
(Bus)
0.3539,
0.2834,
0.3627
2 / 15
0.2,
0.7,
0.1
0 / 15
420
(Dinosaur)
0.5192,
0.1021,
0.3787
0 / 15
0.4,
0.3,
0.3
0 / 15
530
(Elephant)
0.5204,
0.176,
0.3036
3 / 15
0.4,
0.6,
0
0 / 15
641
(Flower)
0.3592,
0.2093,
0.4315
0 / 15
0.4,
0.3,
0.3
0 / 15
788
(Horse)
0.3032,
0.2849,
0.4119
1 / 15
0.5,
0.4,
0.1
0 / 15
804
(Mountain)
0.4121,
0.2021,
0.3858
9 / 15
1,
0,
0
4 / 15
962
(Food)
0.3881,
0.2591,
0.3528
0 / 15
0.4,
0.3,
0.3
0 / 15
Figure 11 performs the error comparison between the proposed automatic weighting mechanism and the manual weighting with best weights selection as shown in Table 1. The proposed image search system with automatic weighting only increased 1.5 of the average errors for each experiment comparing to the manual weighting.
Figure 11. Error comparison between the automatic weighting and the best manual weighting
CONCLUSION AND DISCUSSION
This paper presented an easy-to-use interface of the image search system with providing a new mechanism to automatically determine the weights for selecting features based on the combination of color, shape and structure features. The proposed mechanism analyses the distribution of color information to determine representative features. The color moments of the image are extracted and manipulated to calculate the color distance, the texture density and the shape property to determine respectively color, structure and shape weight features. The system was examined by the well known benchmark 1000 images SIMPLIcity dataset which consists of a-general purpose image database containing 10 categories which are African people, beaches, historian buildings, buses, dinosaurs, elephants, roses, horses, mountains, and foods. The experimental results perform that the errors of the proposed automatic mechanism increased 1.5 times comparing to the errors of the best performance of manual weighting. For future works, we will apply the proposed automatic weighting mechanism for our semantic image search system of cross-cultural images.
