Improved Image Fusion of Colored and Grayscale Medical Images Based on Intuitionistic Fuzzy Sets

Image fusion is the process of combining the properties of two images into one single image that will show the features of both the images. There are various methods available in the literature to fuse the images. In this paper, an intuitionistic fuzzy logic-based image fusion approach has been implemented for medical images that firstly suppresses the noise and enhances the input images, and merges them efficiently in Hue-Saturation-Intensity domain. Here, enhancement is included because these input images are not always well contrasted and may contain some noise due to the inherent properties of the modalities used for capturing the images. The intuitionistic fuzzy sets are incorporated to handle uncertainties that are often due to vagueness and ambiguity. The results certify that this method significantly improves the output fused image than the image obtained by existing technique both visually and metrically.


Introduction
Present day application areas like medical, satellite, and military require various kinds of images and pictures as a source of information for interpretation and analysis. Medical imaging is an emerging field in which imaging data are important. There are various modalities used for imaging and each image provides different information, for example, the CT image gives information of dense structures like bones whereas the MR image gives information about soft tissues. So, there is an enormous need to fuse images like MRI, CT, PET, and SPECT to diagnose effectively. A brief introduction to these modalities and fusion methods in medical field is provided by A. P. James in [1]. Fusion is a process in which data or information from multiple sources are improved and combined to produce a single product that contains relevant features of all input sources. A general definition of image fusion is given as 'the combination of two or more different images to form a new image by using particular algorithm' [2]. Images have different geometric representations as they are produced from different sensors, so we need to bring the images into a common orientation before the fusion process. This alignment process is also known as image registration.

Related Work
Image fusion can be done at three different processing levels: signal, object, and decision. Signal-level (also known as pixel-level) techniques directly modify the pixel values while in object-level techniques features are extracted and fused, finally in decision or symbollevel techniques decisions are made to select which features to be fused [3]. Pixel-level methods are generally more powerful than others because direct modification of the pixel takes place in these techniques. Many researchers worked on the pixel-based fusion techniques. Fusion by taking the average of the pixels of source images is the simplest one, but it does not provide effective output. In the past few years, various methods had been proposed to fuse the images obtained from different modalities. For this, methods based on wavelets (DWT), contourlet transforms (CT), discrete cosine transform (DCT), etc. are available in the literature. In [4], authors surveyed some pixel-level fusion techniques based on principal component analysis (PCA), IHS, and wavelets. With these methods, the problem of color change in fused image is associated. In [5], authors analysed the techniques based on PCA, DWT, and DCT, where DWT-based methods outperform others in many quality measures, but some uncertainties also exist. Furthermore, in [6], authors combined the CT and Wavelet-based method for fusion of PET and MRI images. Pradnya and Sachin provide a comparative study of different wavelet-based methods in [7]. They compared the results of WT with hybrid architecture, Stationary wavelet transform (SWT), and WT with adaptive decomposition, and found that though the results of the Hybrid approach are good, wavelet with adaptive decomposition has better than it and other existing techniques. So, in this way various methods exist with different strategies, but, there are uncertainties that are not always due to randomness, but often due to vagueness and ambiguity. To tackle this problem, methods based on fuzzy logic were also proposed that enhances the quality of fused image significantly.
Initially, fuzzy set theory was proposed by L. A. Zadeh [8] nearly five decades ago, since then these are widely used in many real-time complex problems. The complex systems can be simplified by engaging a tolerance margin for a reasonable amount of imprecision, vagueness, and uncertainty during the modelling phase. As an outcome, not entirely perfect system comes into existence; nevertheless, in most of the cases, it is capable of handling the problem in an appropriate way. However, these sets have their limit due to their dependance on only one value i.e. degree of membership so that cannot show the evidence of hesitation, support, and opposition at the same time. So, later in 1985 Atanasov [9] proposed a new fuzzy set theory that is intuitionistic so-called Intuitionistic Fuzzy Sets (IFS). These sets have more strength and flexibility than the Zadeh's fuzzy sets because IFS' computation depends upon three values i.e. the degree of membership, non-membership, and hesitation. In these sets, the problem of ambiguity is more efficiently managed as these take into account both the membership and non-membership degrees. So, IFS can deal more efficiently with fuzzy information and uncertainties associated with it. Every image has some fuzziness in it, it may be due to poor illumination or inability of modality being used. And it is obvious in medical images.
Balasubramaniam and Ananthi [10] proposed a method based on IFS. In their work, they compared the results of many popular fusion techniques based on PCA, DWT, SWT, DTCWT, MSVD, and NSCT with the IFS-based method, and their results evidence that IFS-based technique provides a better quality fused image with better quality metric results. But, the method may not be well suited for all types of medical images. As the medical images are noisy and low contrasted in appearance due to inexactness of modalities and environment in which images are taken, due to which the fused output may come inappropriate or with some additional enhanced noisy artifacts, so there is a need of some pre-enhancement of inputs to suppress the noise and increase of contrast. Also, the method cannot be used to fuse the colored image (e.g. SPECT or PET) and grayscale image (e.g. MRI or CT) because, as the method intuitionisticly enhances the inputs using entropy value while fusing, the problem of color change in fused image lies there. And, for fuzzification of input, some better membership function like S-shape or sigma may be used for the better quality fused picture.
In this paper, a modified method based on IFS is proposed with better results than previous ones. Firstly, the input images are pre-processed by applying some noise removal filter mechanism for noisy images and contrast enhancement to improve the appearance. Secondly, the processed images are converted to HSI model and then fuzzified using fuzzy membership function. After that entropy model is used to calculate intuitionistic fuzzy images (IFIs), and then fuzzy rules based on blackness and whiteness of images are applied using MAX, MIN, and AVG functions to fuse the images.

Material and Methods
To use the IFS for image processing tasks, there are three degrees that are to be used i.e. membership, non-membership, and hesitation degree.
Let us have a finite set represents the degree of membership of x. So, the degree of non-membership of x will be equal to 1 − μ F (x). It can be represented by ν F (x). After that, Atanassov [9] and Atanassov and Stoeva [11] give the idea of fuzzy sets by using both the degrees. An IFS can be mathematically defined as Due to the lack of exactness, a new parameter π F (x) also originates which was introduced by Szmidt and Kacpryzk [12]. It is termed as the degree of hesitation. So, now the IFS can be generalized as This method is an extension to the work done by the Balasubramaniam and Ananthi [10]. Due to some drawbacks , various modifications to the method have been proposed in this paper. Initially, the input images are pre-processed for suppressing noise and enhancement of contrast. For contrast enhancement, a fuzzy logic-based histogram equalization technique is implemented which fuzzily enhances very fine details in the images. Fuzzy contrast enhancement is implemented on grayscale images only because while implementing it on colored images, the problem of color change appears. For noise removal, the non-local mean filter (NLMF) [20,21] is applied on noisy images. The particular filter is used, because in general, medical images (MRI) have Speckle, Poisson, Gaussian, Rician noises that can be handled well by NLMF.
After the pre-processing phase, the input images are now converted from RGB color model to the IHS color model so that while fusing the inputs and intuitionisticly enhancing them, the color problems do not encounter. Now the inputs are fuzzified using the rightangled triangular membership function [11] so that the degree of belongingness of the pixel value is computed that is calculated as Here, μ represents the fuzzy membership of (i, j)th pixel of x in image, a and b are lower and upper bound values that are dependent on input image and chosen experimentally. The motive of using FSs in the image processing tasks is to reduce uncertainties and ambiguities in the images. Input images are ambiguous in appearance due to the inabilities of used modalities or acquisition environment, because of that their arises vagueness and uncertainties about the brightness of the pixels, so that sometimes it is difficult to identify that weather a particular pixel is an edge or gray. The proposed method tries to remove this unclearness by considering another parameter hesitation in membership calculation. Hence, the generated fuzzy image is converted to the intuitionistic fuzzy image (IFI) by calculating intuitionistic membership, non-membership, and hesitation degrees of the fuzzy image.
By using fuzzy membership μ calculated previously, the membership degree can be calculated as [15]: The non-membership degree can be calculated as follows: The hesitation degree can be calculated as follows: Here, λ is a parameter used that depends upon input image. Since a number of IFIs can be generated for a single fuzzy image by changing the value of λ, it is necessary to choose an optimum value of the λ for a particular image. For this purpose, the entropy is used as in the previous method [10]. Originally, this entropy skeleton used here is given by Sergiadis and Vlachos [16] which is described as where the value of λ is opted at which the entropy is maximum. After IFI generation, the two images are now decomposed into small-sized blocks so that their respective blackness and whiteness of corresponding blocks can be calculated and then fuzzy rules can be applied to fuse them according to these values. The whole technique is implemented by applying the following algorithm. Fusion Algorithm: (1) Take two source images I 1 and I 2 that are to be fused.
(2) Apply pre-enhancement phase on I 1 and I 2 i.e. if images are noisy, then use the NLM filter to suppress the noise, and if inputs are low-contrasted, then apply fuzzy-based histogram equalization.
(3) Convert the enhanced images I 1 and I 2 from RGB or Grayscale to HSI color model. (4) Fuzzify the HSI images (Intensity part) to find the belongingness of the pixels. (5) Generate the IFI images by using the entropy equation to compute membership, non-membership, and hesitation degrees of IFIs, and represent them as I f 1 and I f 2 . (6) Decompose the two IFIs into small p × q parts to fuse them blockwise so that rules can be applied according to the properties of corresponding small blocks. Represent the k th block of both images as I f 1k and I f 2k . (7) Calculate the total count of Blackness and Whiteness of the corresponding blocks of two images. (8) Rebuild the k th block of fused image I fk as In the above procedure, two algorithms are used in step two for enhancement, one is NLM filter for noise removal and the other is fuzzy histogram equalization to spread the pixel values of the image on the whole intensity range for the purpose of contrast enhancement. For noise removal, NLM implementation provided by D.Kroon, University of Twente in [17] is used. Fuzzy histogram equalization implemented here is an extension of the procedure proposed by Khan and Ren in [18]. The algorithm used here for HE has also included the intuitionistic sets implementation to reduce inexactness and over-enhancement. The procedure is as follows: Fuzzy Histogram Equalization: (1) Take a grayscale input image whose contrast has to be enhanced, and calculate its fuzzy histogram by implementing the following procedure. (a) Calculate its histogram h (k) and then covert the image into fuzzy domain by the following formulas. f 1 (I (i, j))) 2 , else (b) Calculate fuzzy histogram by applying the following formula: .
Here, f (k) is the fuzzy value from f for particular intensity level.
(2) Now break the histogram into two segments according to the median value ( M) of the occupied intensities so that the lower and upper bounds of both the histograms are 0 and M for first, and M+1 and 255 for the second. (3) Now calculate the cumulative density function of both the histogram segments. (4) Then equalize the sub histograms by using the following formulas.

Experimental Results and Statistical Analysis
The results are obtained for 512 × 512 sized without reference, very low-contrast medical images. Here, MRI, CT, PET, and Spectroscopy scans are included for fusion in which some are downloaded from 'The Whole Brain Atlas -Harvard Medical School' [19]. Here, MRI and CT are grayscale images whereas PET and Spectroscopy are colored images.
As the reference images are unidentified for multi-modality image fusion, it is too hard to materialize the performance of fusion approaches. To quantify the performance of the proposed strategy, following quality metrics have been considered which will be computed in the sequel.
• Spatial Frequency: It is a measure of how often sinusoidal components of the structure repeat per unit of distance. In other words, it refers to the level of detail present in a stimulus per degree of visual angle. It can be computed as follows: SF of the fused image is high when its activity level is huge. • Standard Deviation: Standard deviation comprises both original image and noise acquired during transmission. It is more powerful when there is no noise in the transmitted image and portraits the contrast of an image. STD can be calculated as follows: • Objective Image Fusion Performance Measure (OIFP): It is a measure for objectively assessing the pixel-level fusion performance [20]. This metric reflects the amount of edge and visual information obtained in a combined image from the input images to be fused, so that the performance of different image fusion algorithms can be compared. In this measure, a Sobel edge operator has been applied to images to calculate the edge strength and its orientation. In this work, to compute OIFP an open-source MATLAB program developed by [21] had been executed for source and output images.
The fusion is done for two types of image combinations i.e. grayscale-grayscale (MRI-CT), and grayscale-colored (MRI-SPECT and MRI-PET).

Grayscale-Grayscale Image Fusion
In Figure 1, the fusion results of the noise-free MRI and CT scan are shown. With the analysis of the resultant images, it can be clearly seen that the fusion performed by modified method produces a better class fused image that exhibits the extra amount of finer detail which is clearly visible in the figure. Also, the image shows better quality and pleasing effect visually.
In Figure 2, the fusion results of the noisy MRI and CT scan are shown. Here, it can be clearly seen that the resultant fused image (c) produced by the previous method is very noisy, and some unwilling artifacts have been arisen. Also, the visual appearance of the input images is lost. But the image (d) fused using modified technique is of superb quality with improved contrast. The noise present in input images is entirely removed and it provides excellent visual appearance. Table 1 shows the quality measure results for the output fused image obtained by both the existing and modified grayscale-grayscale fusion of low-contrast and noisy images. Results of modified procedure are shown by highlighted background and the best results among previous and modified method are shown in the bold font. It can be clearly identifiable that all the metrics have improved at a large extent. As the OIFP has been increased from ∼3% to ∼40%, it shows the edges similarity between source and output image, so it can be concluded that resultant images formed by modified strategy are more similar (with enhanced edges) to source images than by existing ones. But in the case of noisy images, though the STD and OIFP have been increased for fused image by modified method, but  the spatial frequency is decreased. The reason behind it is the presence of noise in the image fused obtained by the existing procedure, so, with image content, noise is also being computed as information in the calculation.

Grayscale-Colored Image Fusion
In Figures 3-5, the implementation of MRI fusion with PET and SPECT images is shown. In these, PET and SPECT are colored images that are fused with grayscale MRI image. Also in this criterion, the low-contrast image has been generated for the grayscale MRI image that is to be used as a source image for fusion. By examining results, it has been observed that the fused image using improved method exhibits an extremely high contrasted sound quality images while preserving their natural appearance. Also, the color change and other problems that lie with the previous scheme have been resolved. Figures 3-5 shows the results of MRI-SPECT and MRI-PET image fusion. It is clearly visible that the images (e) fused with the modified procedure are highly contrasted and provide more fine details as compared to the images (c) and (d) produced by the previous procedure. Also, the natural appearance of the picture has not been lost and exhibits good eminence. But in the case of previous work, the image shows very low-contrast and   color-change problems. Here, two fused images (c) and (d) have been produced using the previous technique by taking defuzzification function of the grayscale input image and the colored input image. In the first fused image (c), the low contrast of grayscale input image gets inherited in fused image and color has been changed slightly. And in the second fused image (d), while the color of the input image has been retained but it gets imposed on the whole fused image. In Table 2, calculated metric values of the output images produced by both the existing and modified grayscale-colored fusion method are shown. It can be clearly identified that all the metric values show extremely remarkable results for the image fused with the modified procedure. Spatial frequency has been increased to, a very large extent, for images produced by the amended method, so it signifies that this image has a great level of details as compared to others. On the other hand, OIFP is also amplified significantly, it means the improved method's output image resembles more with source images which certify that the modified procedure is highly efficient than the existing method.

Conclusion
In this research work, an intuitionistic fuzzy sets-based image fusion technique for medical images has been proposed, which, before fusion first enhances: the low-contrast input images by applying a fuzzy logic-based histogram equalization technique, and noisy images by applying a non-local means filter to remove noise. This method is an extension of the method suggested in [10]. The previous method has some limitations regarding lowcontrast and noisy images. Also, it is not applicable for the fusion of colored images as there lies the problem of color change in the fused image. But, the proposed method overcomes all these difficulties and provides a better class high contrasted fused image without any information loss.
After analyzing the statistical and visual results, it is to be concluded that, output fused image obtained from the modified technique has better visual quality and contrast as compared to the image obtained from the existing method. Also, there is significant improvement in performance measures. The problems related to colored and noisy images have also been resolved. Even the amended method produces a high contrast fused image for low-contrast input images.
Future Scope: As discussed earlier, the heart of this fusion strategy is the intuitionistic fuzzy membership calculation, and the membership is calculated using a parameter (λ) provided by the entropy equation. So in further work, efforts may be made to improve this intuition-based entropy skeleton. Also, some expert-based fuzzy rules can be designed to apply at fusion time to get the relative information in fused image for effective diagnosis.