Binocular microlens imaging system based on micro fabrication technology and its application in vein-enhanced display

Abstract Vein display device can significantly increase the success rate of intravenous injection. However, traditional vein display devices only contain near-infrared image information, which inevitably loses skin color information. In this article, a binocular microlens was fabricated by micro-fabrication technology. Then, a smart binocular microlens imaging system is set up. By this system, the near-infrared image with venous distribution and visible light image with actual skin color can be obtained simultaneously. In addition, an improved image fusing algorithm is proposed. The image fusing result shows that the venous details can be clearly shown and the background color of the hand also can be well preserved. Therefore, the proposed scheme can provide more real vein information. As the smart structure of the binocular microlens, the dual-bands microlens imaging system has the potential for miniaturization and could integrate with intelligent glasses for assisting intravenous injection.


Introduction
Venous distribution image has important application value in identity recognition [1,2] and medical assistance. [3] The venous distribution can be captured by near-infrared (NIR) imaging as the hemoglobin in the blood can strongly absorb the infrared light. However, traditional vein display devices only contain near-infrared image information, which inevitably loses skin color information. In the actual process of diagnosis and treatment, it will give the doctor unreal feelings. Therefore, getting a more real image which contains both the venous distribution information and the background color of skin has become a new goal for researchers. An effective method to solve this problem is to fuse the NIR image with a color image.
In this article, a binocular microlens was developed by micro-fabrication process contains photolithography, coating, and thermal reflow. [4,5] After that, a smart binocular microlens imaging system is set up. By this system, the NIR image with venous distribution and visible light image with actual skin color can be obtained at the same time. Besides, a new algorithm is also proposed to fuse the two images. The experiment result shows that the venous distribution can be clearly displayed and the background color of the hand can be well preserved. As the smart structure and effective algorithm, the proposed imaging system can be applied to the venipuncture assisting device such as intelligent glasses.

Fabrication of the binocular microlens
The fabrication process of the binocular microlens is schematically shown in Figure 1.
First, the light-blocking-layer was fabricated on the substrate. As shown in Figure 1(a), the fabricate details were as follows: a glass substrate was cleaned by acetone, rinsed with isopropanol (IPA) and dehydrated in the oven at 130 C for 2 min prior to use. Then, AZ4620 photoresist (Model: AZ-4620, Company: Shipley, Kaysville, UT) with about 12 lm thickness was spin-coated on the substrate. The spin-coating parameter was 1200 rotations per minute (RPM) for 45 s and then 2400 RPM for 15 s. After baking in the oven at 90 C for 8 min to remove the residual solvent, the film was exposed to UV light under cover of the prepared mask. The mask pattern was designed using CorelDraw software (Corel Co., Ottawa, Canada) and print out on the PET film by high DPI printer. The opaque parts of the mark were two circulars, and each circular diameter is 900 lm. In this step, the exposure duration and intensity were 25 s and 20 mW/cm 2 . After developing in the AZ-400k developer for about 30 s, the sacrificial layer was prepared. Before the sputtering process, the substrate should be treated with oxygen plasma to remove residual photoresist on the substrate after development and increase surface energy to improve the adhesion with Cr metal in the sputtering process. The next step was deposited the Cr film by sputtering. Because the layer was used to block stray light, the Cr film unable to be too thin. The film thickness in our experiment was about 800 nm. At last, the light-blocking-layer was formed after cleaning in acetone.
Second, the two-channel filters were fabricated and the fabricate details were as follows: the AZ4620 was spin-coated on the substrate. As the process described above, after pre-bake, the substrate was exposed to UV light under a new mask. The transparent part of the mark was a 900 lm circular which corresponds to one of the transmission holes made before. After development, a transparent circular region was obtained. Next, the IR-cut filtering film was sputtered on the transparent region. Then, the visible band-imaging region was formed after liftoff in acetone. The process needs to be repeated one time and the NIR filtering film should be sputtered on another transparent circular region (as shown in Figure 1(b)). In this article, the coating materials of IR-cut filter and NIR filter are TiO 2 and SiO 2 . The film structure of the IR-cut filter is A|ð0:5LH0:5LÞ^3|G and the film structure of the NIR filter is A ð0:33H0:64L0:33HÞ^4ð0:5HL0:5HÞ^3G; where H represents TiO 2 , L represents SiO 2 , and A represents the air and G represents Glass.
Third, the binocular lens will be integrated into the filters made above. The AZ4620 was spin-coated on the substrate twice in order to get a thick enough photo-resistant layer and each spin-coating parameter was 1200 RPM for 45 s and then 2000 RPM for 30 s. After coating, the thickness of photo-resist can reach about 23 mm. The temperature of pre-bake was set to 90 C for 4 min to remove the residual solvent absolutely. After that, the sandwich was exposed to UV light under a prepared mask. The opaque parts of the mark were two circulars and each circular diameter was 700 lm. The duration of UV-light exposure process was 12 s and the exposure intensity is about 20 mW/cm 2 . Following the above process, the sandwich was put into the AZ- In order to control the height of the cylinder after developing, multi-development with height measurement by step apparatus (AMBIOS XP -1) at intermission were adopted, and finally, the height of the cylinder was reached around 20 um. After development, the substrate was rinsed by deionized water for removing remains developer and then dried by nitrogen. At last, the sample was brought into the oven for the thermal reflow process. After heating at the temperature of 128 C for 115 s, the binocular microlens was formed on the filter (as shown in Figure 1(c)). The sample of the binocular microlens and the profile of the microlens is shown in Figure 2. The micrograph is in the left corner of the image. The contours of microlenses were measured by the XP-1 step apparatus. The profile curve shows that the diameter of the microlens is about 890 lm and the height is about 18 lm. The average refractive index of AZ460 photoresist is about 1.65 at 400-1000 nm wavelength. By calculation, the focal length is about 8.48 mm and the numerical aperture is about 0.086.

Setup of the micro binocular vision system
In order to verify the imaging performance of the binocular microlens, we built a binocular imaging system. As shown in Figure 3(a), the system mainly consists of a light source, the binocular microlens, a CMOS image sensor, and a 3D platform. Because the vein texture of the back of the hand can be clearly displayed at the wavelength of about 850 nm, we use a customized ring LED array light source which works band is 400-1000 nm. The microlens and image sensor were used to capture the image of the object (hand), the 3D platform was used to adjust the relative position between the microlens and image sensor. The image acquisition principle of the system is shown in Figure 3(b). Each microlens and the filter behind it are together considered as an image unit. As a result, two image units (visible band and NIR band) could complete the imaging process independently.

Original image acquisition
In order to obtain the venous distribution, the back of the hand was chosen as the imaging target. The original imaging result is shown in Figure 4. The infrared image with venous distribution and visible light image with color information could be obtained simultaneously. However, the clear vein distribution and actual color information cannot be displayed in one picture. For achieving this purpose, a new image fusion algorithm was proposed in this article.

Image fusion process
The new image fusion algorithm is based on histogram mapping, principal component analysis (PCA) and bilateral filter. The process detail is shown in Figure 5. The core-processing step is to convert the dual-bands images to the Intensity-Hue-Saturation (IHS) space at first and match the brightness component of the NIR image with that of the visible image. We named this image processing process as histogram mapping. Next, the color information is extracted from the visible image by PCA transform and processed by residual enhancement to enhance the color components. Then, the bilateral filtering process is used to get brightness and vein texture information. And finally, we get a fusion picture containing all the information. The above processing algorithm can ensure that the venous texture of NIR image and the color information of visible image without lost.
First, the NIR image and visible light image were segmented from the original image. Then, they were cut and aligned to the same scale and position. The two images are shown in Figures  6(a and b).
Second, the two pictures were transformed by IHS transform to obtain the brightness component. Then the brightness component of the NIR image was histogram mapped with the brightness component of the visible light image so as to get the image I N : Third, performed PCA transformation on the visible light image and enhance its color component. After that, image E could be obtained by PCA inverse transform.
Finally, image I N and image E were processed by bilateral filtering and superposition fusion algorithm to get the final fused result.

Normalize and histogram mapping
During the imaging process, the captured images will show different characteristic as the different light intensity of visible light and NIR light. After IHS transformation, [6] the brightness component histograms of the two images were obviously different (as Figures 6 (a and b) shown). If the two figures are fused directly, the final fused results would show obvious brightness difference compare to the original image. So, we propose a method named as Histogram Mapping to settle this problem. The processing procedure is shown below.
First, apply the maximum-minimum normalization process to brightness component of the NIR image, the formula is shown as follows: pix 0 x;y ð Þ ¼ pix x;y ð Þ À pix min pix max À pix min (1)    where pix ðx;yÞ is the original value of each pixel. pix 0 ðx;yÞ is the corresponding pixel value after normalization, pix max and pix min respectively represent the maximum and minimum of brightness component value.
Second, record the maximum and minimum brightness component value of the color image, the maximum and minimum value are expressed as pix R max and pix R min ; respectively. Third, make the extremum of the NIR image corresponding to the extremum of color image. The processing procedure can be expressed as follows: pix fusion By this processing, image I N is obtained and the brightness components of NIR image and color image will change in the same range.

PCA transformation and residual enhancement
Through PCA transform, [7] we can get three components. The first component represents brightness information which without color signal. The other two components represent color information. Through selecting an appropriate weight ratio for the two components, we can effectively improve the color distribution of the fused image.
First, the three-dimensional color image matrix X ½A;B;N is transformed into a two-dimensional matrix X ½AÂB;N : A and B represent the length and width of the color image, respectively. The dimension of the image is M ¼ A Â B; N ¼ 3: Thus, X can be expressed using the following equation: In the formula, x i ði ¼ 1; 2; :::; mÞ is a column vector and X is a M Â N matrix. The covariance matrix can be gained by the following formula: Then calculate the eigenvalues and eigenvectors of covariance matrix XX T ; and marked as ; where r is the rank of matrix XX T ; q j ð Þ is its eigenvalues and c j is the eigenvectors. Transformation matrix can be represented as Q ¼ q ð1Þ q ð2Þ q ð3Þ Â Ã ; q 1 ð Þ > q 2 ð Þ > q 3 ð Þ : For each x k ð Þ ; there is a relationship as shown in Equation (5): The Y ¼ ½y ð1Þ y ð2Þ y ð3Þ is a projection matrix, where Y is a N Â M matrix. The k-channel enhanced image signal y ðkÞ e can be defined as the following equation: [8] where W is a M Â M matrix, y is the mean of the y k ð Þ : The result of the enhanced image is determined by this matrix. The line p and column q of matrix W can be calculated by Equation (7) as follows: In this formula, @ is determined by the mean difference of two images' color component. This process can ensure that the color information of the final fused image is consistent with the original color image.
Then through PCA inverse transformation [7] on the output matrix Y, we can get the image E. Because the bilateral filtering algorithm has a good performance on edge preservation, the edge and texture information can be effectively extracted from the image I N : Therefore, next, we will process the image I N by bilateral filtering algorithm and fused it with image E to get the final color image.

Bilateral filtering fusion
First, bilateral filtering algorithm [9] was applied to the brightness component of I N to obtain I N BL : Second, image I N BL was subtracted from image I N to get image I DT : This process is described as following: It can be seen that the image I DT contains a lot of texture information. After superimposing the image I DT with I N ; we got the fusion image I fusion as follows: At last, the final image is obtained by fusing image E and I fusion : Image E is in charge of the color information and image I fusion is in charge of the details and brightness information.
By the method proposed above, the venous distribution and texture information from NIR image can be effectively retained, and meanwhile, the real color of the skin can be preserved. Figure 6(b) is the fused result of the proposed method. It can be seen that the venous distribution details were shown obviously and the color of the skin is real as expected. This image can help doctors or nurses to accurately determine the location of blood vessels and carry out the intravenous injection.

Results and discussions
In order to evaluate the fusion effect of the proposed algorithm, three typical algorithms, IHS fusion algorithm, [10] PCA fusion algorithm, [11] and GIHS fusion algorithm, [12] are used for image fusion comparison testing. Additionally, we evaluated the result by three parameters, [13] namely standard deviation (STD), average gradient (AG) and average distortion (A.D).

Standard deviation (STD)
Standard deviation represents the discrete degree of the pixel value relative to the mean. The larger the deviation is, the more dispersedly the gray grades distribute, and if the probability of each gray grade equals, the information capacity is the maximum. The STD r is calculated by the following formula: where M Â N is the image scale, x i;j is the value of the position (i, j) in certain component (red, green, and blue). l mean is the average value of all the pixels in the image, which is calculate by formula (11):

Average gradient (AG)
Average gradient is able to reflect the fine contrast of the image. Generally, the larger the average gradient is, the clearer the image is.
where Dxf x; y ð Þ and Dyf x; y ð Þ are the difference of f x; y ð Þ at x and y direction in a certain component, respectively.

Average distortion (AD)
The definition of the spectral distortion in the kth component is the following: In the formula, V k i;j V and V 0k i;j are, respectively, the gray values of the original color image and fused image in the kth component at position (i, j). As the spectral effect is expressed by the synthesized of the multiple bands, we employed the average distortion, noted as AD, to illustrate it. The greater the AD is, the larger the distortion is.  Figures 7(a-c) show the fusion results obtained by the IHS fusion algorithm, PCA fusion algorithm and GIHS fusion algorithm, respectively. As shown in Figure 7(a), the IHS fusion algorithm could preserves the vein distribution in general, but the skin color is distorted, especially at the edge of the hand (as indicated by the white circle in the picture). From the PCA fusion image (Figure 7(b)), it is shown that the color of the veins is close to that of the skin around them. As a result, the veins distribution is not prominent, especially for the region around by the white circle. Besides, the edge of the hand is blurred to some extent. Figure 7(c) shows the GIHS fusion result. From the picture, it can be seen that the edge of the hand and some veins blurred (as the white circle shown in the picture). Figure 7(d) is the fusion result of our algorithm. It can be seen that the vein distribution is highlighted and the skin color of the hand is more real. This also can be confirmed by the calculated results of the image evaluation parameters abovementioned.
The calculated results of STD, average gradient (AG), and average distortion (AD) of Figures  7(a-d) are listed in Table 1. Higher values of STD and AG indicate that the veins distribution is more outstanding. If the value of AD is low, it means that the color information does not change much during the fusion process. From the table, we can see that the image-fused result by the proposed method has maximal STD, maximal AG, and minimal A.D. It indicated that the proposed method is better than the contrastive algorithms as the distribution of veins can be highlighted and the real color of the skin can be preserved.

Conclusions
In this article, we developed a binocular microlens based on microfabrication technology and set up a smart binocular microlens imaging system. Through this system, the visible image and NIR image could be captured at the same time. Then, the system is used for vein enhance display. Besides, an improved image fuzing algorithm is proposed. By this algorithm, the visible light image and the NIR image is well fused. The fusion result shows that the vein distribution and the color of skin were well preserved. Compared with several typical image fusion algorithms, the results show that this method has more advantages in fusion effect. As the smart structure of the binocular microlens, the proposed imaging system has the potential for miniaturization and could integrate with intelligent glasses for assisting venipuncture.

Disclosure statement
No potential conflict of interest was reported by the authors.