Depth-sensing technology using a negative microlens array

Abstract In this study, an optical depth-sensing system whose focus distance was equal to its object distance was constructed by using a lens with a shortened depth of field. In addition to a mechanical control lens that captured two-dimensional images, a set of negative microlens array films used to shorten the depth of field was installed in this system. The developed system can scan a region and obtain the relationship curve between the focus distance and the motor step. Depth information is acquired by discerning the sharpness of the object’s outline and determining whether the captured object is in focus. The motor step is initialized when the object is in focus. The optical path of the developed system is simple, and its volume is minimal; thus, the developed system is suitable for being combined with cell phone lenses.


Introduction
Depth sensing is regarded as a basic technology for many optical applications. This technology was developed to address dissatisfaction with existing flat-imaging techniques and to capture more comprehensively the information of the surrounding space beyond two-dimensional (2D) shapes and colors, such as the contours of the third dimension, depth, and material and surface defects of objects. Depth sensing has been used in numerous optical applications that require three-dimensional (3D) information, such as machine vision, geomorphological systems, biomedical systems, and 3D reconstruction.
Recently, mainstream depth-sensing technology can be categorized into active and passive scanning. The measurement process of active scanning is sensitive to external effects, and the resolution of active scanning can be suboptimal because of the difficulties in decreasing the size of light spots. [1][2][3][4][5] Multiple lenses are required in passive scanning to observe a target from different angles; therefore, passive scanning has a complex setup and involves complex computations. Moreover, the resolution of passive scanning cannot reach that of 2D images. [6][7][8][9] To overcome the limitations of existing depth-sensing technologies, in, [10] we proposed the use of diffusers to alter the depth of field in biomedical optics. We added a diffuser behind a mechanical control lens that captures 2D images to shorten the lens' depth of field. [11] The obtained clear image then covered only a short distance, which essentially equalized the focus distance and object distance. Thus, the aforementioned sensing system must only know the focus distance for acquiring the depth information of a target, without no complex calculation being required. The feasibility of this system was validated experimentally.
In addition, other research teams have proposed lensless light-field imaging [12,13] to create a shallow depth of field through the replacement of the traditional lens in an imaging system with a Fresnel zone aperture (FZA). Instead of a traditional lens, an FZA pattern is added into a single-piece Fresnel zone plate (FZP), which is placed in front of a photosensitive element as a mask. The photosensitive element can record the light intensity distribution projected onto it through the FZP. When the image is restored, the projected light intensity map is superimposed with a virtual FZA pattern and the generated moir e pattern contains the spatial information of the object light (i.e. the direction of the object light). Therefore, the image can be reconstructed after a fast Fourier transform is used. The virtual FZP with different depth parameters is optimized for the corresponding depth image after the Fourier transform is restored. Thus, through the adjustment of the depth parameters of the virtual FZP, images with different depths can be obtained.
However, when the lens size is reduced to that of cell phone lenses, limited resolution is achieved. Considerable space is required for the movement of the diffuser and FZP; therefore, their application in cell phones is difficult. Our proposed design uses an optical imaging system lens. According to the design of Galileo's telescope, a negative microlens array film was used to reduce the volume of the designed optical system. This film was used to calculate the depth information of an object and to achieve a telescope-like shallow-depth-of-field effect. The minimal thickness of the aforementioned film (in microns) markedly reduced the size of the developed optical system.
On the basis of the sensing principle of sensing systems with diffusers, the depths of various objects within the field of view were calculated through the continual adjustment of the focal distance while capturing in-focus or out-of-focus 2D images for achieving depth sensing. The developed depth-sensing system with a negative microlens array film can acquire color information effectively in any environment.

Method
The depth-sensing technique proposed in this study requires multiple 2D images with a shallow depth of field to be captured from several focal distances for calculating depth information. A negative microlens array film based on the design of Galileo's telescope was used to achieve a shallow depth of field. The negative lens units of the array were combined with a sensor to create a telescope-like structure. However, Galileo's telescope is intended for astronomical observations and thus requires a larger aperture to increase the structural volume and lengthen the optical system. Because this design is unsuitable for depth-sensing purposes, we modified the structure with reference to the telephoto lens systems of cameras. [14,15] As depicted in Figure 1, two sets of lenses are employed in a telephoto lens system, namely a convex lens set for concentrating object light at the entrance and a concave lens set for lengthening the back focal distance at the center. A 2D image can be captured through the placement of the sensor at the effective focal length generated with these two sets of lenses. The lens imaging formula can be Three-dimensional FZA Fresnel zone aperture FZP Fresnel zone plate written as follows: where EFL is the focal length of the lens and t and s are the image and object distances, respectively. The effective focal length is expressed as follows: and where EFL t is the effective focal length of the telephoto lens; f 1 and f 2 are the effective focal lengths of the convex and concave lens sets, respectively; d is the distance between the lens sets; and e is the distance between the concave lens set and the sensor. The effective depth of field of the telephoto lens can be expressed as follows: where N is the aperture number of the lens and c is the diameter of the circle of confusion. The depth-sensing system illustrated in Figure 2 was designed to acquire the depth information of objects. A negative microlens array film was placed between the sensor and a fixed focal  lens, with the negative microlens array units forming a combination similar to a telephoto lens. With the aforementioned structure, a shallow depth of field was achieved by minimizing the circle of confusion and generating a partial magnification effect. By determining whether an image is in focus, depth information can be acquired according to the concept that the object distance is equal to the focus distance. The focal distance of the fixed focal lens was changed within a certain range by using a step motor, which enabled the sensor to capture a 2 D image with a depth of field of up to a few meters.  3 and 4 present the simulation results of imaging with the sensing system before and after the installation of the negative microlens array film. The Zemax software was employed to obtain the aforementioned results. The left parts of Figure 3 depict the optical paths after the object light was captured by the lens, and Figure 3(a) illustrates the image obtained after the object light passed through the lens. Because each negative microlens unit corresponded to a certain area of the lens and constituted a small optical system, we set the unknown parameters of the negative microlens to correspond to a lens aperture of 5 mm and a focal length of 20 mm. These parameters served as a reference to verify that the simulated image of the focus was clear.  Figure 3(b,c) illustrates 1 m forward and backward adjustments of the object distance, respectively. Moreover, these figure parts represent a ± 5 mm difference in the in-focus position compared with that displayed in Figure 3(a). The right parts of Figure 3 display the images corresponding to the optical paths depicted in the left parts of Figure 3. As displayed in the aforementioned figure, no obvious change occurred for the imaging. Figure 4 displays the image obtained after the addition of a negative lens. The left parts of this figure represent the optical paths after the object light was captured by the lens. At this time, we increased the assumed focal length of the system to 40 mm. Following optimization, the negative lens (indicated in red in Figure 4) had an approximate aperture of 1.5 mm. From the optical path depicted in Figure 4(a), the object light entering the lens extended the focal length through the negative lens and focused on the sensor. We adjusted the object distance by 10 cm forward and backward, as detailed in Figure 4(b,c), respectively, which resulted in a ± 5 mm difference in the in-focus position compared with that depicted in Figure 4(a). As illustrated in the right parts of Figure 4, the clearest image was obtained when the object light was focused on the sensor. After the focal length was increased and the object distance was marginally altered, the focus was in front of and behind the sensor and the image became blurred. Thus, with the addition of a negative lens, a shallow depth of field was achieved.

Experimental results and discussions
To verify the feasibility of depth sensing by using a shallow depth of field achieved with a negative microlens array, a depth-sensing system capable of capturing 2 D images was constructed ( Figure 2). The negative microlens array film was the key element and was arranged in an aspheric design. Traditional lenses are only produced in plane, spherical, and paraboloid shapes, which poses a challenge for achieving high image quality; thus, optical aberration can only be eliminated through combinations of multiple lenses. An aspheric lens with a film-like thickness was developed to solve the aforementioned problem. It ensured that light could pass through and focus on the same spot easily. The traditional equation of a sphere comprises only one parameter; however, an asphere combines fixed curvature with an arbitrary implicit curve and thus has a fixed curvature of varying heights. One common type of asphere is a curved surface that has a rotational symmetry, whose mathematical equation can be expressed as follows: þ a 4 r 4 þ a 6 r 6 þ a 8 r 8 þ a 10 r 10 , a ¼ 1 where R is the curvature radius; k is the conic constant; z is the thickness of the lens; r is the center radius of the lens; and a 4 , a 6 , a 8 , and a 10 represent aspheric coefficients. The conic constant represents different shapes in different numeric intervals. The generic aspheric lens was oval in design with a condition of À1 < k < 0. In addition, because of the minimal center radius of the microlenses, the diameter of the circle of confusion was equal to 2r. Only a quadratic term was required, and the high-order term could be omitted. According to the diameter of the circle of confusion obtained using Equation (4), suitable negative microlens parameters were set ( Figure 5). The central diameter of the units in a negative microlens array is 250 mm, with the units arranged in a staggered manner and separated by 500 mm. The optimal radius of curvature of the aforementioned units is 155.72 mm, and the conic constant is À1.019592. A negative microlens array with the aforementioned parameters was produced, with the unit thickness being approximately 50 mm. This array was made of transparent resin and produced using the Heidelberg DWL66þ laser lithography system, with a blue laser employed to expose the AZ4620 photoresist and etch the microstructure. The detailed structure of the produced microlens array is displayed in Figure 6, in which the staggered arrangement of the microlenses is visible, with the design specifications conforming to the parameters of negative lens units.
The minimum and maximum focal distances of the fixed focal length of the depth-sensing system were 20 and 150 cm, respectively. Moreover, the angle of view of this system was approximately 46 . The aforementioned system also contained a SmartSens SC2232H sensor module, which was controlled using a self-developed backend program. The developed depth-sensing system comprised a fixed focal lens, negative microlens array film, and sensor ( Figure 2). The backend program controlled the fixed focal lens to capture 2D photos at each node, and a Laplacian mask was used to perform edge detection on the image in the negative microlens for determining  whether the object was at the current focal position. The total length of the depth-sensing system was 7 cm, and this length is similar to that of a small camera.
Prior to the depth-sensing experiment, the developed system was calibrated to determine the relation between the motor step and the focal distance. The test object was a checkboard [ Figure  7(a)] placed on a point at a certain known distance in front of the depth-sensing system. The lens of the developed system was driven from the nearest focus distance and gradually focused on the checkerboard. When the checkerboard was in focus on the microlens unit, the program recorded the current motor step, which was used for determining the corresponding focal distance. The aforementioned process was repeated for different focal distances from the closest to the farthest focal distance, with every 5 cm being considered as the motor step of a node. The relationship between the motor step and the focal distance is illustrated in Figure 7(b). A cubic curve function can be fitted as follows: y ¼ 10 À5 x 3 À 0:0321x 2 þ 23:606x À 5759:8: Thus, the relationship between the motor step and the focal distance in non-node locations can be calculated using Equation (5).
After calibration, the depth-sensing experiment was conducted. The calibrated depth-sensing system was placed on the side of a flat surface of sufficient length and within the visible range of the system. Then, we placed three dolls within the focusable depth range to act as the measured objects. We defined the maximum plane of the doll body closest to the lens as the actual distance. The distance between the three dolls and the depth-sensing system was 30, 70, and 100 cm. The backend program was used to control the lens of the depth-sensing system to scan from the closest to the farthest focal distance.
Some of the 2D photographs captured during depth scanning are presented in Figure 8. Figure  8(a) displays a photograph captured by the fixed focal lens. Since the size of the lens unit of the negative micro lens array film is extremely small, when the negative micro lens unit is not in focus, the image captured by other non-negative-micro-lens areas are the same as when shooting with only a fixed focus lens. At this time, the depth of field is obviously more than one meter, and each measured object can be clearly distinguished. Figure 8(b) is the image of all measured object which not in focus after starting the depth scanning, so each measured object is extremely blurred. At this time, for negative micro lens unit, there is no measured object in focus, so it has no effect. When scanning through the depth area of the measured object, the negative micro lens unit begins to exert the effect of shallow depth of field, the measured object will be in the focus position, and it can get the Figure 8(c-e) which are the images when the three measured objects are in focus respectively. Due to the shallow depth of field, only the edge of the one of the measured objects can be seen clearly in the negative micro lens unit that captures the object light. On the contrary, in the negative lens unit that captures the object light of the other two measured objects cannot be shown the clear edge. And then Figure 8(f-h) show the enlarged images when the three measured objects are within the focal length of the negative micro lens unit, and compare them with the original focused images respectively. The left half of the figure is the original focused image, and the right half of the figure shows the focused image in the negative micro lens. In both cases, the red circles at the same position correspond to each other. It can be seen from the figure that the negative micro lens unit has a very obvious magnifying image with clear edge or feature of the measured object. Therefore, it can be determined whether the image in the negative micro lens unit is in focus, and compare to the non-negative-micro-lens-unit part of the image. To confirm the measured objects to which the enlarged edge or feature image belongs.
Because of the minimal aperture of the negative lens units, only a small region was visible to each unit; however, the visible region was sufficient to determine whether the object was in focus. The spatial relationship between different objects could be identified with ease according to the focus status. Thus, we shortened the depth of field by using a microlens array film. In addition, when a doll was in focus, the focal distance of the clearest image could be determined from several relatively clear images by using the Laplacian mask. Because the motor steps were simultaneously recorded by the backend program, the depth could be calculated using the relation curve illustrated in Figure 8(b). The estimated distance of the three dolls was 31, 75, and 108 cm, which differed only marginally from the actual distance. The relative deviation was within 3%. Thus, the developed depth-sensing system had high accuracy. In the experiment, depth sensing was performed in approximately 1 min, with the sensing time increasing or decreasing depending on the number of measured objects.
In the experiment, the accuracy of the depth-sensing system was measured using a depth test. Figure 9 presents the results of this test, in which the system focused on objects in various positions. The left vertical axis represents the measurements from the near and far focus limits, whereas the right vertical axis represents the different depths of field. With the addition of the negative microlens array film, the depth of field was reduced from several meters to within 50 cm. The shorter the focal distance, the smaller was the depth of field. In contrast to the experimental results, according to the depth-of-field curve, when the object was placed 30, 70, and 100 cm away from the developed system, the depth-sensing result was 30 ± 2, 70 ± 8.5, and 100 ± 17.5 cm, respectively. These results are within the range of focus distances. Because of the miniscule size of the negative lens units and limited resolution of the sensor, the focus effect of the negative microlens unit was restricted. The actual depth of field was shorter than the measured depth of field. If a sensor with higher resolution is used, the accuracy can be enhanced.
In addition, because the depth of field was a few centimeters and the size of the negative microlens unit was small, the system could not image objects with close depths or positions on the same vertical optical axis. When capturing two or more close objects, the number and relative positions of the negative lens units of the object light differ only marginally. Although the image of the non-negative lens area can be distinguished, the accurate depth value of the two objects is difficult to obtain. Therefore, in the future, the parameters of the negative lens, such as the size of the negative lens unit and spacing of the array, can be adjusted to identify the parameter combination that captures the most information. In addition, the position of the depth-sensing system can be changed because the lens is not required to be on the same optical axis as the object. Different viewing angles such as the bird's eye view or lateral movement can be used to obtain depth information closer to the original viewing angle of the object. The depth-sensing system proposed in this paper has a basic structure. The depth information and image of an object can be captured using several negative microlens units that are arranged in a staggered manner and focus on the outline of the object. The optical path of the developed system is simple. Through the adjustment of the intervals and calculation of the suitable negative microlens parameters, the developed system can be applied to different hardware. For example, focusing distance is related to the furthest focal length of the lens; therefore, to expand the sensing distance, a telephoto lens that can focus to a further distance can be used. The SmartSens SC2232H sensor module has a resolution of 1080 p, and the system accuracy can be enhanced by replacing this sensor with a higher-resolution one. Furthermore, the sensing speed can be improved using a 960 fps high-speed camera. The scanning time can be considerably reduced when the images required for depth scanning are captured in a short time.

Conclusion
In this study, a depth-sensing system was developed. By using a negative microlens array film to shorten the depth of field of a fixed focal lens, the focal distance was equalized to the object distance. A negative lens array structure was successfully produced with suitable parameters on a resin film and installed into the developed shallow-depth-of-field depth-sensing system. In a depth-sensing experiment, we verified that after the developed system scanned a certain distance, the depth information of objects could be acquired by distinguishing their outlines in the shallow depth of field by using the negative microlens. Objects placed at distances of 30-100 cm were imaged, and the focal distance of the lens was successfully reduced from 2 and 1 m to 35 and less than 10 cm, respectively. The adopted technique only requires the use of a single lens with a negative microlens array film. The developed system, which has a simple light path and minimal total volume, can be incorporated into the small lenses of various consumer electronics for enhancing their functionality.

Funding
The authors would like to thank the National Science Council of the Republic of China, Taiwan for financially supporting this research under Contract No. MOST 109-2221-E-035-075.