Development of a scalable tabletop display using projection-based light field technology

We present a light-field display optical system design, state-of-the-art hardware technology, and computational software that are readily scalable (due to their modular structure) and provide naturally immersive and volumetric display systems. We explain the integration of 72 microprojectors into the tabletop display system. In addition, we used 6 workstations for all images and 18 high-performance graphics processing units. Sophisticated image generation through author tooling and the pixel re-arrangement algorithm in the Unity engine enabled us to produce complete three-dimensional images from the radiant rays of two-dimensional pixels using diffuser screens and microlens arrays.


Introduction
The last several decades have seen tremendous efforts to develop and realize natural and realistic displays, i.e. tabletop three-dimensional (3D) displays, which are the ideal and ultimate displays). Major researchers and developers were involved in the development of basic 3D displays to resolve vergence-accommodation conflict [1] and to realize multiviews and full parallax. Among these 3D displays, light-field technologies that employ plenoptic functions or four-dimensional functions have recently attracted interest [2][3][4][5]. The light field can be described as a set of rays that pass through every point in space. It can be parameterized for computer vision in terms of a plenoptic function L, which is computed as follows: where (V x , V y , V z ) corresponds to a point in space, and ( , ) corresponds to the orientation or direction of a ray depending on the wavelength of λ [6]. In a light-field display, every direction of the rays in every position of the unoccluded space enables easy handling and convenient rendering of new views. Radiant rays in the light-field display are rendered descriptions of the light field that can be projected through each microlens array for 3D images in space. Many two-dimensional (2D) pixels represent radiant image sources, where the origin, direction, and intensity of the light rays are described by the plenoptic function. The light-field display is designed to display 3D content in a manner that appears natural to the human visual system by providing natural and consistent stereo, parallax, and focus cues. Light-field display techniques are categorized mainly into two types: group/ multiuser light-field displays and personal light-field displays (e.g. near-eye and headmounted displays). The main task of a light-field display is to create a correct retinal blur that corresponds to the 3D location of an object. We can embody this task by presenting multiple views (the integral imaging approach) to create a parallax across each eye that produces the correct retinal blur that corresponds to the 3D location of the object being viewed. In this study, we used mainly multiprojectors to realize these tasks [4,6]. Over the years, some studies were carried out by Holografika [7], featuring only horizontal parallax, but it is an excellent motion parallax. Holographic displays from Zebra Imaging [8,9], Fovi3D [10], the National Institute of Information & Communications Technology and University of Tokyo groups in Japan [11,12], and the Samsung Advanced Institute of Technology have been developing light-field displays in laboratories [13].
To address the issues presented in the abovementioned studies, we introduce a light-field display (tabletop display prototype) system, including the computational methods and optical systems that we used. This system is quite flexible and scalable (i.e. expandable due to its modular structure) and has a collaborative architecture. Specifically, we used multiple (72) commercial microprojectors to illuminate images onto the local screen regions, generating entire images by tiling several individual images most precisely. Our study results differ from those of previous studies for the following reasons. First, we used commercial microprojector engines by slightly modifying the signal I/O and firmware to reduce the cost of the projectors. Second, we designed and developed additional telecentric lenses at a low cost using plastic materials. Third, we used simple computer graphics processing units (GPUs) without fieldprogrammable gate arrays (FPGAs). Fourth and last, we realized the largest size of 35"' for tabletop displays using multiprojector-type light-field display technology, which is readily expandable by adding more projector units and workstations. This is the core of the technology we developed, which provides a flexible, scalable, and modular light-field display system. To our knowledge, this is the first time a 35" and still expandable high-resolution, full-color tabletop display system is introduced.

Experiments and results
We integrated integral imaging into a tabletop display system that uses a projector-based light-field display via a diffuser screen and a microlens array. A light-field display renders all perspective views regardless of the number or position of the viewers around a table. Perspective views are projected through a microlens array to produce a 3D light field for all the viewers simultaneously [1,2]. The schematic concept of the tabletop display design and the principle of integral image rendering are shown in Figure 1. Integral image elements comprise a certain number of pixels covered by lenslet arrays, through which the rays are angularly radiant.
We used 72 commercial HD (1280 × 720) microprojectors to develop a 720-ppi 3D image resolution on tabletop display systems. We modified the circuit design and the firmware of these microprojectors and designed and fabricated four additional pieces of telecentric lenses for the optical 'projector unit' of the tabletop display system assembly. Table 1 shows the parameters of the projector unit, which are the basic configuration of the tabletop display and the fundamental design concept of a single projector unit. The design simulation results are explained in Figure 2, and the detailed specifications of the projector    unit are 169.5 mm from the projector outlet to the screen and 100 × 56 mm from the local projected area. Earlier work has been reported elsewhere As mentioned previously, the projector unit allows us to resize the tabletop display. This tiling type allows the combination of projector units to create a tabletop display of any size and requires additional PC installation and image processing based on the arrangement of the projector units. We designed the telecentric lens part of the projector unit as a single barrel to enable tiling. The manufactured barrel is shown in Figure 3.
The effective diameters of the two aspheric lenses were 23 and 25 mm, and the lens materials (polymer resin) were high-quality optics-grade E48R and OKP-1, respectively. The effective dimensions of the third telecentric lens (the field lens) were 99 × 55 mm, and its two axes had asymmetric shapes with a center thickness of 7 mm. A 5cm-thick BK7 glass substrate was tightly laminated with a 250μm-thick diffuser sheet that had a 40°diffusing angle. The dimensions of the microlens array were 100 mm (L) × 56 mm (W) × 0.5 mm (H), and it had an 800μm pitch, a microlenslet radiant angle of ≥ ±52°, a spot size ≤ 80 μm, a distortion ≤ 2.34%, and the lens material E48R. All three telecentric lenses and lens arrays were injection-molded for mass production. The metal core molds were precisely fabricated based on optical designs. We aimed for the final tabletop display specifications of a 35" diagonal, a ±25°wide viewing angle, a 720p resolution, a 5°angular resolution, and a frame rate of 30 frames per second (fps).
To construct a high-definition 2D display, we propose a system for the synchronization of the image processing of 72 projection units using the spatial multiplexing technique, six PC configurations with three GPUs, and the Mosaic technology of NVIDA, which is famous for its expansion of a single workstation across up to 16 high-definition displays or projectors without sacrificing performance or power. Although the Mosaic scheme is quite useful for multi-displays, there are several limitations to its application. For example, it is only applicable to square-type arrangements (such as 2 × 2 and 3 × 3), so our proposed system should be properly divided into several segments. We split the total system into six parts (with each part using 12 projector displays), each of which corresponds to a single computer equipped with three GPUs (Quadro P4000, NVIDIA), as shown in Figure 4. Each computer was connected to Quadro Sync board II B/D, so all our configurations were synchronized to 20-48 ms, which is the minimum rate for the successful real-time synchronization of interactions.
Each small projection display was designed for seamless spatial multiplexing. The resolution of each projection display was 1,280 (H) × 720 (V), and the single module consisted of 2 (H) × 4 (V) display units. The total system consisted of 3 (H) × 3 (V) modules. Therefore, the system resolution was 7,680 (H) × 8,640 (V). The microlens array for each projector contained 128 (H) × 72 (V) lenslets, each of which lenslet contained 10 × 10 pixels for the best embodiment of light field functionalities, as shown in Figure 5 [8][9][10].
To generate multiview images from a 3D object in a virtual world such as Unity, the 3D objects were captured with virtual camera arrays (10×10). In general, the camera is forward-faced, and it horizontally or vertically equidistantly shifts in a plane to capture multiple images. Such configurations have the drawback of maintaining a large noninformative area of unused pixels. Therefore, for our proposed method, we used the oblique orthographic camera configuration as our re-entering camera configuration. To change from a normal camera capture method to an oblique orthographic camera configuration, we defined a projection matrix using shade in Unity. Once the 3D objects were captured by a virtual camera in Unity, each captured image was rendered using the projection matrix, and the output image was transformed into images captured by oblique orthographic cameras. In our setup, the resolution of the virtual camera in Unity was 768×864, and the number of cameras was 10×10. After this process, the pixel rearrangement algorithm, which is frequently used in integral imaging, was applied to generate elemental images. The pixel (n, m) in the virtual camera #N_M th was arranged into pixels (n', m') in the lens array; therefore, the resolution of the elemental image was (768×10) (H) × (864×10) (V), as shown in Figure 5. To align the projection rays precisely and to adjust the screen height easily, we used a translational aspheric lens housing and microlens arrays. Furthermore, we fabricated several pieces of customized fixtures and microtranslators for the best optical alignment ( ≤ mm order) of each microprojector image. Figures 6-7 shows the horizontal/vertical parallax images of the tabletop display system. We confirmed that the viewpoints of the 3D display system were 10 × 10 horizontally and vertically. We tested the 3D display depth by adjusting our eye positions approximately 75 cm from the display center and looking closer to and farther away from the display screen. We also carefully viewed images at a 360°angle, and finally, we produced the depth-sensitive tabletop displays, which exhibited an optimal distance approximately 80 cm away from the center of the tabletop display and a viewing angle of approximately ±25°from the display normal. Color variations from one projector to another were calibrated and adjusted through software adjustments, and geometrical distortions such as barrel and pincushion distortion were finely controlled electronically from one projector to another. Finally, we developed a scalable display by tiling the modules. Optical alignment and performance testing of the developed scalable tabletop display are underway.

Conclusion
In this study, we developed a 35" and still enlargeable high-resolution, full-color tabletop display system that, to our knowledge, is the first of its kind. Our research results are different from those of previous studies [8][9][10] in that first, to achieve lower-cost projector engines, we used commercial microprojector engines by slightly modifying the I/O signal and firmware. Second, we designed and developed additional telecentric lenses at a low cost. Third, we employed simple computer GPUs without FPGAs. Finally, we developed the tabletop display with the largest size of 35" but was still readily expandable using multiprojector light-field display technology. This is the core technology that we have developed, which provides a flexible, scalable, seamless, and modular light-field display system. In the future, we will focus on improving the 3D measurement system for large display systems.