Hyperrealistic frameless display for future displays

ABSTRACT Described in this paper is a novel hyperreality display called ‘head dome projector (HDP).’ HDP is a head-mounted display consisting of a dome-shaped screen with a very small radius (40 cm), a mobile projector with an ultra-wide projection lens, and light-emitting diode (LED) light sources. The principal feature of HDP is a very wide-viewing angle: 160° horizontally × 120° vertically, comparable to the human visual field, without head tracking, and 360 × 360° with head tracking. In the subjective evaluation and comparison of HDP with a flat-panel display (FPD), HDP achieved 2.5-points higher hyperreality than FPD in the case of a ±5-level evaluation for high-definition motion images. Also proposed herein is a novel hyperrealistic head-up display (HUD) concept, the windshield-refracted augmented reality projector (WARP), and described is the developed prototype WARP system. It uses monocular vision, which eliminates the depth cues caused by parallax information. The developed WARP system achieved free depth control of the HUD image position.


Introduction
The audiovisual-experience automatic virtual environment [1] and CyberDome display [2] are well-known immersive hyperreality or virtual reality displays. Constructing such immersive displays, however, requires a large space in the order of one room, and hence, they have yet to be commercialized for the home or personal use markets.
The head-mounted-type display (HMD) that has been developed and commercialized renders displays compact and personal. HMD, however, does not achieve immersion because of its small viewing angle of less than 30°. In addition, compact mobile projectors with RGB light-emitting diode (LED) light sources have been developed and commercialized [3]. Those devices have advantages, such as a wide color gamut, compactness, and environment-friendly characteristics, but they are not sufficiently efficient for projection television, with high intensity or mobile use under bright illumination.
To create a compact and wide-viewing hyperreality dome projector for personal use, taking advantage of the compactness and wide color gamut of LED light sources, a new-concept hyperreality HMD called 'head dome projector (HDP)' with a curved screen and a compact LED projector was developed [4]. It  80°vertical field of view (FOV). Using the pixel shift technology, not only the FOV was increased to wider than 160°horizontally; the central display resolution was also increased to 1.5 times higher than that of the conventional method [5][6][7].
In this paper, the history of hyperreality HDP technologies is reviewed, and the importance of hyperreality performance for future display technologies is discussed.
A head-up display (HUD) superimposes the reflected display image and background space. One observes a seethrough image through the reflected equipment (combiner) placed in front of the observer. Such displays are used for aircraft and entered practical use for cars in the 1990s [8]. An embarkation model has seen increasing use of late. One advantage of HUD is that the user can glance at the display information with minimal movement of the gaze point. Generally, in the case of HUDs for aircraft, the focus point is set to infinity. For automotive application, however, it is set to 2-3 m, near the front bumper. There are two reasons for this difference. First, when driving a car, one must see the background at various distances, from infinite to close range. Second, it is necessary to reduce the sense of incongruity for a driver while taking a turn at a curve [8]. For many years, there have been requests for the ability to set the display image at any depth position of the driver's gaze point. When binocular parallax is used, generation processing must correspond to the parallax image at an observer's gaze point [9]. This complicates the processing. In this work, the development of the research on the hyperrealistic display was advanced. It was found that when a user faces the non-plane display of a wide visual field, monocular vision affects the reinforcement of the space perception and stereoscopic sense [10]. Based on these results, depth perception enhancement through monocular vision was used for the development of a new-concept HUD.
With the developed HUD system, the windshieldrefracted augmented reality projector (WARP), the display image can be set to any depth position, a difficult task when using the conventional approach. Moreover, monocular HUD is expected to offer the advantage of faster perception speed. In the case of the conventional binocular HUD, when a distant point is viewed, the HUD image becomes blurred. This is double vision caused by binocular parallax, as shown in Figure 1. In the case of monocular HUD, the HUD image can be seen clearly irrespective of the fixed point. The monocular HUD allows the minimum accommodation time and the maximum perception speed. The binocular HUD was observed with both the left eye (broken line) and the right eye (solid line), causing a double image. The monocular HUD, on the other hand, was observed using only one eye (only a solid line in the case of the right eye).  The device consists primarily of a curved screen, a compact projector, and a head-tracking technology. In addition, as the HDP is completely separated from the external world through the use of a helmet-like case including a curved screen, it is uninfluenced by the circumference illumination but increases the feeling of immersion.

HDP concept
The principal features of HDP are as follows: (1) compact and wide-viewing angle (greater than 160°, comparable to the human visual FOV) through the use of a curved screen and a wide-viewing compact projector; (2) motion parallax through the use of a head-tracking technology achieving 360°viewing; and (3) wide color gamut through the use of RGB LED light.
Taking advantage of these features, the target in this work was a personal use and head-mountedtype hyperreality display. The developed HDP performances are compared with those of other immersive   displays in Figure 3. The developed HMD has a curved screen with wide FOV head tracking. It achieves a human-comparable FOV and simulates the human visual system.

Screen shape and size
The HDP concept is based on the authors' assumption that a small dome-shaped screen is the best solution for constructing a personal immersive HMD. To verify such an assumption, experiments related to screen shapes and sizes were performed, as shown in Figure 3(a). There were five subjects with normal visual ability. Figure 3(a) shows the evaluation system with two viewing distances (free distance and screen diameter), three screen shapes (flat, cylinder, and dome-shaped), and three sizes (40, 60, and 100 cm). Figure 4 shows the results of the subjective evaluations. The figure indicates that a dome-shaped screen with a 40 cm diameter at the distance of the screen diameter is the best solution for achieving hyperreality because the hyperreality of a small dome screen with a 40 cm diameter is higher than or comparable to that of the larger dome screens with 60 and 100 cm diameters. A disadvantage, however, is that the eye fatigue is expected to be greater with a small screen. In the case of a dome screen, though, the eye fatigue at a shorter distance (screen diameter) is comparable to that at a longer distance despite the shorter viewing distance. This confirms that the HDP concept implemented with a small (40 cm) dome screen is very useful for HMD-type personal immersive displays.

Vertical and horizontal viewing angles
Immersion and hyperreality are determined primarily by the viewing angle [11]. Therefore, in the second experiment, as shown in Figure 3(b), the required vertical and horizontal viewing angles were evaluated and determined. There were 11 subjects with normal visual ability. The evaluation image was a 10-second flyingforward-motion image, as shown in Figure 3(b). The vertical upper and lower viewing angles were limited to 15-75°, and the horizontal left and right viewing angles were also limited to 15-75°at 15°intervals. Figure 5 indicates that the required vertical viewing angle is different between the upper and lower parts, depending on the human visual characteristics. The required upper viewing angle is greater than 50°, and the lower viewing angle is greater than 70°, under the condition that the evaluation score exceeds level 4.5.    Figure 6 indicates that the required horizontal viewing angle is greater than 150°under the same condition, and that the hyperreality of a dome system with 150°viewing is two points higher than that of an HD system with 30°v iewing.

Intensity, contrast, and color gamut
HDP is completely separated from the external world. Therefore, the viewing illumination condition is nearly the same as that for movies. For a movie, the required maximum intensity is 41-75 cd/m 2 (SMPTE 196M). The film contrast and color gamut are 1000-1500 NTSC, respectively.

HDP prototype
The HDP prototype design is shown in Figure 7. Figure 8 shows an image displayed on the dome-shaped screen.    Table 1 shows the performances of the HDP prototype. In this HDP system, the horizontal viewing angle was changed from 30°to 150°at 30°intervals by enlarging the projected image size, and the hyperreality scores were obtained for 11 subjects. Figure 9 shows the hyperreality performance of the HDP with a 150°horizontal FOV or higher. Table 1 indicates that the obtained performances, with the exception of intensity, meet the specifications. Although the intensity was less than half the specification because of the focus on compactness, it is sufficient for immersive gaming. Figure 10 shows a concept prototype of the HDP. Using a head-tracking technology, 360°m otion images are obtained.

Experiment for the depth perception of the WARP system (monocular HUD system)
An experiment was conducted to investigate the depth perception of WARP, as shown in Figure 11. In this experiment, the distance between the half mirror (combiner) and the subject was changed. The subjects responded with the perceived depth position of the HUD stimulus compared with a movable target, and the target distance was measured with a laser distance meter. There were five subjects with normal visual ability. The monocular stimulus was projected into the dominant eye of each subject. The 'TOSHIBA' logo was used for the HUD stimulus, and a complex composition motion picture was used as a background picture.   Figure 12 indicates the relation of the changed perceived depth distance to the difference between the virtual image position and the distance between the combiner and the observers. If one is 20 cm away from the combiner, the image will be recognized in a deeper direction rather than for a physical distance. The effect becomes greater at more than 1 m, which is equivalent to the distance between the driver and the windshield. Usually, in the case of HMD, the combiner is arranged just in front of the eyes, and the depth perception is not enhanced. This result shows that the WARP system is advantageous for HUD.

Prototype WARP system
A prototype WARP system is shown in Figure 13. The system includes a small DLP projector with an LED light source. The image from the projector is focused at the diffusion-limited screen to prevent binocular observation. The diffusion-limited screen has a lenticular lens panel. The images are reflected at a concave reflector. This reflector corrects the light on one eye of the observer. It was confirmed that this system achieved the monocular viewing condition. Table 2 shows the performances of the WARP prototype system. Generally, the human interocular distance is 6.5 cm. To achieve the monocular observation condition, the developed prototype system limits the viewing area to approximately 6 × 6 cm (the eye box area). This limited viewing area contributes to the effective performance for WARP light efficiency. Typically, up to 5000 cd/m 2 brightness is required for a conventional HUD system [1], necessitating a huge light source power. The developed WARP prototype system, however, used a small light source projector, and it had only a 15 lm optical output. Even with a small light output, a sufficiently high brightness of 16,000 cd/m 2 could be observed.  The WARP system has sufficient potential even for a fine-snow-covered road.

Conclusion
A novel hyperreality display called 'HDP' was developed. HDP is a HMD consisting of a dome-shaped screen with a 40 cm diameter, a mobile projector with an ultrawide projection lens, and LED light sources. The principal feature of HDP is a very wide-viewing angle (160°h orizontally × 120°vertically without head tracking, and 360 × 360°with head tracking). In the objective evaluation and comparison of HDP with a flat-panel display (FPD), HDP achieved 2.5-points higher hyperreality than FPD in the case of a ±5-level evaluation for HD motion.
Also proposed is a novel monocular-projection-type HUD. Monocular depth perception enhancement was confirmed with a liquid crystal display simulator, and a prototype WARP system was developed for the monocular HUD.

Disclosure statement
No potential conflict of interest was reported by the authors.

Notes on contributors
Haruhiko Okumura received his B.Sc., M.Sc., and Ph.D. degrees in electrical engineering from Waseda University in 1981, 1983, and 1995, respectively. He joined Toshiba R&D Center in Kawasaki, Japan, in 1983. He has been engaged in the development of image pickup equipment and video coding technologies for TV telephone and convention equipment. He is now conducting research and development on image processing and driving technologies for flatpanel displays, especially LCDs, and for new display applications, such as AR, VR, and the HUD.
Takashi Sasaki is a senior research scientist at the Toshiba Corporate Research and Development Center. He received his B.S. degree in physics from Yokohama City University at Yokohama, Japan, and his M.S. degree in biosphere from Hiroshima University in Hiroshima, Japan. He had been involved in developing devices and materials for electrical displays. Now, he is working on the development of hyperrealistic displays and AR display systems. His research interests include the human vision and its application to hyperrealistic displays. He is a member of SID, ITE, and JJAP.