Technical note: using ISS videos in Earth observation – implementations for science and education

ABSTRACT A large variety of various passive and active satellite sensors producing panchromatic, multispectral or hyperspectral images of the Earth’s surface are currently in space and are used in different scientific fields such as earth science (e.g. geography, hydrology, geology, oceanography and glaciology), disaster management, as well as for military, commercial and economic purposes. In contrast, video files are a rather exotic data format in the field of Earth observation. Occasionally, video cameras are used in airborne remote sensing, but only recently video Earth observation from space has been established. This paper aims at the integration of video data in the scientific workflow, revealing advantages and disadvantages of moving images. Being the only available source for continuous video Earth observation data, the NASA experiment High Definition Earth Viewing on-board the International Space Station is the basis for our evaluation of the usability of video data. Following the rather coarse resolution of these data, we exemplify some potential fields of application in science and education. We show a basic workflow how to produce 3D models and stereoscopic videos of the Earth’s surface. As a pre-study for videos with better spatial as well as radiometric resolution, the delivered products serve not only scientific purposes, but are integrated in school lessons to evoke the pupils’ fascination for earth sciences and space.


Videos in Earth observation
Videos are not a common data format for Earth observation. Although sometimes used in airborne remote sensing applications, scientists give preference to imaging sensors depicting the object of investigation in a stationary way. This paper gives an overview of the workflows associated with the implementation of ISS videos in Earth observation and its potential fields of application. The ISS itself is an Earth observation platform exhibiting several unique characteristics which distinguishes it from satellite or air borne remote sensing: it is manned, circles sun-asynchronously, in a low Earth orbit, and faces varying dynamics in terms of the roll, pitch and yaw axes (Stefanov & Evans, 2014). This influences not only the lighting and recording but also the storage and transmission conditions. The high-definition videos examined in this paper are part of a unique experiment on-board the ESA Columbus module of the ISS: The High Definition Earth Viewing (HDEV) experiment.

Methods and data
The project "Columbus Eyelive videos from the ISS in school lessons" funded by the German Aerospace Centre has access to video data collected by NASA's HDEV experiment (Muri, Runco, Fontanot, & Getteau, 2017). Columbus Eye acts as an exclusive European partner of NASA's HDEV experiment and aims to integrate Earth observation in schools. The main aim of the HDEV experiment is to quantify the effect of cosmic ray on the HD video quality and the protection efficiency of the HDEV housing against harsh thermal conditions in space. Consequently, the experiment lays the foundation for a new era of Earth observation techniques for space mission to Mars and beyond (Runco, 2015). To compare different camera types, four cameras are attached to the Columbus module of the ISS: Toshiba IK-HR1s, Hitachi GV-HD30, Panasonic AGHMC150 and Sony FCB-EH4300. As seen in Figure 1, the latter are installed parallel, focussing on the aft view. The forward view is covered by Hitachi, whereas the Toshiba points nadir (Stefanov, Evans, & Dasgupta, 2011 (Runco 2015, Rienow et al., 2015. In order to manage large amount of data, the original videos are divided into hourly segments and subsequently stored in MPEG-4 file format. This archive is a unique opportunity to work with video files for Earth observation, as Columbus Eye is allowed to command the camera cycles and thus directly influences the videos filed. The automatic cycle can be interrupted, allowing the user to track objects on the ground by making use of the different camera positions relative to the Earth's surface. Furthermore, the cycle has to be adjusted to a configuration where the front and aft camera lenses are not directly facing the sun. That case occurs as soon as the beta angle of the ISS is below 32° (Runco, 2015). The initial and individual camera calibration and view angle cannot be changed; nevertheless, the turning of the ISS leads to different angles and changing ground resolution. For nadir position, the average ground sampling distance (GSD) is 500 m. The image size achieved with the 1/3 CMOS sensor with 2.1 megapixels is 1280 × 720 pixel (Stefanov & Evans, 2014). This rather coarse GSD leads to difficulties detecting land cover or land use without image correction. Therefore, image pre-processing in terms of the removal of Rayleigh and Mie scattering as well as colour correction is vital. The image processing is quite difficult because the space station passes between 52°north and 52°south latitude and has an inclined equatorial orbit that is not sun-synchronous; moreover, 60 images are received every second. Therefore, illumination conditions change rapidly within several seconds and make an adaptive image correction necessary. All enhanced videos are produced with MATLAB © algorithms (e.g. imadjust, imsharpen and stretchlim functions, see MATLAB documentation (https://de.mathworks.com/help/ matlab)). The temporal resolution is flexible, ranging from 180 min to 3 days (loss of signal and nighttime included). Unfortunately, allocation is impeded by the lack of reliable time stamps; until now, no automatized process has been established to overcome this issue. Selected videos are geo-tagged manually and then published to a web GIS on the Internet portal columbuseye.uni-bonn.de, thus made easily accessible to different audiences. To evaluate the capability of HDEV data to produce a 3D model and stereoscopic images, we selected a video scene from 21 October 2016 showing the Black Sea and the Caucasus region. For comparison, two astronauts images from the ISS showing the Mojave Desert (ISS048-E-68432 and ISS048-E-68433, 30 October 2016, 20:21:00 GMT) are used.

Using ISS videos for stereoscopic images and 3D models
Even though the ISS is travelling with a speed of approximately 7600 m/s, the frame rate of the installed video cameras (60 frames per second) is fast enough to produce a large overlap of nearly 99.99% between the images, suitable for the creation of stereoscopic images and videos.
Anaglyph 3D images are the easiest way to derive stereoscopic videos from the HDEV experiment. Those images contain two differently filtered images, one for each eye. If the image is viewed through a filter, the so-called "colour-coded" or "anaglyph" glasses, each of the two images reaches the eye it is intended for, producing a stereoscopic impression. Consequently, an anaglyph image is a combination of two images showing the same object from different angles. Having rectified the images, we selected the green and blue channels from one image and the red channel from the other. The key question remains: How to select the left and right image from an HDEV video? Especially, the determination of the optimal angle of the HDEV video is quite challenging. Our results indicate that a shift of 40 frames in nadir view is leading to the best results (base to height ratio 15 km/400 km), i.e. the first frame is rectified and combined with frame 41. This procedure is performed for all following frames. The anaglyph images are then merged to a single video file. All calculations, including image enhancement and the projective transformations for the rectification of the images, are carried out using MATALB © . The procedure was tested with different HDEV highlight scenes. The best results were obtained for the Black Sea/ Caucasus highlight. Even though the videos were pre-processed to enhance image quality, 3D effects can only be observed for areas with strong variations in cloud height or high mountain ranges. This is perhaps due to the coarse resolution of the HDEV videos. Furthermore due to the earth curvature, the ISS orbit and varying camera angles, it is difficult to match the images. Therefore, the procedure was tested based on high-resolution images of the Mojave Desert taken by ISS astronauts. The resulting anaglyph image is shown in Figure 2. Although the rectification of the images is more difficult because the images were taken without a fixed tripod, clear 3D effects are visible. Therefore, it can be concluded that the resolution of the HDEV videos is too coarse to produce proper 3D videos. In contrast, good results can be obtained for a 3D model computed with Agisoft PhotoScan © of the Black Sea/Caucasus highlight, displayed in Figure 3. Every twentieth frame was selected and in total only 150 images are used for the 3D model.
Photogrammetry can be used to make accurate and realistically image textured models of buildings or landscapes. The position of the camera is used to

Educational valorization of 3D models and stereoscopic images
Since the ISS video archive is tied to the "Columbus Eye" project and its didactical principle of moderate constructivism, the outcome of 3D models and stereoscopy can be valorized for the use in school lessons. In order to foster the competences of the pupils in terms of self-organization, autonomous learning and spatial orientation, the project aims at the integration of remote sensing products and workflows in schools (Voß, Goetzke, Hodam, & Rienow, 2011). Several working sheets for students and also interactive learning tools covering different topics are available on the Columbus Eye portal (columbuseye.uni-bonn.de). For registered users, it is possible to do an online evaluation, and additionally, we receive constant feedback from our teacher training courses and school visits.
The workflow described in the previous section can be conducted by the pupils themselves if they are instructed, e.g. using a work sheet addressing the topic of 3D in simplified terms. The work sheet currently under development has to explain the physical background of stereoscopy, introducing concepts of absorption, colour and polarizing filters, complementary colours as well as stereoscopic vision, and giving information on all relevant techniques and methods to acquire, e.g. anaglyph or polarized images. This first section of the work sheet is followed by a comparison of 3D images which were derived from different sensors -ISS imagery and satellite data. The acquired knowledge is tested in a quiz to prepare the pupils for the task of the 3D workflow application on their own. An HTML5based e-learning section will help them with the conduction of this workflow.
Enabling the pupils to work individually on the videos at hand, not simply watching or locating the scenes, but experiencing a new way of understanding the Earth's surface as seen from a satellite's perspective, can lead to an increased interest in space and earth sciences alike (Goetzke, Hodam, Rienow, & Voß, 2013). While selecting the most suitable images for stereoscopic views, the pupils have hands-on experience which is not only evoking fascination for the videos at hand but also broadening the knowledge in the fields of mathematics (X-Y-positioning), physics (optics) and biology (how does our vision work?) (Ortwein et al., 2016). Thus, it can be easily integrated in the school curricula which is the precondition for the use of this techniques in an educational environment. Integration into a 3D model of the Earth based on the ISS videos seems a possible task for the future, helping the pupils to understand underlying physical principles as well as interaction within coupled human-environment systems.

Videos in Earth observation in the future?
Earth observation from the ISS with hands-on or automated equipment such as the HDEV experiment is an important addition to existing satellite-based earth-observing satellites. Besides HDEV, several other earth-observing payloads such as the Agricultural Camera or HREP-HICO (Hyperspectral Imager for the Coastal Ocean) are installed on the ISS. In the near future, additional sensors, e.g. DESIS (DLR Earth Sensing Imaging Spectrometer), will extent the earth-observing capacities of the ISS.
Based on the HDEV experiment, several learning materials have been developed within the Columbus Eye project in order to integrate Earth observation in schools sustainably.
This paper shows how videos and images from the ISS can be used to develop 3D models and videos. Based on this material and worksheet, pupils can discover the physical background of stereoscopy. Furthermore, we have shown that images and videos from the ISSeven though their spatial resolution is lowhave great potential for professional image processing in the field of stereoscopy. This prestudy gives evidence that the use of videos with higher radiometric and spatial resolution will extend the capabilities of existing unmanned orbital sensor systems (e.g. Poli & Toutin, 2012;Rupnik, Pierrot Deseilligny, Delorme, & Klinger, 2016). Especially SkySat (d 'Angelo, Máttyus, & Reinartz, 2015) has already shown that sub-meter resolution imagery and high-definition video are valuable for Earth observation. Regarding the future, it is likely that more high-resolution video sensors in satellites and on the ISS will extend the existing sensor systems and demonstrate their usefulness for observation of the earth, space and beyond.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This work was supported by the German Aerospace Centre DLR [50JR1307];