A methodology for acquisition and processing of thermal data acquired by UAVs: a test about subfluvial springs’ investigations

ABSTRACT An accurate description of watercourses helps to easily understand their behaviour both for routine hydrological investigations and in disasters cases. Knowing the position and the dimension of the supply sources of the river is crucial, since they have a direct influence on its behaviour; however, they are not often easily identified, and almost always completely unknown. Aerial thermal data can be critical in order to rapidly reveal and map the river supply source points for extensive areas. Using unmanned aerial vehicles (UAVs) to collect very easily and quickly high-resolution information is thus very interesting. UAVs can house many sensors for performing investigations in different bands of the spectrum, including thermal data. This work explains how UAVs can be used to collect this kind of data (through RGB and thermal devices) and presents a strategy to easily integrate the information from these different sensors. Integration procedures, also applicable in emergency situations, were developed. These methods do not require points on the ground and can be performed using commercial devices.


Introduction
The significant increase over the last few decades in hydrological disasters (Guha-Sapir 2011), also due to climate changes, has necessitated the wide-scale monitoring of watercourses (Dabove et al. 2015). The knowledge of water springs (Vigna 2014) is fundamental in preventing disasters, evaluating risks, and managing post-disaster activities. This includes the dynamic knowledge of physical (e.g. flow rates) and chemical parameters of the rivers and streams, plus the characteristics of subterranean water resources and the entire hydric network. To use these data effectively to prevent risks of disaster, georeferencing activities and geomatics skills have become essential for both territorialscale interpretation and modelling, and local, detailed analyses. The support for decision-making provided by the produced analysis and products is of great worth, since geographical issues (terrain topography, soil nature, land use and coverage, etc.) are a crucial factor.
An important issue in identifying the hydric network is to detect subfluvial springs, that is, the supply source points, in the stream or a river, which are located in the riverbed, and are not therefore visible with an initial, superficial analysis.
Such springs remain very often unknown, since they are not visible from the surface, being the point from which the water comes out, below the water surface of the main stream. For this reason, their position could only be supposed. The problem could be overcome by using a thermal sensor to measure their different temperatures with respect to the stream water, and, consequently, identify and investigate them (Mallast et al. 2013).
The usual procedures consist in manually measuring the temperature by a high-precision thermometer. This is time consuming for the operator, who has both to know exactly at what points it is appropriate to take measurements (deducing this from other evidence), and to take regular measurements, dense enough to detect the points where the water temperature is different. Furthermore, the accessibility for on-the-ground operators can be limited: the ground near the river may be unsuitable or inaccessible as a survey station (with trees or rocks on the banks), and the areas in the middle of the riverbed can be obstructed by the water.
Although the quality of the information acquired with traditional methods can be high, considering the precision of the measured data, they are often not sufficiently accurate in the georeferencing in order to build a useful temperature map to identify the exact position of the supply source points. Furthermore, the data acquired by traditional methods (thermometers) are punctual compared to those acquired by a thermal sensor. In fact, this sensor allows to produce a raster image that maps the temperature of the objects, thus offering continuous coverage.
However, in terms of using a thermal device from a terrestrial point of view, some problems can also occur. Besides accessibility problems, the angle of view may reduce the possibility to exactly detect the position of the points with different temperatures, which are more apparent from a nadir point of view. It would be, therefore, more effective to acquire the data from an aerial point of view. An aerial acquisition can consider wide areas simultaneously, enabling the whole context to be studied, irrespective of accessibility issues.
Unmanned aerial vehicles (UAVs) can record useful data for environmental applications from aerial points of view, with a sufficient resolution, speed, and low-cost instruments, for performing specialist analyses (e.g. Grenzd€ orffer et al. 2008;Aicardi et al. 2014;Nex & Remondino 2014).
The goal of this research was to develop a methodology for quickly identifying the subfluvial springs of a stream. These represent the lowest points of the hydrological structure, which is the level of the valley floor, and are the points of outcoming of the aquifer water. The obtained results could be used as base for making further analysis on the behaviour of the aquifer and, consequently, of the stream. This can help in very accurately predicting the possible risks, which could be generated by different meteorological or hydrogeological conditions. The proposed methodology considers the simultaneous acquisition of images from a thermal sensor and an action-cam installed on a UAV (Bendig et al. 2012;Amici et al. 2013). The resolution and the scale of the data enable even a few degrees of temperature differences in a few centimetres of space distance to be measured by means of a portable and fast system.
The acquired data were used to generate georeferenced and measurable orthophotos. A raster temperature map was used to identify the position and size of the supply sources and subfluvial springs in a stream. Various automatic procedures were also developed to correctly georeference the acquired images without measuring the ground control points, in order to be effective for inaccessible sites or emergency surveys.
The whole workflow was studied and calibrated in laboratory and local test sites, and was then tested on a case study: the Vermenagna stream in an alpine area in Italy (Piedmont). It is a tributary of the Gesso stream, monitored during the ALCOTRA ALIRHYS European project (www.polito.it/ alirhys/). The Vermenagna is characterized by multiple springs that feed it. This makes it difficult to be monitored using individual instruments, and an aerial survey provides a more extensive description of the area and in-depth study of the springs.

The methodology
For thermal investigations on a river, point data are not always adequate. The aim of the proposed approach is to acquire all the necessary information during a flight with UAV and then perform hydrological investigations in laboratory.
From a photogrammetric point of view, the use of UAV for three-dimensional (3D) modelling with RGB data is well established (Eisenbeiss 2009). The new topic today regards sensors and data integration in these systems. In the proposed study, a thermal camera was housed on board the UAV and the main issues were the on-board installation and data recording and processing.
The proposed methodology uses both RGB and thermal data simultaneously acquired for a complete description of the case study. RGB images cannot help to identify the springs, but they are essential in understanding the thermal data and to georeference the information (if no GPS data are available for the thermal sensor).
Following an integrative approach, correctly acquiring the data entails assessing the best conditions to perform the flight. In fact, for good thermal inspections, the sky should be clear, since clouds reduce solar radiation and produce interference in the form of reflections. Calm wind conditions are also preferable, since any airflow will lead to convective cooling, thus reducing the thermal gradient. This is why thermal inspections are best made in the early morning.
Best practice confirms summer and winter to be the most suitable periods for such investigations, since the water of the aquifer has a quite constant temperature throughout the year, and the difference with the temperature of the surface water can be better identified when it has a major gradient.
Another issue is the trees that can be an obstacle to visibility. Thus it may be more effective to carry out the acquisitions with bare trees, although this could give rise to other problems (for example, in the automatic photogrammetric processing). On the other hand, for a good photogrammetric acquisition, it is also important to evaluate the light conditions.
An initial application of the proposed approach based on the use of a UAV for thermal and RGB data collection is investigated in this paper.

Description of the used UAV and device configuration
A multi-rotor UAV (Mikrokopter Hexakopter) was used. This kind of drone permits vertical takeoff and landing, which are often needed in mountain sites, where there are few open wide spaces without obstacles. It is also more reliable to perform an irregular, linear, and low flight.
For the equipment, it was necessary to find a thermal sensor suitable for aerial acquisition (highperforming, but light enough). Thus, we acquired a FLIR A65 thermal camera (the cost is about €15,000, specifications are summarized in Table 1) to identify the springs along a section of the riverbed. Since the sensor was not autonomous in the acquisition and storage of the data, a dedicated mini-PC was assembled and installed on board. This records single images or videos and can transmit the data in real time to the ground station. In order to correctly manage the system, a dedicated configuration file was created with Drone Easy Fly software and stored in the mini-PC ( Figure 1). This contains the parameters of the specific acquisitions: temperature range, emissivity, environment temperature, humidity, and distance from the object. These parameters are essential to correctly estimate and understand the recorded thermal data.
Since at the time of the survey, we did not have a GPS antenna to geotag the thermal images, which is a practical and faster method to georeference the data. We developed an alternative technique to obtain the same result: we installed an RGB camera on board with an integrated GNSS receiver. The two types of data were then used to contextualize the thermal images and georeference them. To ensure a small payload for the UAV, a Sony action-cam (HDR AS100V) was used to record an RGB video joining a GPS track. To install the thermal camera and the action-cam on board, a carbon fibre support was designed ( Figure 2).

UAV data collection
The data acquisition was carried out in December 2014, since, theoretically, the temperature difference had to be enough for the subfluvial flows to be visible in contrast with the dry objects and the cooler stream water.
The flight took about 10 minutes to cover 300 m of the river. During the acquisition, thermal images (about 2700 images per flight) and a video with the Sony action-cam (1920 £ 1080 of resolution) were recorded.
The images acquired by the FLIR camera are generated in a proprietary format (.FFF) which can be read and analysed using Drone Easy Fly, FLIR Tools, and IRT Analyzer. We opted for the IRT Analyzer, which performs thermal measurements and manages the data in order to generate profiles and histograms of the area. The images can also be exported to more common formats managed by GIS or raster processing software (e.g. TIFF format). This enables the images to be exchanged and the information to be used in more complex systems (for example, in a GIS) or for further analyses by algorithms available in other application tools.
In order to use the thermal camera as a measuring instrument, the sensor had been geometrically calibrated beforehand (Conte S. 2015).
The data were processed at three levels of detail in order to analyse the individual RGB and thermal data, and then to integrate them, as described below: Thermal image analysis: extraction of significant points and profiles through the IRT Analyzer software. This step was performed considering only the thermal data. 3D model generation from the data acquired by the Sony action-cam: having a full georeferenced reconstruction of the area obtainable using a photogrammetric approach is very useful. Development of thermal and visible integration procedures to overlap and integrate the two available information.

Investigations on the single thermal frame
The first important step is to correct the temperature offset of the camera. It is essential to record the real external temperature during the survey using a high-precision thermometer in order to calibrate the thermal frames. Other instruments, such as equipment installed to monitor the stream temperature, could be equally effective. After the offset application, the analyses can be performed in the IRT Analyzer (Figure 3). In particular, it is possible to perform point inspections; analyse profiles and areas of interest; define a temperature range on which to base the analysis; start from the identified range, to automatically search for hot and cold points.
As shown in Figure 3, this tool was used to extract the spring temperatures and to analyse some temperature sections along the river.

3D model generation
To correctly read, interpret, and georeference the thermal data, a photogrammetric 3D model was generated. The process presented two main difficulties: the use of unsuitable image data (large distortions) (Figure 4) and the need to directly orient images using GPS data.
The RGB data were acquired by the Sony action-cam in video mode acquisition (mp4, 1920 £ 1080 video resolution). The frames were extracted from the video using a dedicated MATLAB script, which also associates the camera's position and attitude with each frame.
For the Vernante survey, no markers were located and measured on the ground. Thus, to georeference the images, the GPS track of the Sony action-cam (.kml file) was used, and also the .gpx data recorded by the UAV itself (Dabove & Manzino 2014). To visualize the differences between the two types of data, both tracks were uploaded on Google Earth, as shown in Figure 5, where the red track  is the UAV data and the blue track is the Sony recording. The track acquired by the action-cam for most of the flight is not in agreement with the track recorded by the UAV. This was probably due to the type of low-cost GPS receiver installed on the action-cam.
Thus, the track acquired by the UAV was considered as the reference data for the georeferencing procedure. The precision of these positions is about a few decimetres; however, this is sufficient to identify the springs. In addition, the use of the .gpx data can also be useful since it contains other information, such as the flight height and attitude, acquired by the internal sensors.
The characteristics of the extracted frames present different problems for being used in a typical photogrammetric processing. The quality of the lens is poorer than those usually employed for photogrammetric or structure-from-motion acquisition techniques. Action-cams were primarily designed for use by amateurs, thus the geometry and stability of the lenses are not a priority. The image distortions are thus similar to a fish-eye lens, and the borders of the images are very distorted.
The stability of the acquisition is also probably not optimal since a lot of frames were blurred, which has led to some difficulties in processing them with some photogrammetric software. Furthermore, being extracted from a video, they present other two anomalies that can affect the processing: The possible rolling shutter as a result of the way the CMOS sensor is read. Instead of capturing the entire frame at once, information is read from each row of the frame, one after the other, top to bottom. The whole process takes up to 1/30 of a second with most cameras. This creates a wobbling effect in the recorded video in cases where the camera moves a lot (Girod et al., 2014). The absence of an EXIF file, since the alignment phase of images implemented in some photogrammetric software needs basic parameters to initialize the iterations and build the model. Another difficulty is due to the specific surveyed object. The water is not static, thus it changes from one photogram to another, making it difficult to correctly identify tie-points on its surface. Moreover, the water is translucent in some points, which can also confuse the algorithms in finding common points through images.
Lastly, the season in which the acquisitions were carried out should be suitable for measuring temperature discrepancies among elements, and for seeing the river, however, the lack of leaves on trees represents a further difficulty. Branches have a very complex geometry and thus produce shadows that often disturb how the 3D model is generated. The branches also create noise in the point cloud and generate confusion due to their complexity and thinness. The processing software was chosen due to its low cost and user-friendliness since the software also needs to be available for users with different levels of expertise, also in emergency cases (Boccardo et al. 2015). We tested two free and Open-Source Software (OSS) applications -VisualSFM and MicMacas well as a proprietary software, Agisoft PhotoScan. They all exploit Structure-from-Motion algorithms to compute orientation parameters of a set of images and reconstruct the 3D model of the represented object. First of all, we tested VisualSFM (Wu 2011) which integrates SIFT (Scale Invariant Feature Transform) algorithms, the Multicore Bundle Adjustment and the PVMS/CMVS tool (Furukawa et al. 2010;Teeravech 2013). Using only the frames as input, the implemented algorithms correctly aligned the images (Snavely et al. 2006). However, the 3D dense point cloud reconstruction using the SURE solution (Rothermel et al. 2012) was not positive. In fact, it took a lot of the time for the processing and the produced point cloud was completely lacking near the shore of the river, because of the branches, and in the central part, because of the water.
The other OSS we tested was MicMac (Deseilligny & Clery 2011). This needs the EXIF file for an initial value of the focal length, and to initialize the process. Usually, these parameters are not mandatory, but are required with this type of camera to start with approximate values. The preliminary step was thus to calibrate the action-cam using three different software applications (Leica Photogrammetric Suite, Agisoft Lenses, Mathworks MATLAB). These parameters had to be included in the EXIF files of the images, thus a further step was performed through an ExifTool MATLABprogrammed script. This included the sensor dimension, focal length, and name of the camera.
Our strategy enabled us to perform the bundle block adjustment with the approach typically used for fisheye lens; however, the results were not satisfying: the 3D model was incomplete, and the orthophoto was not successfully generated, probably due to the high percentage of trees and shadows in the images.
Finally, Agisoft Photoscan was used, which also requires the EXIF files associated with the images. For the processing, a low-accuracy alignment was performed, followed by a highly accurate cloud densification. As shown in Figure 6, the orthophoto was not complete and in some parts was Figure 6. Final orthophoto extracted from the PhotoScan software. heavily irregular; however, it was sufficient to contextualize the thermal frames. Thus, for the subsequent analyses, we used the PhotoScan results.

Thermal and visible integration
In a thermal analysis, the main problem is interpreting the viewing information. Overlapping the two thermal and visible layers can thus provide a very fast solution for interpreting the acquired temperatures.
In the specific case of the Vermenagna stream, we tested two different procedures to overlap this information, based on ESRI ArcGIS and Matlab.

Coarse registration with GIS georeferencing tools
The first methodology is less accurate, since it is based on the ESRI ArcMap manual solution. The RGB orthophoto and the thermal images were uploaded into the ArcMap workspace and the overlap was carried out through the raster georeferencing tool.
This was performed by manually collimating some control points in the images (3 is the minimum number of points for the registration, but in practice, 8-10 points are needed) in order to properly align the two data-sets. The alignment was carried out by a roto-translation with scale variation. As the test site was a natural environment, few sharp and defined points were recognizable (and very often they were asymmetrically distributed on the image). This can further reduce the accuracy in the collimation and, consequently, in the transformation.
An example of the result achieved is shown in Figure 7. The overlap of the two layers enables the thermal information to be read starting from the visible orthophoto. For this specific application, it was possible to generate a text file in which the coordinates and the thermal value were stored for each point selected.

Fine registration with MATLAB
A more refined approach was implemented in MATLAB.
The methodology involves the use of two images, one fixed (visible) and one moving (thermal), and performs roto-translation algorithms to refine the matching between the two images. The procedure ( Figure 8) starts with an approximate registration and then refines it. The first registration between the two data can be performed manually by the introduction of common points. This is possible in MATLAB, thanks to the control point selection tool (cpselect) that shows a window with the two images in which it is possible to manually select and collimate the points and export them in a .mat format. The use of these points allows to perform a first roto-translation of the thermal images.
Then, to refine the registration we introduce the imregister command, which is an intensitybased image registration approach that internally build a multi-resolution pyramid in memory and solve an optimization problem on each level of the pyramid. It has already implemented four different algorithms: Translation: rigid movement in both X and Y directions. Rigid: rigid transformation composed by rotations and translations. Similarity: that is a roto-translation with scale variation. Similar: roto-translation with scale variation and cutting of the area of interest.
It is possible to apply all the needed transformation and, in most of our images, the first algorithm (rigid movement in both X and Y directions) has been used. Figure 9 shows an example of the final registration between a thermal image and the RGB orthophoto. It was thus possible to read the coordinates and thermal value for each spring or point of interest.
It is important to remember that the purpose of the paper is not a metrical assessment of the thermal acquisition, but a fast and easy tool for thermal investigation that can be shared with nongeomatics users.

Discussion
The data regarding the whole stretch of river were acquired in a 10-minute flight, which was then be analysed in laboratory. The proposed methodology describes the thermal behaviour of the river in a replicable, cheaper, and less time-consuming way than traditional terrestrial surveys.
The resolution of the final data was sufficient for hydrological analyses and the grafting of the new springs was easily identified as a thermal gradient in order to understand the derivation of the wellsprings group. This is the point where a different aquifer put its water in the stream, carrying new contributions, in terms of flow rate and further physical parameters, to be evaluated for hydrogeological management and risk prevention. Furthermore, also other chemical elements are supplied, and are therefore to be evaluated for the use of the water by the local water network.
Together with this information, the georeferenced 3D model obtained from the photogrammetric processing was used to locate the position of the springs and to validate the connecting hypothesis along the stream. As reported, the use of visible images from an action-cam is not the best way to georeference the data. It would be better to use a classic camera with a calibrated lens. In addition, the calibration of the thermal camera must be considered if the purpose is a metrical analysis.
This integrated approach highlights the connection between the identified springs and the hydrological network, which can then be investigated through an in-depth hydrological analysis in order to evaluate the possibility of installing lift stations along the river.
Another issue is the choice of the survey period. Although the survey was carried out in December, following the current practice for thermal surveys, the period when the case-study flights were carried out was not the best choice for having a sufficient temperature gradient, despite being in late winter, when extreme temperatures are the norm and, theoretically, most suitable for our purposes. In fact, few differences were visible in the acquired images between the different water sources (possibly due to the recent climate anomalies). However, the used instruments and the developed methodology permit to rapidly measure even few degrees of temperature gradients (the thermal sensor has a precision of one-tenth of a degree); this permits to easily identify the areas in which the water having a different temperature is input in the main stream (as visible in the example in Figure 3).

Conclusion
The survey of the Vermenagna stream was carried out in order to identify the springs and analyse their temperatures. However, these investigations can also be very useful to monitor the springs' behaviour along defined sections or the water temperature in relation to new springs. An aerial survey through a UAV provides a complete description of a part of the river both from visible and thermal points of view in a very fast (10-minute flight to cover 300 m of river) and repeatable way. In fact, this low-cost acquisition system enables to repeat the survey in different times of the year and to perform multi-temporal analyses. The availability of GPS information also would obviously provide faster and better georeferenced data. However, the performed analysis shows that low-cost sensors embedded in action-cam are not very reliable. A possible solution can be the use of external sensors (GPS or IMU that can be mounted on UAVs) or internal sensors included in UAVs (sensors embedded in the Flight and Navigation control). The main issues that must be taken into account are related to the data acquisition, since it is fundamental to make flights when the temperature gradient is good and the vegetation not covers the river (both for the best photogrammetric process and a good thermal result).
About the choice of the survey period, December is suitable for the survey of the Vermenagna stream, since the absence of leafs on trees permits a better visibility of the ground as a base for georeferencing the thermal images. Other periods, such as summer, can be more suitable for the thermal survey, but the presence of leafs would obstruct the stream view from the UAV. This could be solved by integrating the data by means of other methods (including, for example, the manual measurement or the acquisition of some terrestrial thermal images, caring the suitability of the point of view).
On the other hand, this highlights the increasing need for a methodology focused on easy replicability without high costs and which is not time consuming. Although the instruments may not seem to be particularly low cost, they can be reused a number of times and provide results rapidly with high accuracy.