Structure from motion (SfM) processing of UAV images and combination with terrestrial laser scanning, applied for a 3D-documentation in a hazardous situation

ABSTRACT The integration of Terrestrial Laser Scanning (TLS) and Structure from Motion and MultiView Stereo techniques allows to obtain comprehensive models of complex objects by using each technique in contexts presenting the optimal operating conditions, as widely reported in bibliographic references. A different situation occurs for emergency surveys. In this case, time and security act as constraining factors, requiring the use of these techniques also in the most unfavourable conditions. In the case of photogrammetry, these include areas where the object surfaces are not perpendicular to the camera axis, and in the case of TLS, they include areas where laser beams are almost tangent to the surveyed object surfaces. These situations are anyway necessary for safely carrying out these surveys in the minimum possible time and cost. Although this kind of survey results locally in lower precision levels than those obtainable by these techniques in ideal conditions, it entails the possibility of obtaining complete models, e.g. including vertical external walls in inaccessible buildings, with controlled precision.


Introduction
Metric surveys in areas presenting safety issues require not only planning and execution according to classic standards, i.e. the choice of technique/methodology and its correct deployment based on accuracy requirements, but also to detect and apply different surveying techniques based on the ability of each and every technique to fill in for any operating shortcomings of the others, rather than their inherent features (Koska and K remen 2013).
Operating in such conditions challenges surveyors with seeking methodologies for collection, investigation, processing and integration of non-standard data.
Laser scanning is a well-established methodology, backed by many decades of scientific research and applications in several fields. This allows to provide a given accuracy for survey planning purposes based on the features of the instruments and of the survey object, and to foresee any significant disadvantage of the methodology depending on the peculiar application (Caroti and Piemonte 2010;. The recent photogrammetry methodology, based on Structure from Motion (SfM) and Multi View Stereo (MVS) incorporates the features of Computer Vision into classical 3D photogrammetry, generating high density coloured point clouds from high resolution images (Caroti et al. 2015a;Robleda et al. 2016). This is possible due to the ability to handle many images at once, which entails, on one hand, high numbers of 3D coordinates, and on the other, very high requirements in terms of computing power.
Performance of this technique depends on the image quality and collection mode, as well as lighting conditions and the presence of sharp edges, which could cause some problems in feature detection due to a poor signal-to-noise ratio (Remondino et al. 2014;Dandois et al. 2015;Tannant 2015;Bevilacqua et al. 2016).
Its applications using UAVs (Unmanned Airborne Vehicles) fitted with remote sensing instruments as vectors offer several opportunities in the presence of safety issues (Julio Miranda and Delgado Granados 2003;Bendea et al. 2008;Ion et al. 2008;Giordan et al. 2015).
In case of surveys in built areas, correct integration of these two techniques provides deploying UAV-borne photogrammetry with horizontal flights for data collection of objects lying on orthogonal planes relative to the shooting axis. Surfaces mainly developing on different planes require either specifically planned photogrammetric flights or ground-based image shooting (Xu et al. 2014;Balletti et al. 2015;Caroti et al. 2015b).
Terrestrial laser scanning (TLS) integrates UAV-borne photogrammetry surveys carried out with horizontal flight by providing point clouds documenting geometries lying mainly on vertical planes (Achille et al. 2015). The present work contributes in this field.
In emergency conditions as outlined for the present case study, logistics and safety issues may sometimes require application of these methodologies beyond best accuracy boundaries in order to obtain a geometrical model as comprehensive as possible of the scene, which will provide the basis for subsequent interventions (Mart ınez-Espejo Zaragoza et al. 2015).
Such conditions require the use of areas surveyed with suboptimal reliability by each of the methodologies. As an example, it can happen that some sub-horizontal or gently sloping areas, such as roofs, are surveyed only via TLS, i.e. with a very small incidence angle of the laser beam or, vice versa, that the vertical walls are surveyed only by UAV-borne photogrammetry with nadiral shooting axis.
In the present case study, these areas, which represent criticalities for either one of the methodologies, have been surveyed via both TLS and UAV-borne photogrammetry, with horizontal flights and nadiral shooting axis.
In order to evaluate the ability of mutual integration of TLS and UAV-borne photogrammetry in these critical areas, portions of a 3D model surveyed with both methodologies will be compared, assuming one point cloud as a reference and measuring the shifts of the other. The modes of comparison are detailed as follows.

Point clouds comparison algorithms
The most common methodologies (Lane et al. 2003) used for 3D point clouds comparisons are: (1) Digital Elevation Model (DEM) of Difference (DoD) (2) Direct Cloud-to-Cloud (C2C) (3) Cloud-to-Mesh distance or Cloud-to-Model distance (C2M) (4) Multiscale Model to Model Cloud Comparison (M3C2) (1) DEM of Difference (DoD). It is most commonly used for objects close to horizontal planes. Each of the point clouds is first framed in a regular grid (DEM); subsequently, a DEM of the differences is produced, by a pixel-per-pixel comparison of the DEMs previously created. This comparison methodology does not perform well with objects mainly developing along vertical planes or featuring highly sloping surfaces.
(2) Direct Cloud-to-Cloud (C2C). This is a direct 3D comparison of the point clouds, without any need to create regular grids or meshes of the original data, or to compute the normals to the surfaces. For each point in the checked cloud, this technique seeks the nearest point in the reference cloud, defining the distance based on different performing algorithms (Tsakiri and Anagnostopoulos 2015): Nearest neighbour Nearest neighbour with local modelling Normal shooting Iterative Closest Point (ICP) The simplest approach to cloud comparison is the nearest neighbour search, which computes the Euclidean distance between the closest points. This calculation does not provide hypotheses, but its results can be unreliable due to (image) roughness and different point density between clouds.
The nearest neighbour with local modelling algorithm is a variant of the above, which calculates the point distances from a local surface, obtained by a set of points in the proximity. This methodology is less sensitive to point density differences, but can sometimes yield unexpected results, due to the limited amount of points on which the modelling is based (corners and close curvature surfaces). Modelling can use several mathematical surfaces.
The normal shooting algorithm calculates the Euclidean distance between two points, where the point in the reference cloud has the smallest distance from the normal vector to the point on the compared cloud.
The ICP algorithm searches for the smallest distance between clouds by splitting the global cloud into smaller ones. It is mostly used for point cloud registration. It is less sensitive to local noise, although it may not be fit for shift evaluation between clouds representing very wide areas.
(3) Cloud-to-mesh distance or cloud-to-model distance (C2M). It creates mesh or triangulated models of the reference point cloud, which are used to measure the orthogonal distances for each point in the compared cloud (Cignoni and Rocchini 1998, see also Monserrat andCrosetto 2008 andOlsen et al. 2010 for recent reviews). This procedure is most suited with sub-planar objects, due to a tendency to smooth out details possibly relevant in local properties evaluation.
(4) Multiscale Model to Model Cloud Comparison (M3C2). It works directly on raw point clouds, with no meshing or gridding, and is split in two main steps: Estimation of surface normal orientation at a scale consistent with the local surface roughness Quantification of the mean cloud-to-cloud distance (i.e. surface change) along the normal direction (or orthogonal vector), which includes an explicit calculation of the local confidence interval. A point-specific normal vector is calculated by fitting a plane to neighbouring 3D points that are contained within a user-specified search radius. To avoid the fluctuation of normal vector orientations and a potential overestimation of the distance between two point clouds, the radius, or scale, used for normal calculation needs to be larger than the topographic roughness, which is calculated as the standard deviation of the local surface elevations. The orientation of the surface normal around a point is therefore dependent on the scale at which it is computed (Lague et al. 2013).

Test area
The present case study refers to Harzburger Hof, a luxury hotel built in 1874 in the forests near the spa city of Bad Harzburg (Germany) (Figure 1). The hotel had a wooden structure and included several bodies connected to each other to form internal courtyards. In May 2014, a fire burst out in the southern body, quickly extending to all floors; as a result, the whole structure and the immediate surroundings were not safely accessible.
After a few weeks, researchers of the Geodesy and Photogrammetry Institute -Technical University in Braunschweig carried out a 3D survey campaign of the involved area (about eight hectares) in order to produce metric documentation of the status quo.
Survey data have subsequently been made available to the authors.

Survey methodologies
The following geomatic techniques have been used for surveying:

TLS UAV-borne photogrammetry with horizontal flights
The use of multiple surveying methodologies was required in order to allow, after the integration of the different surveys, achieving a complete survey of the whole complex inspite of the difficulties entailed by the disaster. This has also highlighted the problems arising when applying these methodologies to dangerous, precarious areas.
Three scans have been performed outside the hotel using a Riegl Vz1000 terrestrial laser scanner (standard deviation 8 mm at 100 m, as stated by the manufacturer): two on the front, henceforth referred to as L1 and L2 respectively, and one on the back, with a mean distance of roughly 40 m and resolution just about 2 cm. The scans have been provided in ' Ã .pts' format, which also includes intensity values for each point.
UAV-borne photogrammetry has been carried out by a Canon IXUS 220HS with 35 mm equivalent optics, 12.1 Mpx backlit 1/2.3" CMOS sensor and weight equal to 141 g with battery and memory card. This has been mounted as payload on an AirFrame F450 multirotor, fitted with an APM 2.6 Autopilot system and Global Positioning System (GPS).
Planning for UAV-based data collection provided for image shooting along the horizontal strips with pseudonadiral axis; the mean flight level was approximately 45 m with Ground Sample Distance close to 1.74 cm/px and 85% overlap along both axes, collecting 886 images during a 7minutes flight. Mission Planner 1.3.7 software was used for flight path management (Figure 2). In order to achieve a 3D model of the status quo ante of the exterior by integrating the survey methodologies, the coordinates of some Ground Control Points (GCPs) were measured in the field (Figure 3). Fourteen GCPs have been identified by markers or studs and laid out both at ground level and on the building according to the access restrictions imposed on the survey area. Measures have been carried out by both total station and Global Navigation Satellite System (GNSS) receiver in order to overcome the obstacles and logistical problems the operators have met with in the survey area.
The topographic survey was carried out by means of a Leica TS02 Total Station, separately set up near the main entrance (west side) and in the park area on the back (east side). Since these positions lacked reciprocal visibility and no common points could be sighted from both stations, they have been linked via a GNSS real time kinematic (RTK) survey, carried out by means of a Leica GS15 GPS receiver.

Post-processing software
The following images were processed by Agisoft Photoscan v.1.1.3 software to obtain the model, while TLS point clouds were managed and aligned via Leica Geosystems 3D Cyclone and JRC  Reconstructor. Comparisons between point clouds were carried out using Cloud Compare GPL software, using M3C2 algorithm with the local model defined by the quadric function Z = aX 2 + cXY + dY +eY 2 + f, with Z defined as the vertical axis.

Integration and comparison of TLS and UAV-based photogrammetry models
In the subject case, where logistics issues hampered the rigorous planning as to GCP number and layout, two separate photogrammetric models have been processed: one (SfM_A) obtained with topographically measured tiepoints only, and the other (SfM_B) where 35 accessory tiepoints were added on the images so as to ensure an even layout, with at least three tiepoints on each image (Figure 4).
GCPs provide a common reference system used for framing of laser scans and scaled rototranslations of the photogrammetric model obtained by Photoscan.
As previously stated, the disaster area presented accessibility issues that prevented its complete survey using only total stations; besides, buildings and vegetation affected the GNSS survey, with accuracy values ranging from typical RTK (2/3 cm) to up to 3x greater.
Since some portions of the building had been surveyed with both methodologies, the surveying performances have been checked for accuracy in the case of reference points with sub-centimetre precision.
To this purpose, the photogrammetric point cloud has been compared directly with the single laser scans in their original reference system.
The comparisons have been carried out on the two laser scans L1 and L2 ( Figure 5) referring to the main façade of the building.
In order to rototranslate and scale the SfM model, GCP coordinates have been measured directly on the TLS clouds ( Figure 6). Five points have been selected in both L1 and L2 on the sub-horizontal portions of the building, where higher rendering accuracy is expected on the photogrammetric model, according to the following criteria: From the TLS models, which lack colour information, well-identifiable points were extracted, such as summits of railings (point 4, in L1 and L2) (Figures 7 and 8), chimney masonry (points 1, 2, 3 in L1) ( Figure 9) and gutters (point 5 in L1 and L2; points 6 and 8 in L2) (Figures 7,8 and 10) In order to improve the detection of apparent vertices, cloud normals were computed, so as to exploit 'inclination' colouring, which highlights the changes in plane sloping Points on the ground were avoided, due to poor detectability on TLS clouds, as well as points close to vegetation Points selected via image detection were collimated in the Photoscan environment     Once the point clouds have been framed in the same reference system, processing was carried out according to the following steps: Creation of the CloudCompare project: the rototranslated block is exported in the ' Ã .ply' (binary) format, which holds geometrical information along with RGB data and 3D normal data (nx, ny, nz). This is read and managed in CloudCompare Point cloud cleaning: the point cloud is purged from vegetation, isolated points and points specific for either cloud. To this purpose, cutting polylines and common masks were set up in order to delete the same portions of space from all clouds For a more effective precision check, portions of point cloud were isolated from the models based on geometric homogeneity, considering first the sub-horizontal portions. Figure 11 shows the selected model portions: three sloping roof planes (T1, T2 and T3) and a vegetation-free ground portion (TR).  Subsequently, vertical portions were also considered, both with (AG1 and AG2) and without (V1 and V2) overhangs ( Figure 12).
Assuming TLS methodology as the reference, the photogrammetric model was then checked for precision, applying cloud-to-cloud check algorithms.

Results
A check between TLS point clouds T1 and T2 and photogrammetric point clouds SfM_A yields a mean difference of about 10 cm, with a standard deviation of about 1-2 cm on the ground, and a mean difference of about 3-4 cm with a standard deviation of about 2 cm on the roofs (Tables 1 and 2).
Tables 3 and 4 summarize the results of comparisons between point clouds obtained by TLS and by SFM_B. Results show comparable standard deviation (s) values, with a substantial improvement  in the mean difference. It is obvious that higher number and better layout of the tie points entail better modelling.
In this case, UAV-based photogrammetry and data processing with SfM and MVS algorithms can provide models with comparable precision as those obtained via TLS, and therefore can integrate the latter for else unavailable data. Photogrammetry expresses most of its potential in terms of precision when surveying objects mainly extending on a plane orthogonal relative to the shooting axis. The case of Tn and TR zones is an appropriate example.
However, images include also vertical parts, which are of course viewed with small looking angle. The photogrammetric model was checked for precision also on these portions, differentiating simple vertical walls (V) from those featuring projecting parts (AG).
Tables 5 and 6 summarize the results of these comparisons. As expectable, standard deviations are greater than those verified for sub-horizontal elements.

Conclusions
The results presented above show how, in case of necessity due to emergencies, integrating TLS with UAV-based photogrammetry can be quite useful.
It has been shown that the precision of UAV-based photogrammetry is comparable to that of laser scanning in parts lying almost orthogonally to the shooting axis. In the present case study, it can provide a useful solution to integrate the survey of roofs, gardens and inner courts, for which stability issues prevent accessibility.
On vertical parts, instead, UAV-based photogrammetry with nadiral axis provide lower quality. However, the possibility to use UAV-based surveys to process also portions acquired with limited view angle, such as vertical walls, is surely interesting.
In fact, as the rendering scale for these portions will not be the same as for sub-horizontal parts, the survey also allowed the restitution of all inaccessible and vertical external walls of the building (including inner courtyards), with structural stability issues.

Acknowledgments
Thanks to Federico Capriuoli for his patience and continued cooperation throughout the tests, providing fundamental help in conducting this research. Thanks to GRAVIONIC GmbH (spin-off of the Institut f€ ur Flugf€ uhrung (IFF) of TU Braunschweig) and GRAVIONIC CEO and publicly appointed surveyor Eng. Ralf Heyen, for contributing to this research by making available all necessary equipment and sharing his technical knowledge. Thanks to IFF PhD degree students, Christian Tonh€ auser and Markus Bobbe, for providing, respectively, TLS and UAV-based photogrammetry data, without which the present study could not have been carried out.