Reconstruction of 3D digital heritage objects for VR and AR applications

ABSTRACT Constructing the 3D objects based on geometric modelling and graphical techniques is a well-known research activity applied in computer graphics. Techniques to process graphical models are widely used in the field of digital heritage and 3D game. Virtual reality (VR) and Augmented Reality (AR) are technical trends nowadays that can be studied and used in digital high-tech fields like medical training, digital heritage, entertainment industry, digital tourism and museum, etc. In this research, we present a new proposed method for reconstructing the 3D objects of tangible cultural heritages in the virtual environment based on the combination of geometric modelling, computer graphics, VR and AR technologies. The method consists of the following steps: we first collect data of a real object by using a 3D scanner. After processing obtained data, the output is a 3D point cloud. In the next step, we remove noisy data and triangulate the object surface. The novelty point focuses on reconstructing the 3D object by filling the holes. At the end, we build VR and AR applications for visualizing a virtual museum. The contribution of this research leads to open the door for applying in other fields such as 3D Game industry or digital tourism.


Introduction
The mathematics is always the foundation for building algorithms applied in the field of information technology (IT). As a topic of mathematics, geometric modelling is the background used to process the 3D objects in the research field of computer graphics, computer vision and image processing. These research are more and more attractive and widely used in practice for 3D applications, both in scientific research and industry. Building an object from a real dataset, reconstructing the 3D objects or creating a virtual environment to simulate, render or restore the real objects, etc. are popular topics of the researchers in IT, 3D art, designer, mechanics or even in archaeology, etc. Therefore, processing the 3D objects can bring us interesting research topics that is not only applied in the fields of graphical industry but also used in the application of digital tourism, heritages or museum.
Recent years, VR and AR applications are regarded as a hot keyword on the internet searching. They are developed and mostly used in a virtual environment or combined in AR applications. They can be a simulation program, an experiential system based on virtual models, even a virtual museum or virtual tourism, etc. Digital heritage is also a popular research based on the application of IT. It can be considered as a real example in the digital transformation step applied in the field of historical and cultural conservation of the world.
Due to the time, the war, the natural disaster and even the devastation of human our tangible cultural heritages have been destroyed. Therefore, how to restore these cultural heritages, especially the tangle heritages for the purpose of research, conservation and development are not only the duties of researchers in the field of culture but also the duties of scientists in mathematics and IT. With the supporting of computer, the ancient heritages can be restored exactly and rendered on the graphical environment. The research products in this field not only help to reconstruct the real models but also bring us a lot of values in history, cultural and scientific evidences of human civilization. The several researches in the field of computer graphics, image processing, computer vision are more and more developed. The techniques, utility tools, applications and their libraries in IT are also studied and widely applied in digital heritage. We are lucky living in the digital age, the age of digital transformation with a rapid and very high development of IT that is always impact and made our life changed. That are advantages and necessary conditions for us to do our research in this article.
In this article, we propose a new method for building and restoring the tangible heritages based on the combination of geometric modelling and computer graphics. The reconstructed 3D objects are then rendered on the virtual environment and also augmented the interactive processing based on the VR and AR technologies. The novelty and our contribution in the research are focused on restoration and visualization of the 3D tangible heritage objects based on the geometric modelling, computer graphics combined with VR and AR technologies. Our proposed method consists of the following steps. In the first step, we use a 3D scanner to acquire data of the real objects (the statues). The obtained data is then processed and converted into the 3D point clouds for the second step. After removing noisy data points, we triangulate the object surface. The important step is focused on the restoration of the real objects by filling the holes on their triangular surfaces. The reconstructed 3D object is refined to obtain a model that is approximated to its initial shape. In the last step, we build a virtual environment for visualizing and handling these 3D objects by using a VR Headset and interacting them by using an AR application on a mobile phone. Comparing to the several methods, our proposed method is processed faster and more accurate, both in reconstructing and rendering the real objects.
The remainder of the article is structured as follows: Section 2 reviews the several methods in the fields of reconstruction of 3D objects, application of VR & AR in practice. Section 3 presents our proposed method in detail. Section 4 describes our implementation and comparison. In this section, we have also evaluated and discussed our obtained results to show advantages of our method. The last section is our conclusion and future work.

Related work
The usage of high-tech devices in acquisition and processing big data is more and more proved efficiently. They are well known such as UAV, Drone, Laser scanner, Lidar techniques and so on. The research in cultural heritages nowadays are not only interested by the culturalist, historians, archaeologist, but also the mathematics, scientist and researchers in computer science. AR and VR applications are considered as solutions to support visitors in different ways for visiting the cultural heritages. Digital exhibition is also one of the good solution in many fields such as virtual museum, car showroom, virtual tourist and so on, where users and visitors can be immersed in a virtual environment but it is nearly reality .
In this section, we explore and review several methods focused on 3D modelling of real objects based on VR and AR technologies. Jurado et al. (2018) suggested a method for 3D modelling and provided a real-time interaction, visualization and management of underground infrastructure. The method is implemented based on a collaborative web-based application that supports using VR glass to interact with the GIS data and also performed with mobile platforms. The contribution of the method is described as follows: reconstruction of a 3D model; building VR application; real-time handling and interaction with a mobile app. This is an interesting application of VR and AR in 3D presentation and visualization. Quant et al. (2018) presented an experiment in visualization of spatiotemporal 3D data based on a virtual reality (VR) interface. The paper described an implementation of the VR application based on the Unity 3D. However, the important point of algorithm and data processing is not mentioned. Munoz-Arango et al. (2019) presented an analysis of the pixel spread correlation with lenticular lens efficiency on multiuser VR displays. The method is based on the techniques of computer graphics such as color, light processing that applied directly on pixels of the image. Barsanti et al. (2016) proposed a method to simulate and visualize the 3D point cloud obtained from a drone device. The contribution of the method is expressed in data processing. After processing the obtained data, they are rendered on a VR device for visualizing the 3D objects. Although the obtained data have shown the shape of 3D object with additional colour information. However, the method for reconstructing the real object is not mentioned. An applied model for rendering digital heritages is presented by Louis-Pierre et al. (2015). After reviewing several methods for VR and AR applications, authors presented a technical model based on the combination of VR device (Oculus Rift DK2) integrated with the Leap Motion and its software application on the Unity 3D to visualize 3D objects at archaeological museum of Milan in Italy. Li et al. (2018) presented an application of a hybrid VR and AR system for exhibiting and visiting the virtual heritage objects. The model is setup based on Unity 3D environment using HTC Vive for VR and a Samsung Galaxy S7 for implementing AR. The paper described the method to implement and test with an input number of different heritage objects on the devices of AR and VR. The obtained results have also been shown. However, the solution for processing obtained data and reconstructing the 3D objects is not presented. An AR-App is presented by a research of Ozturkcan (2021), which is an interesting and useful tool in practice. The application allows users to choose a location and visualize how furniture will look in their own home.  described a novel rendering system for processing the 3D models on a VR application. The method is processed based on the techniques of computer graphics such as multi-pass rendering pipeline, using power of hardware devices. The application allows users to interact, immerse and visualize the 3D point clouds of heritage models in the virtual environment. Rasheed and Nordin (2020) presented a method for reconstructing the ancient objects from broken fragments based on the image processing techniques using the CNN model. The method consists of two steps: the first step is namely CAF (classification of ancient fragments) based on the geometrical computation and the second one is RAO (reconstruction of ancient objects) based on CNN model to match the components into a completed object. The research work contributed and supported a lot to the archaeologists, obtained a really good precision. However, the challenge is processing the gaps that happen because of missing some fragments. To process a point cloud dataset acquired by laser scanning techniques, Kumar et al. (2019) suggested a nonend-to-end (non-E2E) method for deep learning classification. Using multiple sets of input features, manually extracting features from the point cloud and then feeding them through a fully connected neural network to compare with the state-of-the-art deep learning frameworks. The data analysis is based on the irregular data structure of point cloud to extract features from the normal local vector and additional features that related to spatial point distribution and use them together for the classification. The method can improve accuracy in classification even the training dataset is highly imbalanced in experiment. However, the used dataset is labelled manually, many errors may happen during the labelling process. To reconstruct the 3D facial object, Rotger-Moll et al. (2019) proposed a method based on the image processing techniques. The method consists of two steps: a deep learning approach to obtain geometrical features of a 3D coarse mesh and a graphical technique (like colour, illumination and photometric brightness) to characterize and map each pixel on the image to a corresponding vertex of a mesh. After processing, the obtained data is a 3D face from a single image. However, the 3D object is not fully reconstructed as a real object in practice.
In general, reconstructing 3D objects always brings us the reconstructed objects that are as close to their real shapes as possible. However, it is not an easy task in practice. Which method or solution in the several techniques is the best selection? It is still a challenge to the researchers. The combination of these techniques to propose a full process in this paper is also our contribution. Our proposed method includes many steps: obtaining data, processing data, reconstructing the 3D objects and building a VR & AR application. Among these steps, the novelty is focused on the 3D reconstructive step. A detailed description of our proposed method is presented in the next section.

Overview
Our proposed method is based on a computation of geometric modelling and techniques in computer graphics. We scan data from a real 3D object by using a 3D scanner device. The obtained data is then processed and output is a 3D point cloud. The surface of 3D point cloud object may contain many holes generated by the scanning step. They are then detected based on geometrical computation of points density on the surface based on the method by Nguyen et al. (2021). The data object is reconstructed by removing noisy points, triangulating the surface and filling the holes on the triangular surface based on the previous works (Nguyen et al., 2019(Nguyen et al., , 2018. We build an application for visualizing these 3D objects on both a VR headset and a mobile application. They are built and simulated as a real museum in practice (see Figure 1).

Collecting and processing data
Many several tools and technical devices can help to collect data depending on the materials of the real objects. A digital camera, a Fly-Cam or a Drone that can take a picture, scan surface, record a video file or capture data from the real objects. The obtained data is then processed for building and reconstructing these objects such that they are as close to their initial shapes as possible. In practice, the 3D scanner that can process the real statues at museum is very expensive. In this research, starting from the idea of previous work (Nguyen et al., 2019), we use a small 3D scanner (the Shining 3D EinScan-SE (EinScan, 2021), see Figure 2) to test in different cases to obtain data and check accuracy. Because of limitation of the scanner, we scan samples with smaller size comparing to the real one in practice. After obtaining data, they are processed and converted into 3D point cloud format for the next processing step. The output of data collecting step also support to the PLY and STL file formats in 3D. PLY is a polygon file format storing graphical objects that are described as a collection of polygonal components such as vertices, faces and other elements. Along with some additional information like colour and normal direction that provide characteristics of a real object in 3D space. We use Meshlab (2021) as a tool to convert data into a 3D point cloud (see Figure 2, on the right)

Reconstructing the 3D objects
The obtained data contains noise of 3D point clouds and some parts on the surface are lack of information (called holes) during the scanning step. In the previous work   Nguyen et al., 2019), we used an RGB-D camera to capture data and processed obtained data. In order to remove noise data, the voxel grid filter is used by keeping the central point that represents for each group of points within a threshold distance between the points (called voxel). The redundant data around the real object are removed based on the computation of cluster. In this research, we use the 3D EinScan-SE that can help to process the scanned data. After converting obtained data into 3D point clouds, the shape of real object is presented as in Figure 3. Depending on the shape of the objects and angles of the camera, the scanning process cannot cover completely all surface areasof the object in different direction. Therefore, some data on the surface areas may be missed as in Figure 2    and triangulation of data objects. However, this research is only processed on the 2D planar of dataset, while Nguyen et al. (2015) already proposed a fast method on the 3D point clouds dataset. The proposed method that overcomes the drawback of Delaunay triangulation was also suggested by Nguyen et al. (2014) for filling the holes on a triangulation surface. That method is evaluated processing faster than the traditional Delaunay method for creating a triangular surface of the 2D object as usual. In this research, we combined and applied previous works (Nguyen et al., 2014(Nguyen et al., , 2015 to determine holes, triangulate the surface and fill in the holes on the object surface. We propose a method to reconstruct the object as follows: . We find holes on the surface of a 3D object by detecting hole boundary points p b based on density of points (Nguyen et al., 2018(Nguyen et al., , 2015. For each 3D point p i on the surface, we check its neighbouring points. If one of point in its 8-connectivity is empty, it is determined as a boundary point. After the first looping step, we can determine all boundary points of the holes. The 8-connectivity is computed based on a tangent plane at the point p b (as presented in Nguyen et al., 2018). . Before filling the holes, we triangulate the object surface based on the existing methods in Meshlab (2021). The boundary points on the hole are now becoming vertices of the boundary triangular faces. The edge that is only shared with one triangular face become a boundary edge (see Figure 3(b), detail in Figure 4: the red lines) . For each hole, we process in a sequence, starting from the first ring of hole boundary edge R(e b ) to the inside of hole. At first, the normal vector at each boundary point p b is computed. We apply the method in Nguyen et al. (2018), for each p b , we compute its tangent plane based on 8-connectivity of p b , the missing point in this plane toward the inside of hole is considered to fill. . The process is repeated on the next ring toward the inside until the hole is fully filled.  Depending on each hole, we have two different ways to compute and fill into the triangular holes. In general, we first compute the size of holes based on its diameter (the distance between the p b ). If the hole is small, we then filled only one central point p c in the hole (see Figure 4(a)). If the hole is larger, we base on the hole boundary ring R(e b ) to loop and fill on each ring toward the inside (see Figure 4(b)).

Refining the holes
As illustrated in Figure 4(a), in order to preserve the local shape of the hole surface patch, we first estimate the normal vector of the central point p c in the centre of hole. It is determined by computing average of all normal vector of p b on the first ring of the hole as follows: where n is the number of boundary points. The p c is then considered as a barycentre point. We use it to measure and base on it to compute an approximation of each normal vector of new point p new on each next ring to the inside of hole. If the hole is small (as mentioned above), depending on the distance between the p b and the distance from p b to p c , we only use one loop to fulfil the hole. In the contrary, the process will be repeated on the next ring (it is now become the first ring for the continuous loop) toward the inside of hole. During each time to fill a new point p new , it is refined by adjusting its coordinate p new (x, y, z) to approximate the local shape of the hole surface patch. Mathematically, the slope of line SL(MidPoint(e b ), p eb ) (it is a median line of the boundary triangular face, see the red dash line in Figure 4(a)) may decide the shape of local curvature of the hole patch. In this case, assumption of the axis Oz is up and perpendicular to the plane Oxy, we compute the SL based on the formula as follows: where D x = x(MidPoint(e b )) − x(p eb ), D y = y(MidPoint(e b )) − y(p eb ) and D z = z(MidPoint(e b )) − z(p eb ) respectively. From (1) and (2), we compute the coordinate z(p c ) as follows: After that, the coordinate z of new point z(p new ) is computed based on the z(p c ) and z(p b i ) to preserve the approximation of the surface patch as follows: In order to fill the hole, we first determine the size of hole. If a threshold value of hole diameter is less than or equal to 2.5 of the average of boundary edges (it means the hole is small), we just fill a unique central point p c . In the contrary, we determine the ring of boundary edges R(e b ) and loop (ring by ring) following clockwise direction to fill all new points for each ring from the outside to the inside until the hole is fully filled (as illustrated in Figure 4). In general, the algorithm (Algorithm 3.1) to compute new point p new , and fill in the triangular hole on the object surface as follows: Algorithm 1 HoleFilling(S,H) 1: for each hole H on the object surface S do 2: rotate the object such that the axis Oz is up 3: find all boundary points p b of H 4: compute ds (the average boundary edges e b ) 5: compute diameter of hole dH 6: if d H = 2.5 of ds then 7: insert the central point p c 8: create triangles by connecting lines(p b i , pc) 9: else 10: for each ring starting from the outside to inside of H do 11: determine first hole boundary ring R f (e b ), process in a clock-wise order 12: for During the time of filing in a triangular hole, we always expect to create a triangle that is as close to equilateral triangle as possible. However, depending on the shape of hole, a new triangular face (in some cases) may be created very thin. This drawback is solved by Figure 5. The triangular surface of a real 3D object after filling some holes on the surface. On the left, using method 'Close Hole' on Meshlab, the hole is flat. On the right, using our method, the hole patch is adapted to local curvature of the surface. using the Delaunay criterion (i.e. Delaunay edge) as presented in Nguyen et al. (2014). Figure 5 is a real 3D model after triangulating and filling the holes on its surface.

Building and visualizing
In this research, our goal is to create a virtual environment that allows users immerse in the ancient space of the museum. Users can see and interact with the statues by hands as in the real museum. In the previous work , we have implemented and shown our application of VR and AR. We are now improving both our VR and AR applications with the reconstructed objects of tangible cultural heritages. In practice, at Da Nang Museum Cham Sculpture (Museum, 2021), there are many artefacts, opening to public a part of its storage. Each of them contains a historical story with cultural values during the life of time. However, it is difficult for us to come and see directly; especially in the Covid-19 pandemic now. Therefore, a VR or an AR application for the visitors will also open a research trend, not only for developing VR and AR application in practice but also for researchers in computer science, archeologist and game industry.
After reconstructing the 3D shapes of real objects, we use them as input data for building our VR and AR applications. The VR application will design and implement on the computer based, while the AR is performed on the mobile application based. In detail, the previous work  described as following steps: . Setting a 3D environment for museum . Building rooms and creating 3D objects . Writing functions to handle the objects based on control pointers. (2020), the backgrounds are required to understand and create architectural space and historical locations for visualizing 3D objects using digital tools. It is useful to integrate for building a virtual museum on the VR application. We use the Unity 3D (AR, V, 2021), a useful and professional environment to build our application. Using components, tool-box and their functions to build a 3D space. At first, the layout drawing allows us to create many rooms to display different statues. Depending on idea to display artefacts in the museum (as in practice), the statues can put on a table, a pillar or on the rotation pedestal in the middle or somewhere in the room, where we can handle, interact, and read their history information automatically. Besides, we can create many furniture to support the art space based on 3D art packages such as light bulb, adjust light direction, illumination with color, brightness, etc., (see Figure 6, on the left). For each 3D object that reconstructed from previous steps, we add it into the right place with different controllers. The script packages in C# programming language are supported to write source code to process different interactions. Algorithm 3.2 describes many functions to interact with the objects in both VR & AR applications. After building VR-App on the Unity, we deploy it into a VR Headset device (an Oculus Quest 2) for visualizing and handling objects by using two controllers.

As Kumar introduced in Kumar
The AR application is designed as a supported function in practice for user when visiting the museum. After building AR-App on the Unity, we also deploy it into a smart phone. For any tangible objects, user can check more information by moving a smart phone (with camera) over the object to see more additional information in detail (see Figure 6, on the right). Besides, visitors can also interact the objects directly by touching on their mobile phone instead of touching directly the statues, which is forbidden following the regulation of museum in practice.

Algorithm 2 Controlling objects in VR & AR 1:
Input: 3D object (*.PLY, *.OBJ, etc.,) 2: Output: 3: List of chosen actions for user: 4: if selected action = translation then 5: Translate object up/down/left/right 6: else if selected action = interaction then 7: Touch object and handle 8: else if selected action = rotation then 9: Rotate object in any direction 10: else if selected action = scale then 11: Scale object bigger/smaller 12: else if selected action = information then 13: Display history information of the selected object 14: else 15: Show the name at each object 16: end if 17: Output: VR & AR application

Implementation and comparison
In this section, we implement our proposed method and compare obtained results with the several methods to evaluate the accuracy between them. We perform our method by using C++ programming language as a Plugin and integrate it in Meshlab (2021). After reconstructing the data objects of tangible heritages (i.e. the statues) on the Meshlab, the output data is transferred to the VR and AR applications that are built based on the Unity 3D (AR, V, 2021) on a computer MacBook Pro (2 GHz Intel Core i5, 8 GB RAM). Figure 7 shows an input object (on the left) after scanning a real statue that contained some holes. After triangulating and filling on the triangular holes, the output is a reconstructed object shown on the right. Figure 8 is one more sample that we used our method to fill in the holes to compare the obtained results with the methods Close Hole and Poisson on Meshlab. For both comparison of observation and accuracy, our proposed method is generally better. In order to evaluate the accuracy of a reconstructed 3D object   2016)). In this research, we base on a computation of approximation errors between the triangulation of the object surface (before filling the holes S1 and after filling the holes S2). We use the measurement in Meshlab (the Hausdorff error comparison between reconstruction results and ground truth synthetic dataset Cignoni et al., 1998;Meshlab, 2021). We measure the maximum error D max (S1, S2), the mean error D avg (S1, S2) and the root mean square RMS(S1, S2). We compare these values (D max , D avg and RMS) between the existing methods in Meshlab (Close hole and Poisson) and our method in Table 1.
In order to fill in a triangular hole, the Close hole method connects between the boundary points of hole to create a new triangle until no one boundary point to be connected. Therefore, after filling the holes, it looks flat and did not adapt local shape of the hole (see Figure (8(a))); and the approximation errors are larger than our method and Poisson method (see Table 1). In the Poisson method, the holes are filled regularly. However, the density of triangulation is not adapted and constrained regularly with the boundary faces. Therefore, depending on holes on the object surface and their local shapes, they are not completely obtained by the best results. For example, the obtained value of D max of the Monk in Table 1 is less than our method. However, the values of D avg and RMS are larger. In our method, we compute the average length of boundary edges for estimating and creating new triangle. Therefore, after filling the hole, it is constrained with the boundary faces and adapted local curvatures of the object surface. In conclusion, the obtained results of our proposed method have shown advantages comparing to others and the reconstructed 3D objects approximated their original shapes (see Table 1).

Conclusion
In this article, we have researched and proposed a method for reconstructing the 3D objects of tangible heritages in practice. Our method is proposed and implemented based on many steps. We collected data by using a 3D scanner. The obtained data is processed on Meshlab and exported a triangulation of the 3D objects. They are reconstructed by filling the holes and become the input data for VR and AR applications to simulate for a virtual museum. The whole method is based on the fields of geometric modelling, computer graphics, VR and AR. Our contribution focused on the reconstruction of the real objects. The shape of 3D objects is restored and refined better than existing methods and close to its original shape in practice. This is the most important point, not only to the researchers in computer science but also to the archeologist in restoration of the cultural heritages. The final VR and AR applications have proved an efficient way to provide the historical and cultural values of heritages to the community. Because of the time, the war or natural disasters, many heritages like temples, relics, artefacts, statues, etc. are not enough information and difficult to reconstruct completely. In addition, the price of the 3D scanner is very expensive. The one that we used in this research only scan artefacts with small sizes. To solve these problems, a combination between the researchers in different fields should be extended as our work in the future.