Augmented reality and digital twin system for interaction with construction machinery

ABSTRACT Smart construction utilizes the latest IoT “smart” technologies, such as Cyber-Physical Systems, to aid in construction processes. A Cyber-Physical System (CPS) is an amalgamation of computation, networking and physical objects. There are various examples of CPS being employed in sectors varying from manufacturing, robotics, and smart cities. Where the application and literature are lacking, however, is construction. This paper focuses on construction machinery operation and work tracking through CPS. The relevant technologies required for creating a system with these capabilities are Augmented Reality (AR) and Digital Twin (DT). Two salient features of a CPS are real-time state knowledge and control. These are handled by the sensors and micro-controllers, respectively, allowing for bi-directional communication between the virtual models and physical objects. Remote control and tracking/monitoring of construction machinery can expedite certain on-site processes such as operations logging and 4D BIM, while improving operation precision and safety.


Introduction
Construction management is a multi-dimensional field requiring varied techniques and technologies to facilitate effective project monitoring and management. With the current trend towards implementing smart technologies in various industries, many new mechanisms are being explored to bring the construction sector to the world of IoT and industry 4.0. Introduction of CPS to construction will have many benefits associated with IoT including functionality, real-time interaction, early warning, instruction for workers, non-intrusiveness, remote interaction and safety (Yuan, Anumba, and Parfitt 2016). To create such a system requires a physical structure/object, virtual model, virtual-physical bridge and an optional portable device like a smartphone (Yuan, Anumba, and Parfitt 2016;Genders, Wang, and Razavi 2016)figure 1-9.
Tools for designing and management in construction projects have evolved over the years; from paper drawings to CAD to Building Information Modelling (BIM) and Geographical Information Systems (GIS) (Meza, Turk, and Dolenc 2015). Many of this type of software have now introduced AR/VR compatibility via plugins for viewing their models using the required headsets. Although the technologies of Industry 4.0 have been applied for static data/model visualization in construction, proper frameworks to utilize all the advances of IoT in the construction process have yet to be widely adopted.
AR has been implemented in the construction and architecture (AEC) industry in many capacities including maintenance of facilities, construction safety, collaborative design, inspection and other orientation tasks to aid engineers/site workers (Genders, Wang, and Razavi 2016;Meza, Turk, and Dolenc 2015;Heinzel and Azhar). There are multiple benefits of AR in Civil Engineering like error reduction, marketing, project review, cost and time reduction (Agarwal 2016).
DT technology is being implemented in manufacturing and aerospace industry for various purposes in Computer-Aided Design (CAD), Computer-Aided Manufacturing (CAM) and Computer-Aided Engineering (CAE). So, in the applications of design, monitoring, prototyping and training, DT has been successfully utilized (Madni, Madni, and Lucero 2019). Another application of DT that is relevant to the field of construction is virtually mirroring testing machinery as developed by Haag et al. in their bending beam test concept for Digital Twin (Haag and Anderl 2018).

Literature review
The Fourth Industrial Revolution, also called Industry 4.0, is characterized by the cross-linkage of digital and physical domains for integrated cyber-physical systems. The greater focus in Industry 4.0 is utilizing IoT to increase monitoring and control capabilities rather than simply increasing productivity like previous industrial revolutions. The Internet of Things (IoT) is a relatively recent phenomenon characterized by the inter-connectivity between various computers, smart devices, hardware, electronics and other objects in a specific environment. The automation of day-to-day processes which human interaction minimized to instances of data analysis or system override is a key characteristic of this technology from the Fourth Industrial Revolution.
A key feature of Industry 4.0 is Cyber-Physical Systems. Cyber-Physical Systems (CPS) are defined as dynamic systems that integrate computation with physical processes, often in feedback loops, where physical processes affect computations, and vice-versa (Lee 2008). The essential component of a CPS is the "bidirectional communication" (Akanmu 2012) between its physical and virtual elements. The pre-cursor of CPS known as embedded systems are already widely used in sectors of transportation, healthcare, defensesystems, electronics and critical infrastructure (Anumba, Akanmu, and Messner 2010). Cyber-Physical Systems may be considered an extension of embedded systems that have been developed over the past decades, with a focus on integrating physical processes and computation as a network. The focus of CPS in industrial settings has been on flexibility, customization, interaction and extended functionalities (Leitao, Colombo, and Karnouskos 2016;Monostori et al. 2016).
Lockheed Martin, Boeing, General Electric Aviation, Airbus, Lufthansa and Rolls Royce Aircraft Engines all employ Digital Twin, Augmented Reality or Cyber-Physical Systems in various capacities ranging from production line monitoring to maintenance/repair. Companies that have introduced AR, DT and CPS in their manufacturing facilities also include Ford, IBM, Siemens and Vuzix; with the likes of Porsche or Mitsubishi Electric using Industry 4.0 technologies for training/operational aid purposes. According to Lee et al.: "Cyber manufacturing is the concept of intertwining IoT technologies into a transformative system that uses interconnected tether-free assets to achieve resilient performance." (Lee, Bagheri, and Jin 2016). One idea was put forth by Ding et al. in the form of smart steel bridge construction using BIM and IoT (Ding et al. 2018). Correa et al. mention CPS integration with BIM and GIS requires a transition from Product Data Models to Process Data Models, for smart monitoring/control through the phases of a construction project to operation or maintenance (Correa and Maciel 2018).
The multiple definitions of Digital Twin (DT) in literature synopsize to it being a representation of a physical entity via a virtual model that mimics its change in state whether it is mechanical motion or physical/dimensional changes. According to Cheng et al. (Tao et al. 2018) ". . . digital twin consists of three parts: physical product, virtual product, and connected data that tie the physical and virtual product." It is characterized by its ability to mirror, simulate or monitor the system/physical environment (Zheng, Yang, and Cheng 2019). While there are multiple instances of DT being utilized in industry, most  (Negri, Fumagalli, and Macchi 2017).

Applications of DT
Operation health analysis Digital mirroring of physical entity Engineering and statistical analyses Expected outcomes

Monitor anomalies in physical twin
Study long-term behavior and predict performance of system  (2017) describe augmented reality (AR) as digital graphics being superimposed on a real-world viewfinder. In the Reality-virtuality continuum AR integrates the virtual environment with the real environment, meaning it is locally applicable, hence its significant potential in construction.
Another concept for the application of Industry 4.0 technologies in the construction sector is utilizing "Smart Construction Objects". These are agents in an IoT framework that are construction resources with added capabilities of networking, sensing or processing. The added properties of autonomous communication and awareness are what make traditional construction objects SCOs (Niu et al. 2016).
With the various literature available on CPS, AR, smart construction and MCU nets; this paper's purpose is to present a prototype system utilizing the aforementioned technologies in the context of construction machinery operations and management.

Research objectives
With the current focus on researching smart interaction between industry elements and IoT devices, the construction sector is lagging in technology implementation relative to other industries such as automotive manufacturing and maintenance. Specifically, construction machinery operation in the context of project management is still a very milestone-based affair, i.e., progress tracking of machinery operations is done via completion stages of nD BIM. A real-time smart operation and monitoring system designed for construction machinery will improve project schedule and cost tracking. Additionally, it will add another dimension to sensor-based safety management by tracking not just labor movement and practices on-site, but also that of the equipment.
The research objectives were defined by a need to develop an Augmented Reality (AR) and Digital Twin (DT) based Digital Physical Link (DPL) of computing devices found in most construction projects with construction machinery. The objectives could be defined as individual deliverables to implement Augmented Reality and Digital Twin in a construction context to develop: • wireless link between digital and physical realm • control interface for physical model including AR capability • real-time physical object representation by digital model (3D) Figure 1. Flowchart for development of the digital-physical link system.

Scope and methodology
The purpose of this study is to develop a prototype for a scalable, modular Cyber-Physical Systems framework that is compatible with Building Information Modelling platforms and smart devices, used to interact with construction machinery while monitoring their operations. This research was laboratory level and small robotic analogues were used in lieu of construction machinery. The digital aspect of the DPL/CPS, however, has been developed to be applicable in real-world scenarios once operating components of the robotic prototype have been embedded in construction machinery.
In the initial stages of research, a development sequence flowchart was drawn after initial concept development. For the most part, the research has closely followed the sequence laid out below: The figure above illustrates how the development was approached as a three-pronged process. The physical model was constructed, and its MCU sensor/actuator net was programmed on the MCU's Integrated Development Environment (IDE). The DPL server and the GUI for AR and DT were developed on a C#-based 3D platform. The three components were then integrated on the selected development platform to form the prototype system of this research.
After development of the DPL prototype, the following operating parameters were tested to ensure the satisfaction of research requirements:

Physical model
For simplicity of the robotic armatures, the prototype model is an analogue of a stationary tower crane, which has two planes of motion. The digital interface can be adjusted easily for representations of other types of construction machinery, since they would require just adding and re-arranging the same types of actuators/motors used. While the physical model necessitated only the electronic processing, sensing and kinetic components to interact with the digital model/interface, a robotic chassis was also created to house those components to visualize the effectiveness of the system during operation.
A Micro-Controller Unit (MCU) is a small computer on a single chipset. For this research the MCUs from Arduino were selected because of their ubiquity. The popular Arduino Uno and Arduino Yun were used. The advantage of Arduino Yun over the Arduino Uno for the purposes of remote operation of machinery is a more compact package due to the integrated Wi-Fi module and Linux capability from its second processor. The MCU is used to bring together the sensor/actuator network of the physical model to work harmoniously and facilitate coordination with the digital media.
For the two degrees of motion of the selected prototype, a standard and a continuous servo were    selected. Standard servos operate in gradations according to the angular value serial input. Continuous servos behave more like DC motors as they rotate continuously in either direction as the required speed. Two types of sensors were used in the physical model: • Ultrasonic distance sensor: The HC-SR04 sensor is used for: Various software can be used to program an application that can transmit data to and from the MCU while interacting with the 3D model. This integration of the virtual realm with the physical protype required developing a framework that translates the serial data from the sensors to motion of the Digital Twin. Multiple platforms have such functionality including Rhino3D, Unity and Processing. Unity was selected for its ubiquity and extensive online forum support. A script was coded in its game engine and in the MCU sketch application.

Connection and interaction
The framework for control and operation was based on an application for smart handheld devices. The application needed a graphical widget control panel for operation and sensor data monitoring. The augmented reality module of the app will have the widgets superimposed on a marker-based AR viewfinder. The MCU on the physical model was programmed using the relevant libraries to perform the following tasks: • Send data to the servo motors • Receive serial data from the sensors • Define operating signals for the servo motors • Translate sensor data according to standardized units of measurement • Establish wireless connection with the server and smart devices The DPL required a mobile application for interaction with the physical model. The programmable widgets send and display serial data on the control panel or AR overlay. The interface was developed in a way to be modular, allowing for changes made according to the operating capabilities of the construction machinery. The AR viewfinder detects installed markers on the machinery and display the programmed interface according to the database. The server performs the following tasks within the CPS framework: • Maintain an AR marker database • Facilitate the data stream between the virtual model and physical prototype

AR controller
The AR controller required developing an Arduino sketch that includes the sensor/motor libraries and operating parameters. Additions were made to the sensor/motor network interaction sketch to allow data transmission over the network. The server application maintains communication with the smart handheld devices that are also updated with the required AR interface for sending data commands and receiving serial data of the MCU model. The access to Arduino via Wi-Fi is established using the local host address. Once the wireless parameters have been defined, the Arduino can receive commands and send serial data as defined in the sketches uploaded via the IDE software. The sketch is the terminology used by Arduino for the code developed in the proprietary Arduino IDE software. These codes can be written in C and C++ language and a variety of libraries are available for various functions.
Some of the libraries used in the code for the prototype include the default servo, Wi-Fi bridge, and open-source examples for the sensors. Using the formats defined by these libraries a modular sketch was developed. The advantage of this modular sketch is that more servos or sensors can be added to the prototype while minimum tinkering would be required with the code.
Marker-based AR was used due to lower latency and error potential. Another advantage is that, since a construction project has numerous machineries of every type, each type/model of machinery can have one marker instead of one for each individually. The database that stores information about each marker, with the associated AR interface graphics and functions, was developed using Unity3D in the form of a server with the AR interfaces and marker images uploaded as assets.

Digital twin
The Digital Twin was integrated in the server application on the PC platform. Before the DT was setup, however, direct interfacing with the MCU was established to verify correction operation of all modules of the physical analogue. Direct interaction with physical model in the form of serial data over Wi-Fi was the first step in establishing functionalities of all components. Once the sketch had been uploaded to the Arduino, data can be sent/received from the servo/sensor setup using pre-defined commands in the serial monitor.
Features relevant to creation of this CPS are:    A 3D model was developed with the file format .obj, due to the vast array of modelling software available utilizing it as well as its compatibility with common BIM software such as Sketchup 3D. Steps to developing a 3D model of the prototype physical object were: • Model the basic components of the physical object • Add armatures for motion of the 3D model • Add details to model • Export model to simulation software like Unity After importing into the Unity3D project, parameters were defined so incoming serial data from the Arduino were reflected in the movement of the model.

Testing and operation
The operating components of the developed CPS can be sub-divided into functions of the physical model, the Digital Twin server and the Augmented Reality application. Each function was tested to ensure a serviceable framework had been developed.
From the table 1-3 the various capabilities of the developed laboratory-based prototype are highlighted as they've been tested. Since the DPL/CPS has been applied to a small-scale analogue of construction machinery, quantitative analysis of the connection between the various components is required as the other factors will be dependent on the type of machinery once implemented on a real construction site. In this prototype, the most vital aspect of the connection is the latencies as listed below: • AR controller latency: ~0.25 seconds • Sensor data -AR display latency: <0.1 seconds • DT latency (Physical model motion -3D model animation): 0.5-1.0 seconds The MPU-6050 accelerometer/gyro-sensor data were not used for the Digital Twin due to latency issues and causing the 3D model to twitch continuously due to the sensitivity of the motion capture device. The HC-SR04 ultrasonic distance sensor's real-time readings were also not utilized for the DT; instead using pre-defined distances to display hook motion in steps. This is also because of the sensitivity of the sensor that picked up the swaying due to the hook cord. The log file recorded all data from the accelerometer, distance sensor as well as input commands as soon as the server started.

Conclusion
A DT and AR-based CPS for construction machinery can have multiple applications. Through this research a framework for such a CPS can be established by developing the DPL between MCU and IoT devices. The implications of this research are: • a Digital Twin that utilizes standardized BIM formats so it can be retroactively injected into currently implemented systems of smart design and construction management • Cyber-Physically integrated machinery that can be remotely operated has many implications from improved safety to operator training and greater accessibility • Gateway for implementation of smart technologies in construction projects Current literature on CPS focuses on using IoT technologies for concepts like smart design, visualization of structures and structure health monitoring. This research expands the horizons for integration of smart technologies in a construction site by implementing them in the realm of construction machinery operations.
Cost-effective methods of retrofitting construction machinery to adapt to the proposed framework would make the prospect of Construction 4.0 more appealing. This research was limited by its nature of being a laboratory prototype. Hence, concrete quantitative analysis of the extent to which the implementation of a construction machinery CPS on-site in a real-world scenario could not be conducted. Another limiting factor is the choice of construction machinery being a tower crane, which means translational motion of equipment such as dozers, excavators, etc., in a dynamic construction site could not be explored. This research may be further expanded upon by developing a proprietary MCU sensor/actuator net package to decrease operation lag/latency as well as exploring dedicated connections in place of open Wi-Fi.

Disclosure statement
No potential conflict of interest was reported by the authors. Program. After receiving the degree, he worked as a senior researcher at Korea Institute of Construction Technology (KICT), and conducted research projects in the areas of construction technology and governmental programs encouraging technological advancement of the AEC industry.

Funding
Song Jinwoo received his master's degree from Sungkyunkwan University's Department of Convergence Engineering for Future City. His work has focused on Cyber-Physical Systems, smart manufacturing, semiconductor factories, AR/VR/MR technologies and smart construction management. Seojoon Lee received his master's degree from Sungkyunkwan University's Department of Convergence Engineering for Future City. He has worked on developing algorithms and systems for Cyber Physical Systems for manufacturing facility maintenance, field information visualization and sharing using AR/VR/MR, smart safety for temporary structures, and construction management technology for off-site construction based precast concrete housing.