Relating Driver Behaviour and Response to Messages through HMI in Autonomous and Connected Vehicular Environment

Abstract Mental, physical, and emotional workload is the main source for the driver’s stress, fatigue, and distraction, which affects the driving tasks performed by the driver in terms of responding to the surroundings and the delay in reacting to events. Driver behaviour is an uncertain element within Intelligent Transportation Systems (ITS). Thus, this work presents a mathematical model that accounts for three main elements in ADAS-assisted vehicles, autonomous vehicles, connected vehicles, and autonomous and connected vehicles. The model accounts for processing times by the On-Board Unit (OBU), driver time, which covers both perception and response times, and communication time, which covers exchanged messages between vehicles and/ or infrastructure. The model shows two distinct behaviours, power law trend in the case of ADAS and exponential trend in the case of autonomous and connected vehicles, with both power and exponential interacting in the case of connected vehicles. In all considered scenarios, the mathematical model presents a dynamically interacting usability function that contains probability, Gaussian interpolation, and value limiting functions. The work also shows the effect of each time component on the usability, and hence the effectiveness of avoiding incidents. The model helps in optimizing both HMI, electronics, and communication designs. Driver time, in the case of ADAS, connected vehicles, and under rare conditions in autonomous and connected vehicles is critical, hence HMI optimization is important even though in autonomous and connected vehicles is not always needed. The work shows that the best driving usability is in the case of autonomous and connected vehicles.


PUBLIC INTEREST STATEMENT
Vehicles with driver assistance technologies (including those already in use on the roads) have the potential to reduce accidents and levels of injury, hence saving lives. Autonomous vehicles have sensors and electronics that react in a shorter time compared to human drivers, which would greatly reduce driving errors leading to accidents, especially accidents that result from driver's mental and physical conditions, such as distraction, stress, and fatigue, among others. Researching into driver-vehicle interaction through Human-Machine Interface (HMI) is important as the way HMI is designed can greatly affect driver response to messages received by the vehicles and can lead to accidents if not properly thought of. Thus, the combination of intelligent systems, high-speed electronics and sensors, low latency communication systems, and optimum HMI will altogether help in safer driving. Autonomous and connected vehicles have a promising future, as they can greatly help in many parts of our daily lives from accident prevention to increase mobility and environmental preservation.

Introduction
During driving, specifically at different speeds, the driver needs to interact dynamically and in realtime with traffic events. The driver's reaction to events becomes more critical as speed increases, with the driver's reaction time becomes longer compared to the required reaction time to take an action, such as steering away, braking, accelerating, decelerating, carrying out a manoeuvre, thus, avoiding a particular incident. Various reaction times for both visual and audible signals and messages are reported in the literature, which depends on the available time for the driver to realize, respond, and react to an event (Babić et al., 2020); (Papantoniou et al., 2019); (Adamidis et al., 2020); (W. W. Li et al., 2021).
The time and type of decision made by the driver in facing a traffic event determines the outcome to be positive or negative and if negative its severity. Such a time is not constant, as there are many variables contributing to its length, such as driver's age, mental condition, circumstances, and speed of the vehicle, type of event, weather, and physical condition of the driver, vehicle capabilities, and road topology, among others.
A critical and variable element of road safety is the behaviour of the driver under different conditions. The quality and amount of information provided to the driver with the driver's mental, physical, and psychological condition will affect the acceptance and type of response of the driver to such messages. Thus, the whole mechanism of a driver reacting to an event is very complex as it occurs in a dynamic, real time environment with human and mechanical factors interrelated. Thus, road safety is taking a high priority in transportation organizations, where many methods are proposed to enhance the safety and mobility (Andersson & Peters, 2020); (Droździel et al., 2020); (Hichim et al., 2020); (Kim et al., 2018); (Kuang et al., 2015); (Liu & Wang, 2020); (Sohrabi, 2019); (Szydłowski et al., 2021); (Zhuk et al., 2017); (Zontone et al., 2020); (Žuraulis et al., 2018).
Automated and connected vehicles are increasingly becoming a practical reality. The technologies for autonomous and connected vehicles with various sensors and Advanced Driver Assistance Systems (ADAS) are proving to be effective technologies to reduce or even avoid traffic incidents.
There are many advantages in the technologies implemented within the autonomous and connected vehicles when applied in the field of Intelligent Transportation Systems (ITS) and in particular within a smart city environment, such as improving safety, reducing travel time uncertainty, reducing the need for infrastructure expansion, increasing productivity and efficiency, enhancing mobility, reducing fuel consumption, and increasing energy efficiency, reducing environmental effects, and influencing vehicles design. In addition, autonomous and connected vehicles, will affect the way transportation is managed and will have first, second and third-order effects on vehicle insurance, health insurance, ethical and legal issues related to incidents and accidents. Such advances in vehicular technologies also affect other industries, such as the communication industry with service providers, fuels consumption and suppliers, design of commercial areas and parking spaces, among others (Romano et al., 2020); (Trindade et al., 2020).
Highly automated vehicles will have an important contribution to people's mobility, as they will assist drivers in carrying out stressful tasks and help in circumstances whereby the driver is distracted. Thus, ensuring safety and more efficient mobility. There are circumstances when drivers are in need for assistance, be it the driver is overloaded or underloaded. Driver overload arises when the driver has to carry out different tasks at the same time as paying attention to the surrounding traffic and vehicles. Such stressful conditions may result in degraded performance for the driver.
Driver underload can occur due to driving under non-stressful conditions or driving on highways for long distances, which might lead to drowsiness and losing attention and control over the vehicle. Normally, drivers carry out optimum driving activity that result in a balanced and focused performance, with very little distraction or confusion.
The driver will always be a part of the control loop in the autonomous and connected vehicles environment, which is both legally and socially required. Thus, the driver can always pass over the control to the vehicle under certain conditions, with the vehicle passing it back when the conditions of automation are not met. The automated and connected vehicles operate on three levels: (1) Warning (2) Support (3) Intervention Such levels are affected not only by the vehicle level of automation but in the case of connected vehicles is also affected by received messages, with the driver always kept in the control loop. As this is the case, then, it becomes important and rather critical to concentrate on the Human-Machine Interface (HMI) in the vehicles, which hold a bidirectional relation between the driver and the vehicle (Carsten & Martens, 2019); (Charissis et al., 2021); ; (S. S. Li et al., 2019); (Lotz et al., 2020); (Nguyen et al., 2021); (Rukonic et al., 2021); (Yang et al., 2018).
Such bidirectional relationship and connectivity require careful consideration in designing HMI that is reliable and easy to use with the ergonomic interface, which helps to achieve the following features: (a) Clear and easy to read.
(b) Simple to interpret displayed messages.
(c) Easy to access and operate.
The above qualities, if applied should greatly assist in enhancing safety and decreasing distraction and driver stress. This is supported by applications associated with the HMI that prioritize message display as a function of importance and criticality in order to further reduce driver stress levels and distraction.
Monitoring driver behaviour and its effect on traffic incidents and accidents is critical in the design consideration of HMI. Also, monitoring driver behaviour provides important information regarding the driver's mental condition, particularly in relation to attention and distraction and the extreme cases of stress and drowsiness.
Driving for long hours can reduce driver's attention by a large factor due to loss of focus, which increases the probability of accidents as the driver will be slower in reacting to events or in interacting with the vehicle. Also, stressful driving can have a similar effect due to loss of concentration. The way driver status is assessed directly and indirectly is by using both sensors, vehicle movements, and physical driver behaviour, which considers the following: Drowsiness level (AlShalfan & Zakariah, 2021); (Anani & Appiah, 2019); (Baker et al., 2021); (Cunningham & Regan, 2018); (Kateřina et al., 2019); (Lee & S, 2012); (Robbins & Fotios, 2021); (Stanisław Jurecki et al., 2017); .
Eye activities through blinking and movement: (1) Driver performance through carrying out tasks.
(3) Dynamic response to control buttons.
HMI-based algorithms track and compute driver behaviour pattern for every trip and continuously compare it to the latest one taking into account the current driving conditions. Thus realizing any up normal behaviour of the driver.
In this paper, modelling of the interaction between the driver and the provided HMI, and its effect on the usability of such advanced designs in autonomous and connected vehicles is presented. The work takes into account driver realization and response times, message processing time within the vehicle On-Board-Unit (OBU) that sends messages via HMI to the driver, and the communication time that takes a message to reach a vehicle in case of connected vehicles. This work aims to contribute towards linking driver reaction to sensors, electronics, and communication technologies, interfaced with the driver through HMI. Thus, enabling more efficient interface to be considered to eliminate any negative effects of HMI on the driver reaction time and delay in response, thus contributing towards safer driving.

Methodology
The effect of driver characteristics covering perception and reaction times and their subsequent effect on the driver reaction to messages through an HMI interface is critical in determining the consequences of a driver's actions and affects the response of other connected vehicles to a particular vehicle behaviour, with marked effect on any action taken by an autonomous vehicle. Thus, for connected and automated vehicles, a time response mathematical model needs to be developed. This work is based on the hypothesis that consideration of response times for driver, sensors, electronics, and communication system together with HMI design will greatly aid accident reduction and overcome driver limitations.
The main elements to consider in the mathematical model reflecting the hypothesis in this work are: (1) Perception time (T realize ): The time required for the drivers to realize that an action must be taken, which is the time-lapse from the moment a message is displayed and made visible to the driver to the instant the driver realizes that an action is due. Many factors affect both the time to display the message, the time the driver takes to pay attention to the message, and the realization of the consequences of such message.
(2) The response time (T response ): The time consumed to take an action, which is affected by the mental and psychological status of the driver, age, experience, physical health, required reaction time to avoid a problem, traffic conditions, weather, and HMI design.
(3) Internal processing time for the vehicle electronics and HMI display (T processing ): The time taken to process a message inside a vehicle from OBU through HMI to be displayed.
(4) External transmission time (T comm ): The time taken for a message to be received by the vehicle OBU if the vehicle is connected and communicating to infrastructure or other vehicles.
Based on the previous arguments and assumptions, a mathematical model can be established, which reflects the effect of driver behaviour on the outcome of a vehicular scenario on the road.
The usability model introduced in this work is mainly based on the principles of utilization of exchanged data between devices over a communication network, where the original model considers propagation delays and transmission delays in a network, together with probability of failure of data exchange. The usability model is essential in order to characterize the benefits of the different mechatronic systems appearing in new vehicles and to determine a level of confidence in these systems as the key to all of that is driver safety, mobility, vehicle reliability, and journey comfort. Also, to highlight the effect of the HMI design and sensors and electronics quality in determining the effectiveness of using advanced vehicles, and above all, realizing that many accidents can be avoided if a reliable, vehicular control system is available to overcome driver human limitations.

Usability model
Assumptions: (2) Equations (1) and (2) include the Gaussian interpolation function, which is used to add an element of intelligence. It also acts as a comparator enabling critical decision-making through interpolation. In addition, it is used to enhance HMI design by substituting different values for T driver and T processing using experimental and simulation values, in order to achieve optimum response times that contribute to higher safety and usability. Also, as vehicular design and technology advances with different materials, sensors, electronics, and higher communication bandwidths with faster transfer rate, different values for T maxp and T maxd can also be tested using Gaussian interpolation together with HMI designs and driver responses, leading to different usability values that are correlated to advances in automotive and communication technologies. Moreover, the dynamics of the control achieved using Gaussian interpolation function by employing its reach or spread parameter together with its interaction under different scenarios and conditions with the time response ratio, as presented in the following subsections, optimizes the general controls of the vehicle systems with respect to real-time changes.
The previous is more important especially when considering different drivers' ages, driving experience, circumstances, and scenarios under which drivers have to interact with HMI.
For more parameters affecting the driver and processing times, a weighted summation of the parameters using Gaussian interpolation can be carried out in order to enable a more detailed analysis of the causes behind the response time values as a function of different HMI designs, various processing speeds, and communication bandwidth, different driver status and conditions.
The following usability cases are discussed assuming lumped time responses in order to concentrate and focus on the effect of processing times and driver times on usability for a specific HMI design.

Vehicles with advanced driver assistant systems (ADAS)
The sensory systems in the vehicles detect certain road conditions and pass a message through HMI to the driver. The effect or usability of the messaging system and the influence of the driver on the situation can be expressed as in Equation (3). Where, T maxd and T maxp : Maximum reaction and processing times within assumed safety limits of human and vehicular responses. δ and λ: Spread of the Gaussian interpolation function.
The factor of 2 allows for both the time to deliver a message through HMI and the time the driver executes a command through HMI.
Thus, Equation (6) becomes Equation (7) R 1 ¼ The assumption that the difference in maximum times and actual time is markedly large is a rare case and will mean a major human tragedy and /or a major technical failure, which statistically has a very small probability of occurring. So, this work concentrates on the normal operational activities for both vehicles and driver in order to have one variable under consideration, which is HMI design and its effect on driver-vehicle interaction in terms of response and reaction times for both sensors and their electronics and associates signal processing together with human realization and reaction times.
There is a major technical problem with the vehicle sensors, as processing time is too large (driver time cannot be less than processing time). Also R 1 ǂ1, as driver time cannot equal processing time.
Where; R threshold : The maximum waiting ratio for driver response, beyond which an emergency is declared as driver will be facing health problems.
Equations (3) to (7) indicate that for usability to reach 1 or 100%, then the driver time (realization and response times) must decrease and reach minimum. In order to further emphasize effect of driver time and/or processing time with specific HMI design on the usability function, which indicates efficiency in avoiding accidents by taking action within acceptable time limits, a multiplication time factor is added that indicates accumulative waiting time for the driver to receive a message, which also affects driver time as shown in Equation (8).
Where M indicates additional delay factor in the overall process of receiving message. This is also reflected in Equation (9).
From Equations (8) and (9), it is clear that any technical issues will reduce usability. At M = 1, usability is controlled by Equation (7).
To further enable intelligent algorithms in predicting usability and its consequences on either taking action and to allow for technical issues to be taken into account with their processing times, and to avoid traffic problems, probability factor needs to be included (Equation (10)) in Equations (8) and (9), as shown in Equations (11) and (12) by substituting the function in Equation (8) Usability ¼ 1 À P proces sin g delay ð Þ À � 2 T proces sin g T driver þ 2 T proces sin g À � ! (11) Equations (11) and (12) show that as the probability of processing delay increases, usability decreases as it adds to the overall time for message display, hence an action by the driver will take longer.

Connected vehicles
In this case another time element needs to be considered (T comm ). This time element accounts for message exchange timing between vehicles (V2V) and/or between vehicles and infrastructure (V2I). Thus, the assumptions are: Thus, Equation (3) becomes Equation (16).
Equation (16) is simplified to give Equation (17). Where; T maxc : Maximum communication time within assumed safety limits.
β: Spread of the Gaussian interpolation function.
Based on the presented conditions, Equation (17) can be further simplified by dividing by 2T processing *ϕ processing, resulting in Equation (18).
Then, R 2 is given by Equation (19) and With R 1 is given by Equation (7) and From Equations (18) and (19), it is evident that for effective usability and incidents avoidance, T comm and T driver and their ratio (R 2 ) need to be as small as possible.

Autonomous vehicles
In this case, the usability function is related to the automation level.

Assumptions:
I. During manual driving and during handover.
However, autonomous vehicles need connectivity with GPS and other infotainment sources. So for all practical reasons, we can ignore this scenario.

Autonomous and connected vehicles
In this case, the usability function is related to the automation level and connectivity. Assumptions: (I) During manual driving and during handover.

Results
Tables 1 to 6 present a comparison of usability function for the scenarios presented. The scenarios are: (1) Vehicles with ADAS.
The testing R 1 and R 2 values appearing in the tables, selected according to the assumptions made earlier and the subsequent equations developed, and conditions are: 1. Vehicles with ADAS: a. R 1 < R threshold (To enable setting maximum waiting time for driver response to HMI message (within safety limits), otherwise it is assumed that the driver has health problems).
b. R 1 ǂ 0 (ADAS does not behave as level 5 automation). c. R 1 > 1 (Otherwise, it is assumed that there is a major technical problem with the vehicle sensors). d. R 1 ǂ1 (Driver time cannot equal processing time).

Connected Vehicles:
a. R 1 > 1 (Otherwise, it is assumed that there is a major technical problem with the vehicle sensors).
b. R 1 ǂ1 (Driver time cannot equal processing time).
c. R 2 < 1 (The ratio of communication time to processing time should be small).

Autonomous and Connected Vehicles
R 2 < 1 (The ratio of communication time to processing time should be small). Figures 1 to 3 show usability as a function of both driver time to processing time ratio (R 1 ) and communication time to processing time ratio (R 2 ). From the plots, the following is evident:

Discussion
Assuming fixed processing time (T processing ): (1) As probability for driver response time (T driver ) increase, the usability will and effectiveness of interaction between the driver and HMI will decrease, thus increasing probability of accidents and reducing the safety margin. This is presented in Equations (39) and (40), which show power law trend.
(2) As probability for communication response time (T comm ) increases, the usability will also decrease due to the extended length of time for communicating messages inter and intra vehicle, which will give the driver less time to interact with the HMI interface. This is presented in Equations (41) and (42) for connected vehicles and Equations (43) and (44) for autonomous and connected vehicles.
(3) Connected vehicles will also be affected by Equations (39) and (40) that include the driver time response.
(4) As the values of R 1 and R 2 increase, usability will decrease due to the combined effect of both driver time (T driver ) and communication time (T comm ).
From Figure 1, the usability model as a function of probability of processing delay can be expressed as in Equation (39).
Substituting for R 1 from Equation (6) results in Equation (40).  Substituting for φ and ϕ results in Equation (41), which represents a generalization equation with Gaussian interpolation component.
As previously mentioned, when the Gaussian interpolation function reaches the value of 1, Equation (41) will reduce to Equation (42) with the coefficient α is a function of probability.
Thus, the Gaussian interpolation dynamically and intelligently controls the usability, but when the response times (T driver ,T processing ) are equal or greater than the maximum reference times (T maxd , T maxp ), then the usability drops significantly with a proposed condition that a warning should be issued to avoid dangerous scenarios and safety compromise.
From Figure 2, the usability model as a function of probability of processing delay can be represented as in Equation (43).  (44).
Substituting for γ and ϕ results in Equation (45), which represents a generalization equation with Gaussian interpolation component.
From Equation (45), the Gaussian interpolation dynamically and intelligently controls the usability, but when the response times (T comm ,T processing ) are equal or greater than the maximum reference times (T maxc , T maxp ), then the usability drops significantly with a proposed condition that a warning should be issued to avoid dangerous scenarios and safety compromise.
When the Gaussian interpolation function reaches the value of 1, Equation (45) will reduce to Equation (46) with the coefficient κ is a function of probability.
The connected vehicles case includes the contribution of both R 1 and R 2 .  From Figure 3, the usability model as a function of probability of processing delay can be represented as in Equations (43) and (46). This scenario is dependent on R 2 and constitute part of the connected vehicles response functions that incorporate R 1 and R 2 .
Two main functional trends are observed from the presented equations and plots: (1) ADAS: Power Law (Parameter R 1 ).
In the particular case of connected vehicles, two types of functional behaviour are dynamically interacting with each other. Figure 4 compares all driving usability scenarios described by Equations (47) to (51), with clear indication that the highest usability is obtained for the autonomous and connected scenario, even though, the driver does not need to interact with the vehicle, or interfere with the driving process, still HMI design is critical and the driver needs to be able to take over under some rare occurring circumstances such as: (1) The autonomous vehicle system breaks down due to technical problems.
(2) The vehicular control system might get corrupted due to viruses or external interference, hence there should be great emphasis on vehicular security.
(3) There is a possibility that digital maps for infrastructure changes are not updated in the vehicle system, thus the vehicle can get lost or end in an accident.
(4) The driver may need to report certain information using mobile phone that appears through HMI interface. So, HMI and the way messaging is presented is important.
Thus, despite the fact the this work proves that the best arrangement is for autonomous and connected vehicles, still, HMI design and driver interaction needs to be considered.

Conclusions
The presented functions in this work are affected by the designed and implemented sensors, electronics, and HMI design. In addition the plots illustrate the marked effect such factors have on the interactive environment of Vehicle-Driver-Infrastructure, thus, affecting the driver response time, which can vary due to the driver mental, psychological, and emotional state at the time of receiving a message through the HMI.
The presented mathematical model for usability and plots, also covers indirectly the effect of routing and communication channel status, including, bandwidth, frequency, and traffic density on the response of either the vehicle or driver to received messages through connectivity with other vehicles or infrastructure.
It is proved through mathematical modelling that the best performance and most usability is achieved when the vehicle is in its autonomous level 5 and connected, as both communication time and driver time are eliminated, except for rare conditions, whereby vehicular control system failure occurs due to reasons explained earlier in the paper.
The work also shows mathematically that usability can be optimized better and more dynamically with intelligent content, when the Gaussian interpolation function is included, leading to an interaction of three functions: (1) Probability function.
It is important for future work to correlate specific HMI designs, processing and communication times, together with driver response time in order to further enable design and manufacture of optimal vehicles. This can further be verified by applying Equations (6), (17) and (35) simulate rare and extraordinary conditions and also establish interactive dynamics between the three parameters used to characterize usability, which in extreme cases can approach very low values and under good design criteria should not be reached. Also, using Gaussian interpolation function with a spread or reach factor, will enable more control over the dynamics inter-influence that occurs as the three parameters (T driver ,T comm , and T processing ) interact with each other as a function of vehicular dynamics, communication and routing, sensors and electronics, and human factors. Despite the fact that autonomous vehicles are being developed to a high intelligence level, still ADAS vehicles and connected vehicles are more available and in use due to the difficulty of gaining drivers' trust and confidence to handing control totally to vehicles. In addition more complex security algorithms need to be developed, in order to give higher confidence in autonomous and connected vehicles.

Funding
This work was supported by the Self funded;

Disclosure statement
No potential conflict of interest was reported by the author(s).

Citation information
Cite this article as: Relating Driver Behaviour and Response to Messages through HMI in Autonomous and Connected Vehicular Environment, Mahmoud Zaki Iskandarani, Cogent Engineering (2022), 9: 2002793.