Improvement of a depth camera installation for fall risk estimation

We propose a monitoring system for accidental falling of patients using a point cloud dataset (PCD) of the depth camera-captured images. The conventional system requires the PCD to comprise images showing the bed top view. Consequently, the depth camera installation location is restricted. Therefore, we propose a new system with a new PCD generation method. This system enabled PCD correction, corrected dataset division, human location estimation, and fall risk calculation. The Microsoft Kinect sensor was employed as a depth camera in the validation. We demonstrate that the pitch angle can be set between and within the depth camera measurable range to image the subject's movements. These images were utilized in risk estimation. Further, the horizontal distance from the side edge of the bed and the height from the ground were greater than 1.0 m. Under these conditions, the PCD can be corrected into a bed top view dataset, and can help in estimating the fall risk.


Introduction
Fall accidents are one of the most serious problems at medical sites [1,2]. Many inpatients, such as the elderly, have considerably low physical strength. They can easily break their pelvic and leg bones or suffer from head injuries due to fall accidents [3]. Certain reports have shown that fall accidents in inpatients account for approximately 30% of the total number of medical error events [4]; furthermore, more than 60% of these accidents occur at the bedsides of patients [5].
Sick inpatients are requested to call a nurse when they leave the bed. However, most of them attempt to stand by themselves owing to diminished cognitive functions or disregard for their high fall risk. Therefore, clip-type and/or floor mat sensors have been introduced in many hospitals to mitigate fall accidents. However, not all fall accidents can be prevented because clip sensors are often removed deliberately from the bodies of inpatients. In addition, floor mats cannot prevent a fall accident in advance because they can only detect the fall risk after the patient stands.
To address the above-mentioned reasons, several studies have sought to introduce strategies that prevent or mitigate fall accidents. Inoue et al. [6] used homography transformation on the acquired colour images to monitor the behaviour of an inpatient. Consequently, they implemented image normalization and bed location detection. However, the system was unsuitable in the darkness because it included a colour sensor. Enayati et al. [7] estimated the postures of a patient by applying principal component analysis and a feedforward neural network to the pressure values obtained using hydraulic transducers. However, these sensors had to be placed under the mattress, causing discomfort to the patients. In addition, the breakdown rate of these sensors was high because they were contact-type sensors. The company, Ocuvera [8], has developed a product for fall prevention that estimates the fall risk based on the depth map of an inpatient. This system was located at a specific distance from the patient legs. However, the system was not installed in the rooms where other patient beds were located in this position.
In contrast, a fall risk estimation/assessment system [9] was constructed to detect the conditions of an inpatient on the bed using a point cloud dataset (PCD). A PCD consists of a set of points with three-dimensional spatial coordinates that are measured using a depth camera. In addition, the fall risk is an indicator of the degree of danger to a patient, which increases when the patient is under dangerous conditions. Several depth cameras can be used in a dark room because they adopt infrared light; hence, the sleep of inpatients is not disturbed by the lighting equipment in the room. However, the conventional method presents a strong constraint pertaining to the depth camera location, i.e. the depth camera must be placed at a particular position above the bed. Generally, there are certain panels and pedestals around the bed for the installation of medical instruments. The types of equipment and their locations vary depending on the clinical department and inpatient, even at the same hospital. Consequently, sometimes it is not feasible to install the conventional system onto the beds of the inpatients.
To solve the above problem, we attempted to correct the PCD with the information from an accelerometer and the side edge of the bed prior to risk estimation. By implementing this method, the constraint pertaining to the depth camera location can be relaxed; moreover, this system can monitor an inpatient even when the relationship between the camera and bed positions is being modified. Subsequently, we developed a new prototype fall risk estimation system that included these functions and performed the corresponding experiments in an actual environment [10]. However, this study did not refer to the valid range of the proposed system; thus, it is unclear to what extent the installation constraints tend to be relaxed. To clarify these aspects, in addition to the conventional verification, the constraints related to the depth camera installation position and photographing direction are experimentally derived and discussed in this paper. Figure 1 shows the underlying concept of the proposed method. The proposed system estimates the fall risks of inpatients in four steps (a) to (d). The details of each step are as follows. Figure 2 illustrates the underlying concept of the method used to calculate the parameters required for the correction of a PCD. This method converts the original PCD into a bed top view dataset via two rotations. Steps 1 to 3 are related to the first rotation, while Steps 4 to 6 are related to the second rotation.

[Step (a)] correction of the PCD
(1) Measure the gravity vector p a using an accelerometer attached to the depth camera. Here, p a in Equation (1) represents a quaternion [11], and x a , y a , and z a are the elements of the X-, Y-, and Zaxis in the quaternion, respectively. In addition, i, j, and k are complex numbers used to represent the quaternion.
p a = ix a + jy a + kz a (1) (2) Obtain the gravity vector p a when the depth camera looks precisely down at the bed. (3) Defineq a as the rotation quaternion for p a (Equation (2)).
q a = q a0 + iq a1 + jq a2 + kq a3 (2) Assuming that the conjugate quaternion ofq a is q a * , then Equation (3) can be used to enforce a rotation from p a to p a .
p a =q a p aqa * Place a restriction q a = 1 and derive a set of quaternion elements q a0 , q a1 , q a2 , and q a3 that satisfy the equality in Equation (3) using p a and p a . Here, in case a fall prevention fence is present around the bed, configure these points on the upper side of the fence. Define p b as a directional vector on the straight line passing through these points (Equation (4)). It should be noted that p b is expressed by a quaternion, which has been transformed using the rotation quaternionq a .
(5) Define a directional vector p b as a quaternion. Then, it is necessary for p b to fulfil the condition that it is parallel to the X -or Y -axis, which are the resulting axes following the correction of the PCD ( Figure 2). For example, when the directional vector is set parallel to the Y -axis, then p b = j. (6) It is assumed that the rotation quaternion of p b isq b . Similar to Equation (3), place a constraint q b = 1 and calculate each item ofq b using p b and p b .
Following the above procedure, the rotation quaternionsq a andq b can be obtained for the correction of the PCD. It should be noted that when one of the bed vertexes is defined as the coordinate origin, the  three-dimensional spatial coordinate is translated using Equation (5) after the rotation. Then, t x , t y , and t z are the extents of parallel translation for each corresponding axis, and x c , y c , and z c are the three-dimensional spatial coordinates after the translation.
It should be noted that this transformation is generally achieved when the depth camera is installed; therefore, it is assumed that there is no patient or comforter on the bed.

[Step (b)] division of the bed top view dataset for robustness
Generally, the PCD corrected withq a andq b has a large number of three-dimensional points; therefore, the dataset is easily affected by the differences in the arrangements of the rooms and figures of the inpatients. Consequently, we attempted to improve the robustness by dividing the three-dimensional spatial coordinates into certain sections. The concept of this methodology is shown in Figure 3 and the details are as follows.
(1) Divide the corrected three-dimensional spatial coordinates describing the data on the bed (if necessary, include the data outside the bed) into a grid pattern, and create multiple sections. The division is performed on all the axes. Here, the Z-axis is divided into three layers to distinguish between three types of representative postures: lying down, sitting, and standing. The layers are referred to as the upper layer (UL), middle layer (ML), and lower layer (LL), respectively.
as the separation thresholds (where x s , y s , and z s are the section numbers). In addition, let n o (x s , y s , z s ) be the number of three-dimensional points in the ranges of d X Obtain the number of threedimensional points for each section.
(3) The number of points n o (x s , y s , z s ) is relatively sensitive to the depth camera location and peripheral environment. Thus, using Equation (6), multiply n o (x s , y s , z s ) by v mul (x s , y s , z s ) and subtract v sub (x s , y s , z s ) from it to exclude these influences. Then, v mul (x s , y s , z s ) and v sub (x s , y s , z s ) are the correction coefficients for smoothing.

[Step (c)] human location estimation
The method used for estimating the human location from the grid pattern sections is detailed below.
(1) For each Z-axis layer, extract the section with the maximum number of three-dimensional points in n(x s , y s , z s ). Then, the maximum values of UL, ML, and LL are defined as n max (2), n max (1), and n max (0), respectively. (2) To identify a layer where a patient is located, compare whether the maximum value is greater than a predefined threshold for each layer. Here, let n th (2), n th (1), and n th (0) be the thresholds of UL, ML, and LL, respectively. (3) Select the target layers using Equations (7)- (9).
It should be noted that "No layer focused" in Equation (9) denotes that there is no target layer because the object, which should convey the presence of a human, does not exist (i.e. there is no human) in the measurement range.
Focus on UL (n max (2) ≥ n th (2)) To Equation (8) (otherwise) Focus on ML&LL (n max (1) ≥ n th (1)) To Equation (9) (otherwise) (4) When the target layers are ML and LL, add the number of three-dimensional points in the sections on ML n(x s , y s , 1) and LL n(x s , y s , 0). However, when UL is selected, n(x s , y s , 2) is employed without any change to estimate the standing posture more accurately.
n (x s , y s , z s ) (5) Select a combination of x s , y s , and z s with the highest value of n (x s , y s , z s ) among the sections on the target layer, and assign these section numbers x s , y s , and z s as x max , y max , and z max , respectively. (6) Calculate the centre of gravity on the X-axis g X [m] using Equation (11).
Here, g Y and g Z are derived similarly. However, it should be noted that different values of n(x s , y s , z s ), as opposed to n (x s , y s , z s ), are used when g Z is computed.

[Step (d)] fall risk calculation
In this section, we present the steps performed to derive the fall risk of an inpatient from the centre of gravity values g X , g Y , and g Z calculated in Section 2.4.
(1) Predefine the fall risk weights (FRWs) corresponding to the separation thresholds d X (3) Use Equation (13) to derive the fall risk on the X-axis r X . It should be noted that Equation (13) is a linear equation between g X and r X . However, it is possible to adopt a quadratic equation or an exponential function when the fall risk fulfils the condition where it increases as a patient moves near the outer side of the bed. r Y and r Z , the fall risks on the Y-and Z-axis, respectively, can be calculated in a similar manner.
(4) The summationr of r X , r Y , and r Z is the fall risk of the inpatient.

Experimental conditions
We evaluated the proposed method using a medical bed. A 25-year-old healthy male subject was recruited. However, any differences in age and sex should have little effect on the results. The corrected PCD is divided to improve the robustness in Section 2.3. In this step, the information indicating differences among the figures of inpatients is almost removed once the PCD is compressed into certain sections. Therefore, the proposed system can be evaluated using only one subject. Table 1 lists the experimental conditions. Here, the gravity vector was measured using an LSM330 (STMicroelectronics Inc., Geneva, Switzerland). The PCD was obtained using a Microsoft Kinect sensor [12] as a depth camera. Figure 4 shows an infrared image of the room used for the experiments. The points A and B are required for calibration and were located on the side edges of the bed. This prototype system for fall risk estimation captured and displayed the depth map upon depth camera installation. The system operator examined this map manually and configured the said points to overlap with the individual side edges. However, these points can be set automatically based on the detection of the side edge of the bed combined with the Canny edge detection and Hough transform methods [13].
The FRWs (w X x s , w Y y s , and w Z z s ) employed in this experiment are listed in Table 2. As d X x s , d Y y s , and d Z z s indicated a position closer to the outer edge of the bed, the FRWs were assigned with higher values. These values were utilized to derive the fall risks (Equation (13)).
Note: w X −1 was used for the extended UL.
Here, the minimal values of w X x s , w Y y s , and w Z z s are set to 0, while the maximal values are set to 4, 3, and 3, respectively. Thus, total the minimal and maximal values of fall risk per axis were 0 and 10, respectively. In addition, to estimate the fall risk of the subject in a standing posture more accurately, only the range of UL was extended by 0.5 m to the outside of the bed. w X −1 ( Table 2) was utilized for the extended UL.
The sequence of experiments is as follows. (1) Rotate and translate the PCD without the subject on the bed.
(2) Estimate the fall risk for various postures of the subject (lying down, long sitting, sitting sideways, and standing) after transforming the coordinates. Figure 5 shows the transformation results for the PCD. Figure 5(a,b) represent the results before and after the transformation, respectively. Here, there are four vertexes in one bed plane. It should be noted that the vertex closest to the depth camera was selected as the coordinate origin in this verification. The colour coding in these figures indicates the Zaxis distance. In addition, (1)-(3) represent the labels of the objects in the experiment space, corresponding to the wall, bed plane, and floor, respectively, in Figure 4.

Coordinate transformation of the PCD
We attempted to convert the PCD from the view where the bed plane appeared diagonal to the view where the depth camera looked down on the bed plane from directly above. Focusing on point (2) in Figure 5, we can observe that the Z-axis distance for the bed plane is variable before the transformation; however, it is constant after the transformation. The Z-axis distance of almost all the points showing the bed plane is between 0.0-0.1 m. Moreover, the bed plane is visible as a rectangle, and each side edge of the bed is approximately parallel to the X-or Y-axis. These results demonstrate that it is possible to transform the view accurately even when the depth camera is installed at an arbitrary location. Figure 6 shows the results of the fall risk estimation for the transformed PCD of each posture: (a) lying down, (b) long sitting, (c) sitting sideways, and (d) standing. Based on the results, the fall risk of each posture is in the following order: lying down (3.0) < long sitting (3.6) < sitting sideways (5.1) < standing (7.5). This order conforms to the general perception that "The fall risk is higher when patients are standing and is lower when they are lying down." In addition, from a physical perspective, it fulfils the characteristic that "The higher the position of the centre of gravity of the patient, the more likely that the patient will suffer a serious injury;" therefore, we believe that these results are appropriate.

Fall risk estimation
Incidentally, this verification was conducted without any comforter on the bed. The proposed method searches the target layers in the order UL, ML, and LL. Therefore, when the system detects postures corresponding to the UL and ML (i.e. standing and sitting, respectively), the existence of a comforter, which is normally in or under the LL, does not affect the precision of detection. Moreover, even when a patient sleeps with a comforter, the fall risk can be estimated because of the above-mentioned reason.

Target parameters for the installation constraints
Through verifications in Section 3, we confirmed that the proposed system can estimate the fall risk without fixing the positional relationship between the depth camera and bed by transforming the PCD. However, the installation constraints on the depth camera have not been discussed yet. Therefore, there may be a possibility that the system might not obtain sufficient information to estimate the risk depending on the installation location.
The installation constraints of a depth camera are mainly related to the following three aspects: (1) the distance and height from the bed, (2) the camera orientation (i.e. roll, pitch, and yaw angle), and (3) the measurement specifications of the depth camera. Then, constraint (1) depends on (3); therefore, constraint (1) can be derived from the measurement specifications of the depth camera employed in this verification. For example, the measurable range of a Microsoft Kinect sensor used in this experiment is between 0.5-4.5 m. Therefore, the constraint on the installation distance is that this distance should be 0.5-4.5 m from the measurement objects. Moreover, regarding the installation height, the measurable range is applicable from above the bed. Therefore, the constraint on the installation height is that it should be 1.0-5.0 m from the ground, as the bed height subject to this measurement is 0.5 m.
Next, regarding (2), based on the depth camera specifications, the roll angle should be horizontal. In addition, the yaw angle need not be considered because the depth camera can face toward the centre of the bed irrespective of the camera position. Consequently, when deriving the installation constraints, a target parameter that remains to be verified is the pitch angle. Therefore, we evaluate and discuss the constraints related to the pitch angle in Sections 4.3 and 4.4.

Experimental conditions
As shown in Figure 7, the depth camera was installed at the position where the horizontal distance from the UL was 0.5 m and the height from the ground was 1.0 m. This position satisfies the constraints mentioned in Section 4.1 and is the closest location to the objects to be measured. Therefore, it is difficult for the depth camera located at this position to capture all the information on and around the bed, which represents a particularly severe condition. In addition, the roll angle was set to be parallel to the bed plane, and the yaw angle was set such that the depth camera faced the centre of the bed. Table 3 shows the results of the calculated fall risk for each posture with various pitch angles of the depth camera. Here, the "Pitch" in the first column indicates that the angles become increasingly positive as the depth camera faces down, and 0 • represents the position of the camera parallel to the ground. Moreover, the NLF mentioned in the "Fall risk" column indicates that the system concluded the absence of a human (i.e. "No layer focused" was selected based on Equations (7)-(9)). The third and fourth columns list the "Degree of subject's body part existence" and "Degree of layer existence" within the angle of view (AOV) of the depth camera, respectively. In the third and fourth  Table 3. Relationship amongst the "Pitch," "Fall risk," "Degree of subject's body part existence," and "Degree of layer existence" factors.  T  T  P  T  P  T  T  P  0 3.0 4.7 6.1 7. 6  T  T  T  P  T  T  T  P  10 3.5 3.9 5.8 6. 8  T  T  P  P  T  T  T  P  20 3 Note: Here, NLF indicates that the system judged that there was no human. T, P, and F were determined based on the side-view maps presented in Section 4.3. T: The complete body part or layer exists within the angle of view (AOV); P: Partly exists within the AOV; F: Does not exist within the AOV.

Risk estimation results for various pitch angles
columns, T (True) indicates that the body part or layer is completely within the AOV, P (Partly) indicates that it is partly within the AOV, while F (False) indicates that it is not within the AOV. The elements in bold in the table represent the erroneous results of the fall risk calculation. Furthermore, regarding the "Degree of layer existence," among the four postures evaluated in this paper, both the long sitting and sitting sideways postures correspond to the ML. 1 Thus, there are two ML columns in Table 3 to indicate the correspondence between the postures and layers. Figure 8 shows a side-view map obtained when the pitch angle of the depth camera was −20 • . In this figure, the subject's body parts corresponding to each posture (lying down, long sitting, sitting sideways, and standing) are approximated by a rectangular parallelepiped. In addition, the two dashed lines represent the AOV related to the pitch angle of the depth camera, and the angle between them is 60 • based on the specifications of the depth camera. The classification of T, P, and F corresponding to the subject's body part and layer, which is presented in Table 3, was performed based on the AOV area. It should be noted that a portion of the subject's body part, which is outside the layers, is not considered in the T, P, and F classification because this is not relevant to the fall risk calculations (for e.g. the shaded area corresponding to the standing posture in Figure 8).

Installation requirements
Based on the results presented in Table 3, when the pitch angle was −10, 0, or 10 • , the fall risk increased in the following order: lying down, long sitting, sitting sideways, standing. This shows a similar trend to the results discussed in Section 3. Conversely, when the pitch angle was 20, 30, or 35 • , the lying down, long sitting, and sitting sideways postures exhibited the same trend, but NLF was obtained for the standing posture, which is an incorrect result. Moreover, when the pitch angle was −20 • , despite the subject being in the lying down posture, the fall risk was indicated as NLF. When such erroneous results occur (second column), the corresponding elements shown in the third  Table 3. The two dashed lines represent the AOV related to the pitch angle of the depth camera. column, which indicate the degree of existence of a subject's body part, are always classified as F, and it can be found that the subject's body parts exist outside the AOV.
Meanwhile, as stated in Section 4.3, only the body part in the layers is considered in this verification. Therefore, it can be clearly seen that when all the layers exist within the AOV, the body part can be measured. Accordingly, in Table 3, the F labels in the fourth column can be ignored, and it would suffice to focus only on the F labels in the third column (This is based on the fact that when the subject's body parts are classified as T or P, all the fall risk results are correct, and the layers, shown in the fourth column, are always classified under these cases as T or P). Based on these reasons, we suggest that the installation constraint related to the pitch angle be defined as: "set the angle such that the subject's body part can be captured within the AOV." Here, it should be noted that this constraint is fulfiled when all the layers are classified as T. Thus, the constraint can be defined alternately as "set the location such that all the layers are within the AOV and measurable distance of the depth camera." When operating this system, the depth camera should be installed at a location where the above-mentioned constraint can be fulfiled.

Conclusions
In this study, a fall risk estimation system was developed to monitor fall accidents in patients. We presented a method to correct the PCD and estimate the fall risk from the corrected values. As a result of the verifications, an uncertain PCD was translated into a bed top view dataset, and the fall risk was higher when standing and lower when lying down. Moreover, based on further verifications of the relationship between the pitch angle of the depth camera and fall risk estimation, we believe that the proposed system can fulfil the requirements of fall risk estimation when the subject's body parts in all the layers are within the AOV. In conclusion, by relaxing the constraint pertaining to the depth camera location and verifying its utility, the feasibility of a fall prevention system with minimal constraints and high installation flexibility was demonstrated. Note 1. In the proposed method, it is assumed that UL, ML, and LL are used to estimate standing, sitting, and lying down postures, respectively.