Estimation of grip strength using monocular camera for home-based hand rehabilitation

ABSTRACT Grip strength exercises are commonly used rehabilitation methods for recovery of hand function. They are easy to perform even without the direct support of a healthcare professional. However, without objective feedback, the patient may not be fully engaged in the rehabilitation process. To solve this problem, we developed a system for measuring grip strength in real time using a soft ball and a monocular camera. The system estimates the grip strength using the modelled relationship between the finger joint angles extracted from the camera image and the person's grip strength. A patient can get the feedback as numbers or movements displayed on the screen. Experimental results showed that there is a correlation between the finger joint angles and the air pressure of a ball when squeezed. The average estimation error was 16.1 hPa, and the average measurement range was 100–230 hPa. The estimation error was about 12% of the measurement range. They also showed that there is a correlation between the air pressure of a ball and the applied force.


Introduction
Hands and fingers play an important role in performing such tasks as grasping and operating objects in daily life. Hand injuries such as fractures and symptoms such as paralysis can weaken the hand muscles, reduce the range of finger motion, and reduce reaction time, preventing the person from moving his or her hand as expected. This decline of hand function can prevent the person from living comfortably. Therefore, rehabilitation is often prescribed for improving hand function.
There are various types of hand rehabilitation, such as electrical stimulation [1], mirroring [2], object-based training [3], and virtual reality-based rehabilitation [4]. These methods are performed in accordance with the condition of the hand. The commonly used methods in clinical practice are grip strength exercises such as squeezing a flexible object (often a ball or cylindrical object). A soft ball is generally favoured as it exerts less stress on the hand and fingers. By squeezing such an object, the patient can exercise the finger joints and strengthen the muscles. Repetitive training like this exercise is effective for the improvement of active movements, especially for paralyzed patients [5]. Finger flexion exercise is also effective for muscle activation [6,7]. In addition, such exercises can be easily performed in a home environment as they do not require difficult instructions or specialized equipment.
Rehabilitation in the home environment can reduce the burden on healthcare workers and the time demands on patients. However, grip strength exercises do not provide objective feedback. Moreover, patients tend to lose motivation because the exercise movements are monotonous. The use of ball-type grip dynamometers has been proposed to solve this problem [8]. These devices have built-in sensors for measuring grip strength and can communicate with a computer. A person using such a device gets feedback on their grip strength and its changes and can share the data with the doctor or the therapist. Previous research has shown that visual feedback during rehabilitation is effective [9]. Therefore, feedback in real time could help promote rehabilitation. While a ball-type grip dynamometer is effective for home-based measurement, the user has to buy a special device with built-in electronics. This may prevent smooth progress in rehabilitation.
In this paper, we present a system for measuring grip strength in real time that uses the monocular camera in a smartphone or another mobile device and a soft tennis ball (Figure 1). A mobile device widely used by the general public [10] and an inexpensive ball make this system easy to use in the home environment. The monocular camera captures an image of the hand squeezing the ball, and the finger joint angles are estimated from the image. For an accurate image of the hand area to be obtained, the ball and background must be white. In addition, users need to turn the hand holding the ball towards the camera so that the finger contour is visible to the camera. A regression model is created on the basis of the finger joint angles and ball air pressure. The ball air pressure is estimated using the regression model, and the air pressure is transformed into grip force. Experimental results showed that there is a relationship between the estimated finger joint angles and the air pressure of the ball when squeezed. In addition, we found that the average estimation error was 16.1 hPa for within-participant evaluation and that the average measurement range was 100-230 hPa. The estimation error was about 12% of the measurement range. There was also a relationship between the air pressure and the force applied to the ball.

Hand posture estimation using a camera
A camera can be used to determine hand posture without limiting the range of motion because the user does not need to wear a device on the hands or fingers. A Leap Motion controller, which can track hand or finger movement, can be used to estimate hand gestures [11]. The Senz3D [12] and Microsoft Kinect [13] controllers can track finger movement using depth information. Chan et al. developed a system for recognizing hand gestures by using a ring-style fisheye imaging camera [14]. Sinha et al. demonstrated hand posture estimation by deep learning with an RGB-D camera [15]. These hand posture estimation methods require camera equipped with a depth sensor or a fisheye lens, making them expensive and unsuitable for home use.
A method using a smartphone or another mobile device with a monocular camera is more suitable for home use. A 2D image captured with a monocular camera can be used to estimate hand posture. Tomida and Hoshino proposed using a two-step database search to estimate hand posture [16]. Puttapirat and Charoenpong investigated 3D hand posture estimation using hand landmark identification [17]. Google developed a hand tracking method that achieves real-time performance on a mobile phone [18]. These hand tracking methods require that the hand should not be covered by an object so that the palm can be detected. They are thus not suitable for estimating hand posture when the hand is squeezing an object. Cao et al. proposed using the estimated relationships between feature points and detected joint parts for 2D posture estimation of the human body [19]. This method is targeted for use on a desktop CPU-GPU system, so it is not suitable for mobile devices. In our proposed system, the finger joint angles when squeezing a ball are estimated using image processing, and the hand posture is determined from the angles. By using a real-time image processor, OpenCV, we aim to enable measurement on mobile devices.

Measurement of hand or finger movement for home-based hand rehabilitation
Several methods for home-based rehabilitation for hand or finger movement use a camera. Metcalf et al. developed a system that calculates the range of finger motion and measures finger movement using the Microsoft Kinect [13]. Grubišić et al. developed a hand rehabilitation system that uses the Leap Motion controller [20]. To enable home-based self-measurement of grip strength, several ball-shaped devices for measuring grip strength have been developed. Jaber et al. developed a grip ball that consists of pressure and temperature sensors. They showed a correlation between the pressure measured inside the ball and the force applied and demonstrated that the grip ball could be used to assess grip force [8]. Additional research has shown the validity and reliability of grip strength measurement based on the ball air pressure [21,22]. In addition, Kim and Kim developed a system for measuring finger force that uses a ball-type device incorporating a six-axis force sensor [23]. FOX Co. sells a ball-shaped device for hand training, the oriori ball, that contains a pressure sensor and a motion sensor [24].
Although these devices can easily be used in the home, they may be too expensive for some potential users. The system we developed estimates the ball air pressure not from sensors but from an image. The estimated air pressure is used to estimate grip force. By using a regular flexible ball and a monocular camera, which many users possess, at-home measurement can be done easily and inexpensively.

Healthcare using a mobile device
According to a report by the Ministry of Internal Affairs and Communications of Japan, 95% of households in Japan own a mobile device and 75% own a smartphone [10]. This shows that information communication technology (ICT) equipment is widespread among households in Japan. Smartphones are particularly attractive as they are equipped with various features such as a camera and sensors that are useful for measurement. These features can be used to monitor physical movement and physical conditions. Goodney et al. developed a system for assisting stroke rehabilitation that provides audio and visual instructions. Their Dr Droid system collects a motion trace by using the 3-axis accelerometer in a smartphone [25]. Capela et al. proposed using the accelerometer, magnetometer, and gyroscope in a smartphone as a system for recognizing human activity [26]. Ienaga et al. proposed measuring wrist flexion and extension by selecting the boundary between the arm area and the back of the hand from an image taken by the user [27]. Shin et al. investigated the reliability of a smartphone application that measures the shoulder joint range of motion and found the same level of reliability as with a conventional goniometer [28]. Measurement with ICT equipment contributes to earlier detection of disease and easier monitoring of patients in the home environment. Our aim is to build an environment in which it is easy to measure the grip strength by using the monocular camera in a smartphone or another mobile device.

Overview
When a person squeezes a flexible object, the finger joint angles change in accordance with the grip strength, as shown in Figure 2. By determining the relationship between the finger joint angles and the force applied to the ball, we can estimate the grip strength of the person squeezing the ball. As the flow shown in Figure 3, an image of the hand squeezing the ball is acquired with a monocular camera, and the finger joint angles are estimated from the image. In the learning phase, a regression model based on the finger joint angles and ball air pressure is created. In the estimation phase, the ball air pressure is estimated from the 2D image using the regression model, and the pressure is transformed into grip force.

Hand bones and joints
The thumb is composed of distal and proximal phalanges, and the fingers are composed of distal, middle, and proximal phalanges. Each proximal phalanx is connected to a metacarpal bone on the back of the hand. The interphalangeal (IP) joint is located between the distal and proximal phalanges in the thumb. The distal IP (DIP) joint is located between the distal and middle  phalanges of the fingers, the proximal IP (PIP) joint is located between the middle and proximal phalanges, and the metacarpophalangeal (MP) joint is located between the proximal phalange and a metacarpal bone.

Joint angle estimation
The user holds a ball with his or her fingers around the circumference of the ball and turns the hand towards the camera so that the finger contour is visible to the camera, as shown in Figure 4(a), and an image of the hand is captured. The ball and background should have the white color, which should not be close to the color of the hand, to enable the hand area to be easily detected. We used a white ball and a white paper as the background.

Identify hand area
To identify the hand area, the input RGB image is decomposed into the HSV (hue, saturation, and value) space. The saturation image is binarized in accordance with the threshold value using the difference between the saturation of the hand and the saturation of the other areas (Figure 4(b,c)).

Identify finger contour
To identify the finger contour, the hand area is divided into two areas, the index finger on the upper side of the ball and the thumb on the lower side of the ball. The indentation and the corresponding convex hull are acquired by detecting the convex hull in the hand image ( Figure 4(d)). The hand area is split along the borderline consisting of the centre pixel of the convex hull and the pixel farthest from the convex hull. Areas not needed for estimation are deleted, as shown in Figure  4(e). The convex hull is detected from each divided area and approximated so that it is represented by a smaller number of pixels ( Figure 4(f,g)). In this way, we obtain the approximate contour and the pixels that make it up.

Calculate finger joint angles
The finger joint angles are calculated from the pixels constituting the contour. The angle at each finger joint is acquired by tracing the contour pixels from the base of the index finger to the fingertip in a counterclockwise manner, as shown in Figure 5. If the user holds the ball with the right hand, the wrist is located on the right side of the image. Therefore, the base of the index finger, the start point, is represented by pixel S in Figure 5. The joint angles are acquired in MP, PIP, DIP order. The MP joint angle of the thumb is acquired in the same way. The base of the thumb is the point where the thumb area intersects the straight line perpendicular to the boundary line. If the user holds the ball with the left hand, the contour is traced in a clockwise manner.

Remove data that failed to accurately acquire finger joint angles
In some cases, the contour cannot be obtained accurately, so the joint angle cannot be estimated well. We have developed two processes for preventing this.
In the first process, the quality of each estimation result is judged on the basis of the distance between adjacent pixels during joint angle estimation. Basically, the contour along the phalanges is longer than the contour at a joint. If the distance from one contour pixel to the next is less than the threshold, the corresponding line may not be a line along the phalanx, as shown by the points marked in blue in Figure 5. The line is thus removed.
If the contour is too short and collapsed, or if the contour pixels cannot be acquired accurately, the first process cannot remove these data. We call these data "failed data." The second process removes these failed data on the basis of the fact that grip motion is represented by time series data. The input image is a single frame extracted from the continuous motion of squeezing the ball. Therefore, after estimation based on the dataset has completed, the failed data can be removed on the basis of the difference from each finger joint angle in the previous frame. Since the value of the joint angle changes continuously, there should be no significant difference between the preceding and following frames. If the difference from the previous frame is greater than the threshold, the estimation may have failed. The data in the frame is thus removed. Through these two processes, failed data are removed from the dataset.

Verification of accuracy of estimation method
We evaluated the accuracy of the method used for estimating finger joint angles by comparing the estimated angles with the true values measured by applying a goniometer to the images. Images of a hand holding a white ball were taken against a white background. For the thumb, the IP joint angle was measured, and 25 data sets were prepared covering an IP joint range of 20 • -55 • . For the index finger, the MP, PIP, and DIP joint angles were measured, and 25 data sets were prepared covering a DIP joint range of 20 • -60 • .
For each joint, the absolute value of the difference between the true and estimated values was averaged as the estimation error. The average estimation error for each joint angle is shown in Table 1; the overall average estimation error was 2.7 • . A previous study using a Senz3D camera obtained an average estimation error of 9.5 • [12]. We judged that the estimation method we used is reasonably accurate.
Many other methods for estimating finger joint angles have been proposed and could be applied to the proposed system (i.e. [11,12,19]). However, considering that the system runs on a smartphone, we concluded that it is a reasonable approach to estimate finger joint angles for grip motion since real-time image processing using OpenCV is possible, and accurate finger joint angle estimation can be achieved while the ball is being squeezed.

Create regression model
When a person squeezes a flexible object, the hand posture changes in accordance with the grip strength.
We thus perform multiple regression analysis on the hand posture and ball air pressure to investigate their relationship. Multiple regression analysis is a statistical method for predicting the relationship between multiple explanatory variables and one objective variable. In our study, we estimated the ball air pressure from the finger joint angles of the hand. We used a soft tennis ball (Kenko, TSOW-V). The air pressure was measured using a digital pressure gauge with a mini pump.
In the learning phase, the finger joint angles and the ball air pressure are measured at the same time to determine the relationship between these parameters, which is used to create a regression model. We capture an image of the pressure gauge and hand posture together and read the air pressure from the image. In the estimation phase, the air pressure is estimated from the finger joint angles using the regression model.

Overview
To evaluate our proposed system, we examined the two things: the relationship between the finger joint angles and ball air pressure and the accuracy of the ball air pressure estimation. We collected data on the finger joint angles and ball air pressure while the participant was squeezing the ball. We removed the failed data from the original dataset as described in Section 3.3.4. We then evaluated the relationship and estimation accuracy.

Experimental participants
Ten individuals (six male, four female; all right-handed; average age 21.3) participated. The length from the tip of each participant's middle finger to the base of the hand was measured as shown in Figure 6(a). The participants were instructed to firmly stretch their palm and fingers. The measurement results are shown in Table 2.

Collect data
The participants were instructed to tighten and loosen their grip on the ball five times without pausing. They were also instructed to • turn the hand holding the ball toward the camera so that the finger contour was visible to the camera. • make similar movements with all four fingers and thumb for each motion. • bend each finger so that it flexed.
• not to squeeze the ball so tightly that their fingers were buried in the ball.
The series of motions took 3-5 s. We video recorded the movement with a smartphone camera (Figure 6(b)). The initial ball air pressure was 100 hPa. Five sets of measurements (finger joint angles, air pressure) for each participant were acquired over about 200 frames (i.e. 10-15 fps × 3-5 s × 5 trials). By recording the hand posture and ball air pressure as the ball was being squeezed, we collected a dataset for each participant. Each video frame contained one set of correspondences between the finger joint angles and the ball air pressure at that moment. We call this set a "data point." The speed of grip motion differed from participant to participant, so the number of images in each video was uneven when each video was separated at the same frame rate. Therefore, we separated the videos of the participants whose grip motion was slow at 10 fps and those of the participants whose grip motion was fast at 15 fps. Table 2-1 shows the total number of data points for each participant. We visually reviewed the estimation results and counted the number of failed data points. As defined in Section 3.3.4, failed data are data that are unable to acquire the contour pixels accurately. Table 2-1 also shows the number of correct data points and failed data points in the original dataset for each participant. Nine of the 10 participants had failed data points in their dataset. They were removed as described in Section 3.3.4 before the regression model was created.  Total data  188  272  184  251  193  200  227  223  223  288  in the original dataset  Correct data  116  258  159  194  174  200  160  176  169  266  Failed data  72  14  25  57  19  0  67  47  54  22  (2) Number of data points  Correct data  113  254  158  189  152  199  129  178  139 Table 2-2 are the data that could not be removed on the basis of the difference from each finger joint angle in the previous frame. This means that not all of the failed data could be removed from the dataset. However, 88% of the failed data points were removed from the original dataset. At this process, 6% of the correct data points were also removed.

Evaluate relationship between finger joint angles and ball air pressure and estimation accuracy of ball air pressure
Using the dataset with 88% of the failed data points removed (Table 2-2), we evaluated the relationship between finger joint angles and the ball air pressure and the accuracy of the estimated ball air pressure. Note that the dataset contained a small amount of failed data, as shown in Table 2-2. We performed two evaluations (within-participant and cross-participant) to examine whether the user's own dataset was needed for the estimation or whether the estimation could be done with a pre-prepared dataset that did not include the user's own data.
For the within-participant evaluation, we performed five-fold cross-validations using the five sets of measurements for each participant and calculated the estimation error, i.e. the absolute value of the difference between the air pressure measured with the air gauge and the estimated air pressure. As a metric for the goodness of fit for the regression model, we used the average coefficient of determination ( R 2 ) acquired from the regression models for each training dataset. Table 3 shows the estimation accuracy for the withinparticipant evaluation. The average R 2 was 0.79, the average estimation error was 16.1 hPa, the average standard deviation was 13.8 hPa, and the average error rate was 12.2%. The estimation error by air pressure band by participant is plotted in Figure 7. The horizontal axis shows the air pressure measured with the digital pressure gauge. The vertical axis shows the estimation error.
For the cross-participant evaluation, the 10 participants were divided into two groups of five people each. Within the group, we then created a regression model from the dataset of four participants and used it to estimate the ball air pressure for the remaining one. One set was randomly selected from the five sets of measurements for each participant, and a total of four sets were used as training data for creating a regression model. We then calculated the estimation error, the absolute value of the difference between the actual measured air pressure and the estimated air pressure. We used the average R 2 to measure the goodness of fit for the regression model. Table 4 shows the estimation accuracy for the cross-participant evaluation. The average R 2 was 0.85, the average estimation error was 50.2 hPa, the average standard deviation was 23.0 hPa, and the average error rate was 41.5%.

Calibrate air pressure to grip strength
We examined the relationship between the ball air pressure and the force applied to the ball. This relationship was investigated in a previous study by using  a force transducer while applying force to the ball and then recording the relationship [21]. Another study showed that one way to calibrate a Smedleytype grip dynamometer is to compare its measurement results with weights suspended from the handle of the dynamometer [29]. From these findings, we determined that the applied force mechanically and the force applied by the human hand are the same. Therefore, we acquired the dataset for creating the regression model using the set-up as shown in Figure 8(a). It is composed of eight frame sections and one acrylic plate. Four frame sections compose a bottom frame, and the four other sections are vertically fixed to the corners of the bottom frame. Metal rods with protrusions on the inside are attached to the vertical frame sections. These rods can move up and down. We fixed the acrylic plate to these rods so that it could move up and down while remaining horizontal. We placed the ball and the pressure gauge between the acrylic plate and the table top and placed an object under the ball to prevent the pressure gauge from interfering with the movement of the acrylic plate. The initial ball air pressure was 100 hPa. We recorded the weights placed on the acrylic plate and the corresponding air pressure and then used that data to create a regression model. Figure 8(b) showed the results: R 2 was 0.997, and the standard deviation was 192.9 g. These results are consistent with those of previous studies, i.e. that the air pressure and force are correlated. This regression model can thus be used to calibrate the ball air pressure to the grip strength.

Finger joint angles estimation
If the finger contour is obtained accurately, we can get the correct estimation as shown in Figure 9(a). Even if not, we were able to remove most of the failed data using the process described in Section 3.3.4. Two factors can cause failed data. One is the roundness of the finger outline. If the outline has a gentle curve, its approximation can fail because the contours are approximated, as shown in Figure 9(b). The other factor is hand size. If the hand is too large for the ball, the contour of the finger on the backside may overlap the contour of the finger on the front side (Figure 9(c)). If the hand is too small for the ball, the fingers may not be visible to the camera because they are buried in the ball (Figure 9(d)).
Thus, failure to accurately estimate the finger joint angles was due to differences in the user's hand features. In this experiment, the parameter for approximating the finger contour was kept constant. A more appropriate parameter could be determined on the basis of the characteristics of the user's hand. Given the results of this experiment, we expect that the parameter for approximating the contour should be small when the user's hand is large and that it should be large when the user's hand is small. The roundness of the finger outline is also related to the acquisition of an accurate contour: the more gradual the user's hand contour, the larger the appropriate parameter. In other words, there is an association between these two hand features and an appropriate parameter. Therefore, the parameter can be determined by asking users themselves to report the size and contour of their hand in comparison with those of a reference before using the application. It is also possible to estimate the hand size from the number of pixels in the hand area.

R 2 evaluation
After removing the failed data, we created a regression model for each participant using the corresponding dataset. The results showed that there is a correlation between the finger joint angles and ball air pressure in both cases: the regression model for each participant and the regression model for several participants. Participant D had the lowest R 2 . It is likely that D did not apply enough force to the ball or squeeze the ball steadily because his or her hand was too small for the ball. In fact, there was little difference in hand movement between squeezing the ball and not squeezing it. Hand size is thus a factor in the degree of fit of the regression model. This means that the ball size should be matched to the user's hand size. Thus, we found that hand size contributes not only to accurate estimation of the finger joint angles but also to the fitting of R 2 .

Air pressure estimation
Previous studies have shown that the force applied by a person to a Smedley grip strength metre is typically about 80 kg, of which the measurement error is about 5 kg [8,30]. The error for the expected measurement range is thus about 6.3%. In contrast, the error for the range of measurement in this study was about 12.2%. This means that we cannot be certain that the estimation of air pressure, or grip force, from images is sufficiently reliable compared with existing methods.
However, the main purpose of this study was to develop a system for providing feedback to people using a ball for grip strength rehabilitation. The accuracy of our estimation method is not as good as that of existing methods; however, it could provide useful feedback if the estimation results were classified into several levels. The results of the within-participant evaluation indicate the feasibility of evaluating grip strength using our estimation method by stages. In addition, the maximum measured ball air pressure was 230 hPa for all participants on average. This means that the average ball pressure varied between 100 hPa and 230 hPa, with a measurement range of approximately 130 hPa. In particular, participants with large or small hands compared with the average hand size tended to have a smaller pressure range. Providing participants with a ball that is appropriate for their hand size should expand the range of measurement.
The error tended to increase as the air pressure approached the maximum and minimum values, probably because the shape and posture of the hand became less stable as the force applied to the ball increased.
Before the experiment, we assumed that, if the training data included more than a certain number of data points, there would be no significant difference in the estimation results. However, when we compared the estimation results for each participant, we found that the dispersion of the error tended to be large when there was a bias in the number of data points in each air pressure band. In particular, participant E and G squeezed the ball faster and thus had fewer data points between the minimum and maximum air pressures. We speculate that increasing the number of intermediate data points reduces the error variation. By investigating the appropriate numbers of data points and the data fineness for creating the regression model, we can determine them required for practical use.
We found that the mean errors tended to be higher for the cross-participant evaluation than for the withinparticipant evaluation. We attribute this to each participant having a different grip. First of all, since the size of the hand varies from individual to individual, the posture of the hand when squeezing the ball at the beginning also varies. Moreover, we observed that there was a difference in which part of the finger joint was bent first and which parts of the finger joint were worked on. This suggests that it would be difficult to create a regression model that fits all users. However, there were some participants, such as participant G and H, for whom the estimation errors were even between the crossparticipant and within-participant evaluations, or for whom the cross-participant evaluation result was better. It is possible that we could categorize users into several groups on the basis of their ball grip data and then share the regression model among the members of each group.

Grip strength estimation
According to the equation shown in Figure 8(b), the average estimation error corresponded to a force of 0.9 kg, and the measurement range was about 0-7.5 kg. The focus here is on measuring the change in grip strength during rehabilitation rather than measuring the maximum grip strength accurately. Therefore, we did not compare the relationship between the actual grip strength and the air pressure during gripping. Nevertheless, it is important to investigate the direct relationship between hand posture and grip strength as future work.

Application
We implemented our proposed system as an application on a smartphone (Huawei Can L-12, Android 6.0) using Unity software. On this device, the system runs at about 10 fps. This system is designed to provide hand rehabilitation at home for outpatients and nursing home residents. We envision a usage situation in which the person begins using the system at the suggestion of a doctor or an occupational therapist and then shares their rehabilitation progress with the doctor or the therapist. The process begins with the creation at the hospital or nursing home of a regression model of the relationship between the measured grip motion and the corresponding air pressure. Once the regression model data has been loaded into an application downloaded to the user's smartphone, the user can begin using the rehabilitation system.
We also implemented our proposed system as a prototype game for grip strength rehabilitation ( Figure 10). Users move an avatar up and down as it moves across the screen, with the distance moved depending on the strength of their grip on the ball. When the user grips the ball harder so that the grip strength increases, the avatar moves up. When the user loosens their grip so that the grip strength returns to normal, the avatar moves down. Users move the avatar so as to avoid a series of walls coming from the right side. They thus perform a grip strength exercise while enjoying playing a game.

Limitations and future work
Our proposed system has four limitations. The first is a color problem. To facilitate the extraction of the hand area from the image, the user must use a ball and background with a color that does not match their skin tone. If the hues are close to each other, the boundary between the hand area and other areas becomes ambiguous and cannot be detected well. In addition, a dark shadow around the hand area may make it difficult to obtain an accurate outline of the hand area, so it is necessary to pay attention to the change in color hue due to illumination.
The second limitation is the hand posture. Estimation of the finger joint angles is based on accurate acquisition of the finger contours. Therefore, the user must keep the posture of their fingers from overlapping; hence they must adjust the orientation of their hand relative to the camera. In addition, the camera cannot capture images of the middle, ring, and pinky fingers because they are hidden by the ball. If the user varies the movements of these finger, the grip strength cannot be estimated accurately. Consequently, our proposed system is not suitable for patients who cannot move their fingers due to damaged nerves or broken bones. The current system can be used only by patients who can move their fingers freely to some extent. As a means of overcoming this limitation, we are considering using an image taken from the direction in which the fingernails can be seen. We expect that more detailed finger movements can be measured by capturing an image from the direction in which all four fingers and thumb are visible. This will enable users to perform the exercise without having to worry about the precise orientation of their hands.
The third limitation is that a regression model needs to be prepared beforehand. As mentioned in Section 4.3.3, it is difficult to create a regression model suitable for all users. Therefore, in the proposed implementations, we assumed that a regression model was created for each participant at a medical facility before use. However, it may be feasible to categorize the grip motions into several groups and then share the model for each group among the members of that group. This would simplify the procedure for using the application. For example, the user could be asked to grip the ball several times at the beginning of the application. Using the time series data of the finger joint angles, the application would then recognize how each joint moves and decide which group's regression model to use in accordance with the data. Future work includes increasing the number of participants with various hand characteristics and investigating whether there is a tendency in the way they grip the ball and whether the regression model can be shared.
Finally, the current system cannot determine whether the reason for large estimation errors frequently occurring during system usage is because the user changed their grip on the ball or because their grip has changed due to restoration of hand function. If this can be determined, it may lead to the user being instructed on the next stage of rehabilitation.
Future work also includes possibly improving the joint angle estimation method by utilizing the interlocking nature of the finger joints to reduce the number of failed data points in the dataset. Our finding that accurate estimation of the grip strength may be affected by hand size suggests the need to identify the appropriate relationship between hand size and ball size as future work. We also need to investigate whether the current accuracy of grip strength is sufficient for users and occupational therapists. Therefore, we will conduct an accuracy experiment with both users and occupational therapists as participants in order to judge the validity of the proposed system by checking its usability and accuracy.
In this study, we focused on the variation in grip strength during rehabilitation. There are, however, other motivational factors, such as counting the number of times the ball was squeezed during an exercise session. Future work will thus also include investigating whether grip strength or the number of times the ball is squeezed contribute to improved motivation.
In the work reported here, we used a smartphone and a commercially available flexible ball for homebased rehabilitation. Smartphones are now found in most homes, but users may not possess a flexible ball. Our results showed that a user needs to obtain a ball with a size appropriate for the user's hand size. These factors can prevent users from using the proposed rehabilitation system easily. To mitigate this problem, we will investigate whether our proposed system can also be applied to the situation of squeezing something other than a ball. For example, we will investigate the feasibility of estimating grip strength and squeeze count from the image of squeezing an object such as a rolled-up towel or a plastic bottle.

Conclusion
Our proposed real-time grip strength measurement system using a soft tennis ball and a monocular camera is aimed at facilitating hand rehabilitation in the home environment. The system estimates the grip strength on the basis of the modelled relationship between the finger joint angles extracted from a camera image and the person's grip strength. The experimental results showed that there is a correlation between the finger joint angles and the air pressure of the ball when squeezed. The average estimation error was 16.1 hPa for within-participant evaluation, and the average measurement range was 100-230 hPa. The estimation error was about 12% of the measurement range. The results also showed that there is a correlation between the air pressure of a ball and the applied force. Future work includes considering the accuracy required for users in rehabilitation and evaluating the relationship between accuracy and hand size.