Towards the investigation on the effect of the forearm rotation on the wrist FMG signal pattern using a high-density FMG sensing matrix

Abstract Force myography (FMG) is an emerging technique to predict extremity movement. The FMG signals captured near the wrist can be used to predict many hand gestures. However, the wrist FMG pattern of a gesture changes along with different forearm positions. It is difficult to accurately predict a gesture without knowing the effect of the forearm position on the wrist FMG pattern. To investigate such an effect, we developed a novel setup to register the pattern using the off-the-shelf force-sensing matrix to capture the change of the pattern during forearm rotation for two distinct hand gestures. Based on the collected data, we observed the wrist FMG patterns shifted in relation to the forearm orientation in some scenarios. We analyzed the results and examined the viability to use such a setup to facilitate future studies.


Introduction
Force myography (FMG) is a technique to use force sensors to capture the change of limb muscle stiffness for different limb postures or movements (Z. G. Xiao & Menon, 2019a). Using machine learning, FMG signal can be associated with different action commands for human-machine-interface ABOUT THE AUTHOR Dr. Menon is currently a Professor in the Schools of Mechatronic Systems & Engineering Science at Simon Fraser University in Vancouver, British Columbia, Canada and the Director of the WearBioTech University Core Facility. This Core Facility is to become an international hub for assisting the complete cycle of development and validation of mobile health systems. Dr. Menon founded the Menrva Research Group (www.sfu. ca/menrva.html) at SFU, which is focused on the research of wearable biomedical technology, biorobotics and smart materials. He collaborates closely with physical therapists, neuroscientists, and clinicians at a number of institutions and has strong ties with both local and international industry, which supports his research.

PUBLIC INTEREST STATEMENT
We often use our hands to interact with the physical world during activities of daily living. This information of the hand movement can also help us to navigate in the digital world and beyond. For example, we can use gesture to send commands to a computer or a phone to play video game or control smart home appliance. Being able to capture such information unobtrusively and translate them accurately are the keys for the successful utilization of such an approach. Force myography (FMG) is an emerging technique for capturing this information because of its minimalistic and low-cost nature. However, there are still challenges to be solved for the technique to be applied in everyday scenario. In this communication, we present a novel device for capturing FMG signal pattern of the wrist and investigate the effect of forearm rotation on the pattern. The aim of this study is to facilitate future investigation to improve movement prediction using the FMG approach.
applications, such as turning on-and-off a lamp or controlling a robotic device (Anvaripour & Saif, 2019). Also, FMG can be used to monitor the activity of daily living for exercise or rehabilitation purposes (Z. G. Xiao & Menon, 2019b, 2017aSadarangani et al., 2017;Stefanou et al., 2018).
A method to capture FMG was to place multiple resistive polymer thick film (RPTF) force sensors around a targeted limb with a preload force to register the change of muscle stiffness (Z. G. Xiao & Menon, 2019a). As a resistive sensor, the resistance of an RPTF sensor decreases as the applied force increases. This type of sensor is small, lightweight, and flexible which made it suitable for FMG research. Also, researchers can purchase the off-the-self RPTF sensor, such as the force-sensing resistor (FSR), for a relatively low price (Castellini & Ravindra, 2014). The high affordability and easy access to the sensor make FSR the most popular sensor to be used in FMG research (Z. G. Xiao & Menon, 2019a).
Many studies used custom FMG signal acquisition device to predict upper limb actions. Most of these devices were straps that were embedded with multiple FSR sensors in a single row to capture the signal (Z. G. Xiao & Menon, 2019a). Despite such a type of the FMG device is able to predict many upper limb actions (Dementyev & Paradiso, 2014;Jiang et al., 2016Jiang et al., , 2018, a multi-row configuration is expected to be able to reveal more potential of the FMG approach. A multi-row configuration can capture the FMG signal pattern in a two-dimensional (2D) space (Radmand et al., 2014;Castellini et al., 2018). This type of 2D pattern can reveal the change of the pressure magnitude and monitor the translational change of the FMG pattern.
Some researchers constructed a multi-row FMG sensing device with multiple single FSR sensors in order to capture the 2D FMG pattern (Chengani et al., 2016;Li et al., 2012;Michael et al., 2008;Ogris et al., 2007;Wang et al., 2010). However, such devices were less compact and cumbersome to use in practice due to the sophisticated wiring needs. Other researchers independently or partnered with sensor manufacture to design a custom multi-row FMG sensor in a form of the array or the matrix to capture the 2D pattern (Radmand et al., 2014;Castellini et al., 2018;Ferigo et al., 2017). This approach allows the researchers to design compact hardware with a high spatial resolution to capture the pattern, which is preferable.
The majority of FMG research used the signals captured from the bulk region of the forearm to predict hand actions (Z. G. Xiao & Menon, 2019a). This region spans an area that covers the bellies of the major muscles which control the movement of the forearm, wrist, and hand (Z. G. Xiao & Menon, 2017b). Hence, the signals from this region can be used to predict many hand gestures. For example, Jian et al. showed 48 different hand gestures can be predicted from this region with a comparable accuracy to the more established surface electrode myography (sEMG) approach (Jiang et al., 2016). This study also showed the signals captured from the distal forearm (near the wrist) could achieve similar performance. The distal region covers the tendon portion of the same major muscle groups that are responsible for the hand movement; despite of having a different FMG pattern, the same hand movement can be predicted. Comparing to the bulk of the forearm, the distal end of the forearm is a more convenient location to capture the FMG signals to predict hand action as the force sensors can be embedded inside a wrist band (Z. G. Xiao & Menon, 2017a;Dementyev & Paradiso, 2014). However, similar to the FMG signals extracted from the bulk region, the wrist FMG pattern is expected to be highly subjected to the forearm rotation. Since forearm rotation is one of the essential movements involved in daily activities, it will be difficult to accurately predict a hand gesture without considering the change in forearm position.
Currently, many studies only focused on predicting the hand action using FMG pattern with a fixed forearm position. In order to assess the full applicability of FMG in everyday scenarios, the effect of forearm rotation on the wrist FMG pattern should be investigated. As a step to investigate the effect and produce a solution to accurately predict hand gesture independent of the different forearm positions, we developed a High-Density (HD) FMG sensing matrix for the wrist and conducted a preliminary investigation on the effect of forearm rotation on the wrist FMG signal pattern. The design of the HD-FMG matrix and the result of the preliminary investigation are presented in this paper.

HD-FMG sensor
To capture the HD-FMG signals, we used TPE-901 multi-touch sensor pad from Tangio Printed Electronics as our main sensing elements (Electronics, 2020). The sensor pad, as shown in Figure 1 has a total of 70 sensing elements that covers an active sensing area of about 8 cm by 5.2 cm. Each sensing element has an area of 0.5 cm by 0.5 cm with a 0.2 cm gap between the adjacent sensors.
A single sensor pad was not long enough to fully wrap around the distal forearm, therefore, at least 2 sensor pads were used. As shown in Figure 2, two sensor pads were trimmed and joined together to form a longer sensor matrix. This new matrix had an active sensing length of 15.5 cm, which was long enough to be wrapped around an adult's wrist. To be noted, we also prototyped a sensor matrix that was consisted of three identical force sensing pads to be used for an individual with a large wrist cross-sectional circumference.
The original sensor pad has a smooth and flat surface which was difficult to capture the miniature change of the force induced by the muscle stiffness. In order to improve the sensitivity of the contact point, a plastic bump-on was added to each sensing element. The sensor pads were highly flexible, they could be coiled around a limb as shown in Figure 3. The HD-FMG sensor matrix was designed to be  secured to a limb with a modified Velcro strap an elastic end. The hook side of the strap was used as the backing of the sensor pad to provide support to the sensors. The head of the strap was curled towards the inner wall of the sensor and the tail end of the strap had an elastic element with a Velcro to tighten the strap. The signal cable was coming out of the tail end for connecting the sensor to a custom electronic board to capture the FMG signals.
When the two ends of the sensors were met, the strap forms a circle with a diameter of about 4.5 cm which was sufficient to cover the distal portion of the forearm of an average adult as illustrated in Figure 4. The area of the sensor matrix span over the muscle-tendon of multiple  major muscle groups such as brachioradialis, flexor carpi radialis, palmaris longus, flexor carpi ulnaris, extensor carpi ulnaris, extensor digitorum, etc. The area also covers a region of the flexor retinaculum and the extensor retinaculum that are near the wrist.
For this investigation, we defined the top row as Row 1 of the sensor, and the most bottom one as Row 7. The first column of the sensor started alongside the ulna bone and increased along the posterior side of the forearm towards the thumb as shown in Figure 4.

Design of the signal acquisition device
The sensing elements were arranged in a matrix format with rows and columns. As shown in Figure 5, every single sensor has two terminals, one terminal was connected to the common line of the corresponding row and the other terminal was connected to the common line of the corresponding column. With such a configuration, not all the sensors could be read at one instance; a line scanning method was adopted.
The line scanning method required the common lines of the rows or the ones of the columns to be connected to the digital output of a controller and then the unconnected ones to be linked to the controller's analog input. For the proposed design, the common lines of the rows were connected to the analog input pins; and the ones of the columns were connected to the digital output pins. The controller used in this design was the "CY8CKIT-059" from Cypress Semiconductor Corp (CY8CKIT-059). It has 16 configurable analog input pin and up-to 48 digital pins which were sufficient for the proposed design.
During operation, only one of the digital pins should be set to "high" while the other should be set to "low". Whichever the digital pin was set to "high", the sensor values of the corresponding column would be read through the analog input pins. However, such configuration only could capture the force signal accurately if only one force-sensing element was being pressed. If two or more elements were being pressed, the signal reading of the elements around the ones that were being pressed would also change. In addition, the reading of the one that was being pressed would be lower than actual force reading. This phenomenon is caused by the current leaked within the sensor network and it is commonly referred to as the "cross-talk" phenomenon. The detailed information of such a phenomenon can be found in .
In order to obtain an accurate signal reading, we adopted a circuitry that was proposed in (Wu et al., 2015) to combat the "cross-talk" phenomenon. The proposed circuit used external operationalamplifiers (op-amp) to condition the signal reading. For each digital pin connection, an op-amp with unity feedback was added between the common line and the pin to provide an isolated control signal. For each analog pin connection, an op-amp with the inverted voltage feedback configuration was added between the common line and the input pin to stabilize the sensor reading.
With the circuitry shown in Figure 5, the systems were able to capture the HD-FMG signals without "cross-talk". The signal readings were digitized by the microcontroller and could be received by a personal computer (PC) through a standard universal serial bus (USB) connection. With such connection, the data output rate could reach 150 frames per second and each frame consisted of 140 sensor data. According to an investigation conducted by us (Z. G. Xiao & Menon, 2019c), an output rate of 150 frames per second was sufficient to capture the FMG signal pattern during dynamic actions.

Design of the signal acquisition interface
In order to capture the data, a custom data acquisition interface was developed using LabView from National Instrument. As shown in Figure 6, the interface allowed a user to monitor the HD-FMG signal pattern in real-time through the 2D intensity graph. The 2D FMG pattern is displayed in the middle graph. The bicubic interpolation method was used to create a smooth picture for realtime display. There are two white dots shown in the middle of the graph. These two dots revealed the locations and intensities of two pressure points during testing. As seen in the graph, no visually visible "cross-talk" phenomenon can be observed.
Using the custom interface, a user could save the data by a simple click of the "Save data" button. This interface also allowed a user to capture a reference force pattern and used it to offset the subsequent data to reveal the change of the HD-FMG pattern.

Experiment
A preliminary experiment was conducted to examine the change of HD-FMG pattern for two different hand gestures during forearm rotation. The main objectives of the experiment were to observe the characteristic of the signal pattern and to evaluate the device's performance for future FMG investigation.

Experimental procedure
In the experiment, one male volunteer donned the matrix on the distal end of the right forearm (near the wrist). When putting on the sensor, the forearm was a pronated position. The first column of the sensors was aligned with the ulna bone and wrap around the forearm in the direction as illustrated in Figure 4.
Before the data recording, he sat on a chair with the arm relaxed in a vertical position. After the recording started, he fully pronated the forearm with the hand in a relaxed condition (see Figure 7(a)), then he supinated the forearm at a slow comfortable speed to the maximum achievable supinated position (see Figure 7(c)). After that, he pronated the forearm back to the first starting position. He then repeated the movement two more times. Once completed the actions, he fully extended the fingers and repeated the forearm movement in the same manner (see Figure 7(d-f)).

Results
The captured data are shown in Figure 8 as different colored lines. These lines are the signals from the 140 sensors of the matrix. The first plot shows the data captured which the hand was relaxed, and the second plot shows the data in which the fingers were fully extended. From each subplot, we can see three similar sets of patterns. Each set of the pattern indicates the FMG signals during a single pronated-and-supinated action of the forearm.
Based on the morphologies of the signals shown in the two subplots, we expected the two hand states can be distinguished throughout the rotation. However, it was not possible to identify the location of the pressure points from this plot.
To further examine the change of FMG pattern during forearm movement, the transitional patterns from the pronated to the supinated forearm position were extracted from the dataset and they are presented in Figures 9 and 10 for the two gestures. The first subplot of each figure shows the signals in the time domain; the other three subplots show the instant 2D signal patterns while the forearm was in the pronated, the middle and the supinated positions respectively. The blue vertical lines in the subplot (a) indicate the temporal locations for which the 2D patterns were captured. The 2D patterns are shown as the colored map with row number on the y-axis and the column number on the x-axis. The magnitudes of the signals are presented using a blue-to-yellow color scheme. The darker the color, the lower the magnitude is, and vice versa. For easy observation, the offsets of the signals were removed by subtracting the minimum signal magnitude of each channel from the whole dataset. After the offset removable, the ranges of the magnitude of the signal became from 0 to 400.

Discussion
This paper presented a novel HD-FMG matrix setup for investigating wrist FMG signal pattern. Different from other HD-FMG devices presented in the literature, we used off-the-shelf sensors to capture the FMG signals, therefore, the setup can be easily replicated by others. Using such a setup, we captured a set of preliminary data to assess the signal pattern to determine whether such a setup is suitable to capture wrist FMG for research purposes. The majority of the published studies that used the matrix only focused on capturing patterns from the bulk region of the forearm, therefore, an investigation on the wrist FMG fills a gap in the current research paradigm (Z. G. Xiao & Menon, 2019a).
As seen in Figures 9 and 10, the matrix data are able to show different pressure zones that span across multiple rows. This type of pattern cannot be revealed using a single FMG array. If a single array is used, only a section of such a pattern can be revealed. Since the pattern across the rows was different, the data of the 2D matrix contain more information which has the potential to be used for predicting more movements of the hand.  pressures of the corresponding sensors are relatively low. The only pressure zones can be observed were within the third and the fourth row and within the fifth and the tenth column. The two zones cover the area where the tendons of the extensor muscle group were located (please refer to Figure 4). When comparing the pressure zone pattern between the subplot (b) in Figures 9 and 10, the pattern shows in Figure 10(b) is brighter, the brighter pressure zone tells us the matrix registered higher pressure when the extensors muscles were pulling the fingers.
When the forearm started to supinate towards the other end, the pressure zones of both figures start to fade while new pressure zones appeared. These trends can be observed from Figure 9(a) at about 0.7 seconds on the time axis and from Figure 10(a) at about 0.4 seconds. In these instances, the magnitudes of the green and blue lines decrease, and some magenta-and orange-colored lines start to increase. When the forearm reached the middle position, new pressure patterns have emerged as shown in the subplot (c) of both figures. The new patterns had two main pressure zones. The first zones for both figures were centered around Column 1 and Row 3 and they spanned from Column 20 to Column 3. It is important to note that Column 20 was connected back to Column 1 because of the circular configuration. The second zones for both figures were centered around Column 12 and Row 4 but with different lengths of the span. When the hand was extended, the span of the zone was larger than the one when the hand was relaxed. For each subplot, the two pressure zones covered the areas where the radial and ulna bones were located. Hence, as the forearm rotated to the middle position, the movements of the radial and ulna bones became the main factors that affected the wrist FMG pattern.
As the forearm continued to supinate to the maximum angle (see the subplots (d) of Figures 9 and 10), the magnitudes of the two pressure zones became higher and the zones spanned with a larger area too. Also, when looking closely, the pressure zones with center around Column 12 in subplot (c) were shifted by 1 or 2 column steps in subplot(d). This shift of pattern coincided with the rotation of the forearm as the ulna and radial bone moved under the skin tissue. At the same time, a pressure zone located close to the one appeared in subplot (b) has appeared. The pressure of the one with the extended finger was higher than the one when the hand was relaxed. This suggested the movement of the tendons still an important factor to affect the wrist FMG pattern.

Conclusion
Based on this preliminary investigation, we concluded the proposed HD-FMG matrix was suitable for investigating FMG pattern near the wrist. Using the off-the-shelf sensors, the matrix was easy to construct, and the result showed the matrix could register the 2D FMG pattern. With smaller and more densely packed sensors, the 2D matrix was expected to be more capable than the 1D sensor array that was constructed using elementwise FSR sensors. Also, factoring in the cost per unit of the sensor element, the matrix configuration was more cost-effective.
We conducted a preliminary investigation on how the wrist FMG pattern could be affected by the forearm rotation using the proposed setup. The result showed that we could identify different pressure zones in the covered area for different gestures. We also could track how the pressure zone appear-and-disappear and monitor the shifting of pressure zone due to the forearm rotation. The result hinted both the movement of the bone and the muscle-tendon dictated the manifestation of the FMG pattern near the wrist. The successful demonstration of the device allowed us to further investigate how forearm movement, as well as different wrist positions, change the FMG pattern for hand gestures. Therefore, we will continue to improve the current device and make it wireless and conduct a study with different user populations to investigate the FMG pattern for dynamic movement.