Remote investigation and assessment of vital signs (RIA-VS)—proof of concept for contactless estimation of blood pressure, pulse, respiratory rate, and oxygen saturation in patients with suspicion of COVID-19

Abstract Background Vital signs are critical in assessing the severity and prognosis of infections, for example, COVID-19, influenza, sepsis, and pneumonia. This study aimed to evaluate a new method for rapid camera-based non-contact measurement of heart rate, blood oxygen saturation, respiratory rate, and blood pressure. Methods Consecutive adult patients attending a hospital emergency department for suspected COVID-19 infection were invited to participate. Vital signs measured with a new camera-based method were compared to the corresponding standard reference methods. The camera device observed the patient's face for 30 s from ∼1 m. Results Between 1 April and 1 October 2020, 214 subjects were included in the trial, 131 female (61%) and 83 male (39%). The mean age was 44 years (range 18–81 years). The new camera-based device’s vital signs measurements were, on average, very close to the gold standard but the random variation was larger than the reference methods. Conclusions The principle of contactless measurement of blood pressure, pulse, respiratory rate, and oxygen saturation works, which is very promising. However, technical improvements to the equipment used in this study to reduce its random variability is required before clinical implementation. This will likely be a game changer once this is sorted out. Clinical trial registration Universal Trial Number (UTN) U1111-1251-4114 and the ClinicalTrials.gov Identifier NCT04383457.

Currently available methods for measuring vital signs include several devices that need physical contact with the patients. Therefore, these measurements and assessments are time-consuming, cumbersome for the clinicians, and unpleasant for the patients. There is also an apparent risk of transferring infections from one patient to the next [19]. The subsequent consequences are that vital signs are not estimated as often as they should and they are often inaccurately measured when obtained. This major problem has been observed both in hospitals [2] and in primary care [20].
Gathering vital signs is one of the critical steps to triage suspected potentially life-threatening infections, such as COVID-19. Therefore, any tool that would simplify the data gathering process will reduce the stress among health care workers and increase the accuracy of risk stratifying patients. Furthermore, if this automated process could be made contactless, it would also eliminate the risk for disease transmission between patients as well as from patient to caregiver.
For many years, contactless body temperature estimation has been tried with various accuracy [21]. Estimating respiratory rate, pulse, and oxygen saturation have been attempted, but evidence mainly rests on studies including a small number of healthy subjects [22]. Many phone apps claim they can estimate pulse with high accuracy [23]. However, most of them require that you put the finger directly onto the camera [23]. Hence, they are not contactless. A few phone apps, such as GoodSam V R , Lifelight V R , and Binah.ai V R , claim they can estimate pulse, and other vital signs, using the phone's camera from a distance. However, there is only little published evidence in peer-reviewed scientific publications for their accuracy when estimating pulse [24].
Blood pressure has been estimated from photoplethysmography (PPG) signals obtained from a finger clip sensor [25,26]. The theoretical principle for camerabased contactless estimation of blood pressure has been explored by Jeong et al. [27]. However, Jeong et al. only presented an inter-person correlation between blood pressure and image-based pulse transit time rather than estimating blood pressure. We are not aware of any peer-reviewed publications validating a contactless camera-based method to estimate blood pressure against an acceptable gold standard.
Today, there are available systems claiming they can measure blood pressure contactless using conventional smartphones. However, some of these systems seem to predict blood pressure from demographic data, such as age, sex, and height rather than measuring blood pressure [28]. Predicting vital signs from demographic data is likely to be on average correct but an incorrect prediction might be detrimental to the individual having abnormal blood pressure [28]. Hence, studies need to look at techniques for actually measuring, rather than predicting, vital signs validating them in real-life ill patients.
This study aimed to investigate the concept of a high-speed camera system linked to artificial intelligence (AI) to measure blood pressure, pulse, respiratory rate, and SpO 2 contactless by briefly observing a patient's face from a distance of $1 m.

Materials and methods
This study compared vital signs obtained with the new camera device with those obtained using standard reference methods in patients seeking the emergency department of a tertiary hospital with a suspicion of having a COVID-19 infection.
The study was approved by the Swedish Medical Product Agency (registration number 5.1-2020-36817) and the Swedish Ethical Review Authority (registration numbers 2020-01936 and 2020-02723). All participating patients signed informed consent. The study has Universal Trial Number (UTN) U1111-1251-4114 and the ClinicalTrials.gov Identifier NCT04383457.
In this study, vital signs were first obtained using standard reference methods, immediately followed by a high frame rate video-recording of the patient face for later data processing and analysis. Finally, vital signs were again obtained using standard reference methods.

Trial monitoring
The study was monitored by Scandinavian CRO adhering to ISO14155:2011, ICH-GCP, and applicable regulations.

Participants
Patients aged !18 years attending the Sahlgrenska University Hospital € Ostra emergency department for triage, presenting with typical COVID-19 symptoms were included if they were fluent in Swedish (reading, writing, conversational), mentally capable of understanding and giving informed consent. As a prerequisite, the investigator had to determine that the new method, and the reference methods, could be used as intended with adequate reliability and safety. The study incurred a delay in the management of each patient by $10 min. Therefore, patients deemed being in such a severe medical condition on arrival that 10 min of delay was detrimental were not included.

Estimating vital signs using reference methods
The reference method for blood pressure was an automatic blood pressure measuring device measuring on the upper arm; A&D UA-651 (A&D Company Ltd., 3-23-14 Higashi-Ikebukuro, Toshima-Ku, 170-0013 Tokyo, Japan). The heart rate from the blood pressure measuring apparatus was used as the reference for pulse. The respiratory rate was measured manually by observation for 60 s. For measurement of SpO 2 , a portable pulse oximeter of the brand Dixtal DX-2405 (Dixtal Biomedical Ind. Com., São Paulo, Brazil) Nellcor OXIMAX N-65 with a finger sensor including an alligator clip. The pulse oximetry measurement was 30 s, counting from the moment the reading was stabilized with the device.

Creating a photoplethysmography (PPG) signal
The device, hereafter named remote investigation and assessment of vital signs (RIA-VS), consisted of a tripod with a high-speed camera, one red LED light, and one infra-red LED light. Lights were connected to a timer box timing switching on and off LED lights. The camera was connected to a laptop for recording the video sequences. A 30 s RGB video of the participants face was recorded at 396.5 frames per second sample rate. This recording consisted of 10 s in ambient light, 10 s of additional red light followed by 10 s of additional infra-red light. During the later data processing of the video recording, the subject's face was automatically detected in each frame and bounded by a face rectangle using the Viola-Jones method [29]. The pixels in the face rectangle were blurred to normalize different skin colours and wrinkles. Then the video frames were colour magnified using the Eulerian video magnification algorithm [30]. Next, the region consisting of the top five percent of the face rectangle covering the middle portion of the forehead is segmented to extract a photoplethysmography (PPG) signal.
To obtain a PPG signal from the chosen region of the face, varying colours of the forehead, from red to pale yellow, were used to estimate systolic and diastolic phases, respectively. These were processed so that in each frame, each pixel could be denoted as being in a systolic or diastolic phase.
In the later stage where videos were analyzed, pulse and respiratory rate were estimated using proprietary software algorithms processing the PPG signal, while AI was used for blood pressure. SpO 2 was analyzed twice, with proprietary software algorithms and with AI.

RIA-VS without using AI
Heart rate, respiratory rate, and SpO 2 under red and infra-red light illumination were estimated using proprietary algorithms not involving AI.
Heart rate per minute was computed by counting the number of peaks in the PPG signal normalized over 60 s. The respiration rate was estimated by enveloping the PPG signal. The idea was that during the inhalation of air, the increased level of blood oxygen increases skin redness. When the air is exhaled, the de-oxygenated blood reduces the redness. This increasing and decreasing trend in hue values of the PPG signal was used to encapsulate breathing cycles. The encapsulation was done by computing the upper and lower envelopes in the PPG signal using spline interpolation. Once the upper and lower envelopes are constructed, a peak finder algorithm is used to find the location of peaks in the lower envelope and the location of valleys in the upper envelope. A distance threshold was used between the location of peaks in the lower envelope and the location of valleys in the upper envelope to mark the boundary of a breathing cycle. The total number of envelopes in the signal was counted and normalized over 60 s to estimate the respiration rate over a minute. SpO 2 was estimated using two different methods. The first method incorporated high-speed videos recorded under the consecutive illumination of red and infra-red light. In this method, two different PPG signals were obtained from a subject's face that was illuminated using a red and an infra-red LED light, respectively. The lines of best fit for the two PPG signals were computed using polynomial curve-fitting. Ratios between the PPG signal and the line fit for each of the red and the infra-red recording were computed separately. Finally, SpO 2 was estimated as the ratio between the ratios of red and infra-red light.

RIA-VS using AI
The second method of estimating SPO 2 and the method for blood pressure was based on AI. These methods utilized videos of subjects that were recorded under ambient light. PPG signals were computed from the videos of all subjects. Using time and frequency domain features extracted from the PPG signals, separate random forests algorithms [31] were trained to predict SpO 2 , systolic, and diastolic blood pressure. Important features were selected automatically for training the random forests based on the permutation score of each feature against the gold standard.
The first version of the AI algorithms for SpO 2 and blood pressure estimation were trained on a limited data set of healthy subjects. In this study, the first analysis was to let the first version of AI algorithms estimate SpO 2 and blood pressure for the complete patient data set. Then, a new version of the AI algorithms was trained on the complete patient data set from this study, using leave one subject out (LOSO) cross-validation. In this cross-validation method, the first patient's data is left out and the AI is trained on the data of all other patients. The AI is then asked to predict the vital sign of the first patient without knowing its gold standard value. Next, the data of the second patient is left out and so forth until the AI has predicted the vital signs of all patients. This procedure ensures that the trained AI is tested using an unseen sample, consequently producing unbiased prediction results.
Hence, for SpO 2 and blood pressure, this approach produced two results, one before the AI had been trained on real-life patients and one after being trained on patients.

Statistical methods
The average between the first and second reference measurement was the gold standard used to validate RIA-VS. The mean absolute error (MAE) (mean difference between the RIA-VS and the gold standard), its 95% confidence interval, its range, and the proportion of measurements inside the acceptable tolerance error was calculated for validation. These differences were also graphically visualized with Bland-Altman plots.
The acceptable tolerance error compared with the gold standard was set to ±10 mmHg for blood pressure [32], ±10% of the gold standard for pulse [33], ±5.0 breaths per minute for respiratory rate [34], and ±3.2% for SpO 2 [35]. Blood pressure predictions should be within the acceptable tolerance error in !85% of measurements to be deemed acceptable [32].
The random variability was estimated by calculating the variance for the difference between RIA-VS and the first reference measurement and comparing this with the variance for the difference between reference measurement 2 vs. reference measurement 1.
The training of the AI on the complete patient data set made the AI more tolerant of poor video quality. Hence, for blood pressure only 70 of 214 videos were usable before training while this increased to 191 after training (Table 1). A similar phenomenon was seen for SpO 2 in ambient light ( Table 2).

Estimating blood pressure with RIA-VS
In the first analysis, before training on the complete patient data set, the mean absolute error in SBP and DBP between the RIA-VS device and the gold standard was on average þ3.2 and þ2.4 mmHg, respectively (Table 1). With the new AI version, using the leave one out cross validation described earlier, this improved SBP and DBP to þ0.069 and À0.13 mmHg, respectively ( Table 1). The training on patients also made the 95% confidence intervals narrower. 2.4 a Oxygen saturation was measured twice with the RIA-VS method. Once with the illumination of infra-red and red light from two LED-lights and once only using the ambient room illumination, the latter method involved AI learning. b Number of patients where data was available. c Gold standard was the average between two measurements of the reference standard method. d Tolerance errors as described in the methods section. e Estimating SpO 2 with the new RIA-VS involves AI learning. We first tested the algorithm after previous experience only in healthy volunteers. We then let the computer training on all patients and analyzed them according to the principle leave one subject out (see methods section). f After training (n ¼ 182).  [32]. d Estimating systolic and diastolic blood pressure with the new RIA-VS involves AI learning. We first tested the algorithm after previous experience only in healthy volunteers. We then let the computer training on all patients and analyzed them according to the principle leave one subject out (see methods section).

Estimating oxygen saturation with RIA-VS
The AI was much better at estimating SpO 2 using plain ambient light compared with the fixed algorithm aided by illumination with infra-red and red LED light (Table 2 and Figure 2). Training the AI did not make much difference (Table 2 and Figure 2).

Random variation and normalization when estimating vital signs with RIA-VS
Although the estimation by the RIA-VS device was, on average very accurate for all vital signs, there was a large random variation in the difference between the RIA-VS estimation and the gold standard (Tables 1, 2 and Figures 1-3). For all vital signs but pulse the RIA-VS device tended to normalize the measurements so that high values were underestimated while low values were overestimated (Figures 1-3). Training the AI on patients did not noticeably improve the normalizing tendency for blood pressure or SpO 2 (Figures 1, 2).

Comparison of random variation by RIA-VS and reference methods
The difference between the two measurements with the reference method was large with a substantial random variation (Tables 1, 2). However, in all vital signs, the random variation was more prominent for the RIA-VS device compared with when the reference method was compared itself (Tables 1, 2). Pulse and SpO 2 in ambient light estimated using RIA-VS only had a little higher random variation, 1.4 and 2.0 times higher, respectively, compared with the reference method ( Table 2). The random variation was much higher than the corresponding reference method for systolic blood pressure (3.9 times), diastolic blood pressure (3.3 times), respiratory rate (4.7 times), and SpO 2 in red and infra-red light (6.7 times).

Discussion
The measurement of vital signs by the new camerabased RIA-VS device was on average very close to the gold standard. However, there was a considerable random variation in both the reference methods and the new RIA-VS device. The random variation of the RIA-VS device was slightly larger than the reference method when measuring pulse and SpO 2 in ambient light. The random variation of the RIA-VS device was much larger than the reference method for blood pressure, respiratory rate, and SpO 2 using red and infra-red light.

Strengths and weaknesses
The major strength of this study is that it included reallife patients with a varying degree of deviating vital signs assessed in a busy clinical reality. The planned number to include was 1000 patients with suspected COVID-19. However, during the summer months of 2020, the number of patients attending the emergency department for suspected COVID-19 decreased significantly. In early October 2020, the number of patients included was down to one per week and it was deemed the COVID-19 pandemic was over in Sweden (it was later evident that this assumption was incorrect). Hence, data collection was stopped in early October before the desired number of patients was included. Although this study included fewer patients than planned it is still much larger than most previous similar studies including <100 healthy subjects.
The video registration for estimating blood pressure, pulse, respiratory rate, and SpO 2 in ambient light lasted only 10 s. This may be too short for a reliable estimate. It would be better to use a slightly longer data collection duration for these estimates in future studies to ensure better precision.
The initial testing and training of the AI algorithm to estimate blood pressure before this study was based on  healthy subjects in a controlled environment. During that initial AI training, the video stream was fed directly into a computer analyzing the data on the fly. This procedure was repeated for each individual until a good quality registration was obtained. In this study, the video stream was obtained from ill patients in a stressful hospital emergency department. Only one short video recording was made for later analysis and there was no opportunity to check if the video recording quality was good enough for later analysis. During this video recording, emergency department staff sometimes approached the patient, starting a conversation and making them turn their face and talk. Hence, the registrations obtained in this study were much more challenging to analyze compared to the previous situation in a controlled setting.
This study failed to include any patients of the Fitzpatrick skin type 5-6. Subsequently, the results in this study are only valid for patients with skin type 1-4 while the behaviour of the RIA-VS device in patients with skin type 5-6 remains unknown.
This study also intended to analyze body temperature using a separate infra-red camera, comparing it with the body temperature measured using an ear thermometer. As with other vital signs, the output from the thermal camera was recorded using the recommended software from the manufacturer of the camera, with the plan to analyze all of them later. However, all these recordings were distorted so that necessary meta-data were either not recorded or became corrupt, making camera-based body temperature measurements impossible. Hence, in this publication, we have not included descriptions of the technology for measuring body temperature, nor data from the reference ear thermometer.

AI learning potential
We found it intriguing that the AI was much better at estimating SpO 2 than optimized algorithms analyzing PPG signals supported by illumination from red and infra-red light. The AI tended to normalize observations. This is most likely explained by the fact that the AI has seen much more observations close to the group average and too few outliers. The tendency to normalize is likely to be smaller with further training on patients. Further training will also likely reduce the random variations. However, it remains to be seen if further AI training and other technical refinements can make a future version of RIA-VS good enough for clinical use.

Lessons learned
The RIA-VS device in this study included several manual hands-on procedures. It is essential to build a RIA-VS device that automates the whole procedure, including checking that ambient light and distance to face are good enough for obtaining acceptable data. Furthermore, it would be imperative to ask patients to close their eyes, sit still and avoid talking during the short RIA-VS measurement. Closing their eyes may not be essential to obtain good data but reduces the chances that they turn their head because they saw something interesting or start talking with someone.

Reproducibility
No one knows exactly what an AI does when it solves the designated task. Hence, it may be considered as a 'black box'. Although an AI can't be described in detail that may not be necessary. The outcome of the AI can always be evaluated despite the path the AI took to get there remains unknown. If the AI is found to estimate vital signs accurately, it can be copied for distribution and evaluated by other researchers.
The intention is to further refine the AI and supporting proprietary software and in subsequent future studies evaluate this. If it eventually achieves acceptable accuracy it can be CE-certified as a medical device. Once it is CE-certified it means further development of the AI is frozen and we have a static 'black box' that can be distributed for validation elsewhere by other researchers who can repeat our study using the final version of the 'black box'. Any parallel further development of the AI after CE certification can't be implemented unless it is carefully tested in further studies.

Conclusions and future directions
This study shows that the principle of contactless measurement of blood pressure, pulse, respiratory rate, and SpO 2 works. However, the RIA-VS equipment showed a larger random variation than the reference method. Hence, the RIA-VS, as configured in this study, cannot yet be recommended for use in routine medical care. Efforts are needed to refine the technique, further train the AI, and compile this into an automated, user-friendly device. In addition, future evaluations should investigate the validity and reliability of an improved version and investigate if the device's validity is influenced by the patient's skin tone graded according to Fitzpatrick [36]. Furthermore, it would be important to investigate if the validity and reliability of an RIA-VS device are influenced by the presence of atrial fibrillation with an irregular pulse.