Open Access
Add to BibSonomy
Abstract
Regular and narrow-band RGB cameras are recently explored for contactless SpO2 monitoring. Regular RGB cameras with cross-band overlap provide a high signal-to-noise-ratio (SNR) in measuring the photoplethysmographic signals but possess high dependency on the spectra of incident light, whereas narrow-band RGB cameras have better spectral independence but lower SNR especially in dim lighting conditions, such as in the neonatal intensive care unit (NICU). This paper proposes a notch RGB camera based SpO2 measurement approach that uses an optical notch filter to attenuate the wavelengths of 580–605 nm of a regular RGB camera to improve the spectral independence while maintaining high SNR in signal measurement. The proposed setup was validated in the lab condition (e.g. dark chamber) against the existing solutions for visible-light based camera-SpO2 measurement and further verified in the NICU on preterm infants. The clinical trial conducted in the NICU with 22 preterm infants shows that the notch RGB camera can achieve a mean absolute error (MAE) less than 4% for SpO2 measurement. This is the first showcase of continuous monitoring of absolute camera-SpO2 values in the NICU.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
Corrections
29 January 2024: A correction was made to the title.
1. Introduction
Blood oxygen saturation is a crucial physiological indicator of human body and plays an important role in clinical diagnosis. Arterial oxygen saturation (SaO$_2$) is one of the major indices used to indicate the oxygen saturation level of a human body. However, the direct measurement of SaO$_2$ needs the blood test using blood gas analyzer, which is invasive and cannot provide continuous assessment. Thus, the oxygen saturation of peripheral blood (SpO$_2$) is used to non-invasively measure the real-time oxygen saturation, which describes the relative concentration of HbO$_2$ molecules in peripheral blood w.r.t. the total amount of hemoglobin molecules, specified as:
In human body, the SpO$_2$ below 90% indicates respiratory failure, while below 80% signifies severe hypoxia [1]. Real-time monitoring of SpO$_2$ is therefore important for various clinical applications, especially for patient monitoring such as in neonatal intensive care unit (NICU). Since the preterm infants have irregular breathing rhythms and may suffer from pulmonary hypertension [2], respiratory distress syndrome [3], apnea of prematurity [4] and meconium aspiration syndrome [5] that will lead to oxygen desaturation (e.g. low SpO$_2$), continuous monitoring of SpO$_2$ is essential in NICU for detecting prompt SpO$_2$ drops of infants in order to avoid asphyxia [6,7].
Typically, SpO$_2$ is estimated based on different absorption properties of Hb and HbO$_2$ at different wavelengths. For a standard pulse oximetry, the wavelengths 660 nm and 940 nm are used to estimate SpO$_2$ since HbO$_2$ and Hb molecules have opposite absorption contrast in between [8]. By measuring tissue’s absorption difference between these two wavelengths, the relative amount of Hb and HbO$_2$ can be determined, i.e. the level of SpO$_2$. Quantitative measurement can be achieved by photoplethysmography (PPG) [9]. For a reflectance pulse oximetry, the intensity of light reflected from skin is used to ensemble the PPG signal [10]. Given two PPG signals measured at the mentioned wavelengths, the Ratio of Ratios Principle can be applied to derive the RR feature for estimating the SpO$_2$ level [11,12]:
where AC($\lambda _i$) and DC($\lambda _i$) refer to the AC and DC components of the PPG signal at the wavelength $\lambda _i$. The linear approximation between RR features and SpO$_2$ values is established as [8]: where the calibration coefficients $A$ and $B$ vary for different $\lambda _1$, $\lambda _2$ and even for different PPG sensors with different optical specs. For accurate SpO$_2$ calibration, each oximetry usually has its own calibration coefficients established based on statistical measurement of a wide range of SpO$_2$ values.Current SpO$_2$ monitoring uses standard pulse oximetry in a contact manner, which attaches a clip to the patients’ finger, earlobe or toe to detect the PPG signal. Contact measurement may induce discomfort and skin irritations to patients, especially to newborns with fragile skin (see Fig. 1(a)). The contact-sensors are also cumbersome for caregivers to use in routine care due to the procedure of sensor wearing and cleaning. Besides, impertinent use of clips may cause cross infection especially in a pandemic period. Recently, the feasibility of using optical cameras to estimate SpO$_2$ was explored, which shows potential of continuous SpO$_2$ monitoring in a non-contact manner (see Fig. 1(b)). According to the used wavelengths, camera-SpO$_2$ can be classified into two categories: one is the setup using red and near-infrared (red-NIR) wavelengths [13–16] that mimics pulse oximetry, which needs an extra NIR illuminator in case if the infrared component is absent in the environment (e.g. fluorescent lamp used in indoor settings does not have an infrared component); and the other is the setup using the full visible (VIS) wavelengths [7,17–21], which can be used in most indoor or daylight conditions as it only needs the red and green wavelengths.
Fig. 1. (a) Contact-based infant monitoring approaches (ECG, PPG) in NICU. The skin-contact sensors (e.g. electrodes, probes) may cause arterial blockage and skin damage; (b) camera-based monitoring approach that measures physiological variables of infants remotely.
However, there are a couple of challenges of current VIS setups which confine their usage in clinical scenarios. These challenges are mainly caused by the inherent limitations of employed consumer-grade cameras. There are three main setups used for VIS camera-SpO$_2$: narrow-band monochrome cameras, regular RGB cameras, and narrow-band RGB cameras. (i) For the narrow-band monochrome setup, two or more monochrome cameras with narrow-band filters in visible wavelengths are used, thus the estimation is influenced by the parallax between monochrome cameras [22], where a robust image registration method is required to eliminate the parallax-induced color artifacts. (ii) For the regular RGB camera, its estimation is interfered by the spectral distribution varying by the incident light [18], i.e. different illumination spectra may lead to different RR-SpO$_2$ calibration coefficients. (iii) For the narrow-band RGB camera, due to the reduction of photons that can be received by the camera, its signal measurement is of low signal-to-noise-ratio (SNR) under a dim lighting condition, which is common for clinical scenarios (e.g. NICU), i.e. the illumination intensity in NICU is usually set to be low for protecting the infant eyes and getting sound sleep.
In this paper, we constructed a notch RGB camera based SpO$_2$ monitoring setup and conducted a clinical trial in NICU. It uses an RGB camera equipped with a notch filter that optically attenuates the wavelengths between 580 nm and 605 nm, which alleviates the dependency of spectral distribution of incident light (e.g. the reasons will be explained in detail in Section 3). By using a notch filter, the setup can achieve better independency on the spectra of incident light than regular RGB cameras. Since the amount of photons received by this setup is apparently larger than narrow-band RGB cameras, it keeps a relatively high SNR of the measured PPG signal under dim lighting conditions. The feasibility of the proposed notch RGB camera setup was validated on healthy adult volunteers in a controlled lab condition, and compared with the regular RGB camera and narrow-band RGB camera. The experiments show that among three benchmarked camera setups, only our setup can achieve a mean absolute error (MAE) less than 4% for SpO$_2$ estimation in the dim lighting condition while improving the independency of ambient light. The functionality of the setup was further validated on preterm infants in NICU, confirming the performance of a MAE less than 4% on 22 preterm infants. To our best knowledge, it is the first demonstration of camera-SpO$_2$ measurement on preterm infants in NICU, i.e. absolute SpO$_2$ values after calibration rather than RR trends.
The remainder of this paper is organized as follows: in Section 2, we review the related work in VIS camera-based SpO$_2$; in Section 3, we conduct the mathematical modeling for VIS camera-SpO$_2$ and explain the motivations of designing a notch RGB camera based SpO$_2$; in Section 4, we introduce the experimental setups and methods; in Section 5, we discuss the laboratory and clinical results; finally in Section 6, we draw the conclusions.
2. Related works
Earlier studies on camera-SpO$_2$ mimicked the contact-based pulse oximetry, which estimates the SpO$_2$ values from skin blood perfusion using red-NIR wavelengths. The initial measurement of PPG in red-NIR wavelengths using a camera was introduced by Wieringa et al. [13], and further studied by Shao et al. [14] for camera-SpO$_2$ measurement. Verkruysse et al. validated the generalization ability of camera-SpO$_2$ on 41 subjects with a wide range of SpO$_2$ values from 83% to 100%, where the wavelengths of 675 nm and 842 nm were particularly investigated [15], and the result showed that it can reach an root mean square error (RMSE) of 2.54% in an ideal diffuse illumination condition, which meets the requirement of ISO standard (80601-2-61, 2011).
Since the NIR components are absent in common indoor lighting conditions (e.g. fluorescent lamp), VIS camera-SpO$_2$ that only requires visible wavelengths was studied. Pilot studies on VIS camera-SpO$_2$ used two monochrome cameras equipped with narrow-band filters (520 nm and 660 nm) to measure the PPG signals to derive RR features for SpO$_2$ calibration [17]. However, when investigating the generalization ability among multiple individuals, the experiments only covered a narrow range of SpO$_2$ (97% to 100%). Later Moço et al. [23] validated the generalization ability upon 46 subjects with a wider range of SpO$_2$ from 85% to 100%, based on the dual wavelengths of 580 nm and 675 nm. The result showed that the VIS setups can achieve an RMSE less than 3% among different subjects under a well-controlled condition, i.e. stable subject in an ideal diffuse illumination.
However, parallax introduced by the setup with two or more monochrome cameras (multi-camera system) may jeopardize the SpO$_2$ estimation [22]. This risk does not happen to the setup with a multi-channel camera (e.g. RGB camera) since it has a single optical path. Tarassenko et al. [24] used a regular RGB camera with indoor ambient light to track the SpO$_2$ changes, based on the PPG signals from the R and B channels. It was the pilot work conducted in the clinical scenario (the Oxford Kidney Unit). However, the authors only reported the results obtained on a single patient and did not investigate the generalization ability among multiple patients. Guazzi et al. [18] introduced a mathematical model for the RGB-camera based SpO$_2$ estimation. They found that the spectral distribution of illumination and spectral response of camera sensors were highly related with the SpO$_2$ calibration. A limitation of the modelling performed by [18] was that the different skin penetration depths of photons with different wavelengths in the visible range was not considered.
Since RGB cameras have the dependency on spectra of illumination source due to the cross-band overlap, Gastel et al. [21] introduced a narrow-band RGB camera setup by adding a triple-narrow-band (457/530/628 nm) filter to the RGB camera to reduce cross-talk between the RGB channels and limit their spectral sensitivity range. The results showed an error less than 4% in realistic scenarios (e.g. no restriction on the subject posture nor the illumination). But the authors also noticed the challenge of low SNR with such a narrow-band RGB camera. Dim lighting conditions are a common challenge for clinical applications, especially in patient care units such as NICU. Therefore, though attaching a narrow-band optical filter to regular RGB cameras may improve the accuracy of SpO$_2$ calibration, it has the challenge of low SNR in signal measurement, especially in low-light conditions. Also, the modeling of RGB-camera based SpO$_2$ needs more supplements as it neglects vital factors such as the impact of different skin penetration depths of different visible wavelengths.
In this paper, we first performed a modeling on RGB-camera based SpO$_2$ monitoring by considering the impact of different skin penetration depths of lights with the RGB wavelengths, in which we found that the wavelengths of 580 - 605 nm have major impact on the dependency on illumination spectra. Based on the modelling and understanding, we proposed a notch RGB camera setup that uses a notch filter to suppress the camera sensitivity between 580 nm and 605 nm. This setup is less affected by the varying spectra of ambient illumination while keeping a high SNR of PPG measurement. Experimental validations and comparisons with existing VIS camera-SpO$_2$ setups were conducted in the lab (e.g. dark chamber) and further verified in the hospital (e.g. NICU).
3. Modeling
3.1 Mathematical modeling of skin optics
For reflectance pulse oximetry, skin reflection ensembles the PPG signals and can give an estimate of SpO$_2$ [25]. We use $I(\lambda )$ to denote the reflected light intensity at the wavelength $\lambda$. In each cardiac cycle, the peak and valley of $I(\lambda )$ are denoted as $I_{p}(\lambda )$ and $I_v(\lambda )$ respectively. Based on the Beer-Lambert law, $I_{p}(\lambda )$ and $I_{v}(\lambda )$ can be modelled by Eq. (4) [26], which is a function of incident light intensity $I_0$, molar extinction coefficient $\varepsilon (\lambda )$, substance concentration $C$, and optical path length of light $l(\lambda )$:
But there is still a subject-dependent item $\Delta l_b(\lambda )$ in Eq. (5), so it needs at least two wavelengths $\lambda _1$ and $\lambda _2$ where $\Delta l_b(\lambda _1)$ = $\Delta l_b(\lambda _2)$ to further eliminate the impact of $\Delta l_b(\lambda )$:
Now Eq. (6) associates RR with the concentrations of Hb and HbO$_2$, which indicate the level of SpO$_2$ as specified by Eq. (1)). And Eqs. (5)–(6) are named the “Ratio of Ratios principle”. Besides, based on the following equation given by [8]:
RR can be calculated by Eq. (2) as well.
For regular RGB cameras, the observed light reflection is an integration of a wide range of wavelengths. Thus $I_{p}$ and $I_{v}$ need to be rewritten from Eq. (4) to Eq. (8), which integrates the wavelengths inside the channel response range $\overline {\lambda }$ for modelling:
Therefore, a modification on regular RGB cameras is motivated. Limiting the sensitivity wavelength of channels is one way of modification, but the limited wavelength range cannot be too wide, otherwise photons cannot be sufficiently received by the camera to guarantee the SNR of PPG measurement. Besides, the green-red wavelengths are our main focus of modification as these two channels will be used for SpO$_2$ estimation. Fig. 2(a) shows the spectral responses of G and R channels specified by the manufacturer (IDS UI-3860CP-C-HQ with IMX290LQR-C CMOS sensor) and the absorption spectra of Hb and HbO$_2$ given by [27]. It can be seen that $\varepsilon _{HbO_2}$ has a sheer decline from 580 nm to 610 nm (according to [27], $\varepsilon _{HbO_2}(580)$ = 50104 cm$^{-1}$/M, while $\varepsilon _{HbO_2}(610)$ = 1506 cm$^{-1}$/M, between which is a decline more than 97%). Similar decline also exists for $\varepsilon _{Hb}$. Additionally, according to [28,29], $\Delta l_b$ in 580 - 610 nm also differs from neighboring 500 - 580 nm and 610 - 700 nm. So, $\varepsilon _{Hb}$, $\varepsilon _{HbO_2}$ and $\Delta l_b$ cannot satisfy Eq. (9) in 580 - 610 nm, i.e. suppressing the wavelengths of 580 - 610 nm is needed for GR channels to meet the requirement of Eq. (9) in order to apply the Ratio of Ratios principle. This is our initiative of using an optical notch filter to suppress the camera sensitivity between 580 nm and 610 nm for improving the stability of SpO$_2$ calibration. We found an off-the-shelf notch filter that can suppress the wavelength of 580 - 605 nm (NF03-594E-23.3, Semrock), which is similar to our expectation (see Fig. 2(a)). It was used to implement our proposed notch RGB camera setup as shown in Fig. 2(c). In our feasibility study in the lab, the setups of regular RGB camera, narrow-band RGB camera and notch RGB camera were benchmarked to verify the modeling hypothesis.
Fig. 2. (a) The absorption spectrum of HbO$_2$ and Hb given by [27], the spectral responses of RGB channels and the used notch filter are given by the product white paper; (b) the spectra of various illumination sources given by [30]; (c) the illustration of how to add a notch filter into a regular RGB camera to change it to a notch camera.
3.2 Numerical simulation
Following the modeling, we performed a numerical simulation to investigate if RR given by the notch RGB camera setup can indeed reduce the dependency of illumination spectra. To calculate the simulated $I_{p}$ and $I_{v}$ (denoted as $I_{p,s}$ and $I_{v,s}$ respectively), we combined Eq. (4) and Eq. (8) and replaced the integration sign with summation for numerical simulation:
According to [28,29], $\Delta l_b$ can be regarded as a constant in 500 - 580 nm, so we can use $\Delta l_b(550)$ to approximate the $\Delta l_b$ in 500 - 580 nm, and similarly, use $\Delta l_b(660)$ to approximate $\Delta l_b$ in 605 - 700 nm. For the wavelength of 580 - 605 nm, its $\Delta l_b$ is approximated by:
To estimate $\Delta l_b(550)$ and $\Delta l_b(660)$, we used a LED of 550 nm or 660 nm to illuminate the skin of a subject and measured $ln[I_p(\lambda )/I_n(\lambda )]$ of the subject. A polarizer was used to eliminate specular reflections when measuring $\Delta l_b$ [34]. The measurements and corresponding parameters are shown in Table 1. Applying the numerical values in Table 1 to Eq. (13), $\Delta l_b$ can be determined: $\Delta l_b(550)=0.1\,\mu m, \Delta l_b(660)=2\,\mu m$.
Table 1. The estimated results and corresponding parameters for calculating $\Delta l_b$.
With acquired $\Delta l_b(\lambda )$, $I_{p,s}(\overline {\lambda })$ and $I_{v,s}(\overline {\lambda })$ can be calculated based on Eqs. (11–12). Then, according to the Ratio of Ratios principle, RR can be simulated by:
The simulation results are shown in Fig. 3. For each SpO$_2$ value (71% to 99% with the interval of 4%), we calculated the standard deviation and mean values of all simulated RRs derived by six types of illumination, which are shown in Fig. 3(a). To investigate the error made by the standard deviation in SpO$_2$ estimation, we calibrated the simulated RR to SpO$_2$ value. The calibration model of each camera was based on the mean value of simulated RR and its corresponding SpO$_2$ value. The error range in SpO$_2$ made by different illuminations are shown in Fig. 3(b). It can be seen that the error range of the notch RGB camera is apparently narrower and stabler than the regular RGB camera, and it can reach a similar error range level as the narrow-band RGB camera but stabler for different SpO$_2$ values, suggesting that the notch RGB camera has no bias when estimating different SpO$_2$ levels. Based on this simulation, we conducted experiments in the lab upon real subjects to evaluate the camera setups with different illumination conditions, especially the low-light condition.
Fig. 3. (a) Standard deviation and mean of simulated RR at each SpO$_2$ with six different illuminations (71% to 99%, with an interval of 4%); (b) the SpO$_2$ error range caused by standard deviation of simulated RR under six different types of illumination.
4. Materials and methods
4.1 Experimental setups and protocols
The experimental setups constructed in the lab and hospital are shown in Fig. 4. With these two setups, we can explore four main research questions: i) whether the notch RGB camera can achieve a better performance for SpO$_2$ estimation in ideal illuminations in the lab, compared with regular and narrow-band RGB cameras; ii) whether the notch RGB camera can perform well in dim light illumination conditions for SpO$_2$ estimation, compared with narrow-band RGB cameras; iii) whether the wavelength of 580 - 605 nm is the best choice of notch filter for SpO$_2$ estimation; iv) what is the performance of notch RGB camera in practical clinical scenarios like infant SpO$_2$ monitoring in NICU. The details of each experiment are shown in Table 2. Besides, for the categories of “dim” and “bright”, we define it quantitatively: based on the RGB camera we used and largest aperture, 1x gain, 12 ms exposure time, no gamma correction, with a white illumination, for the case that the skin pixel-value (in the range of 0 - 255) in the G channel below 60 is considered as dim, above 130 is considered as bright.
Fig. 4. The experimental setups constructed in the dark chamber of the lab (left) and the NICU incubator (right). For the setup built in the lab, Lamp Fluo. (A) refers to a fluorescent lamp with bright and stable illumination (ideal), Lamp Fluo. (B) refers to a fluorescent lamp with dim and non-direct illumination (non-ideal). Lamp LED refers to a LED with wavelength 570 - 610 nm.
Table 2. A list of the illumination setups for each experiment
In Exp. I, we compared the notch RGB camera with regular and narrow-band RGB cameras in ideal lighting conditions for SpO$_2$ measurement. In this experiment, only Lamp Fluo. (A) (a bright and stable fluorescent lamp) was turned on, the narrow-band camera centered at 550 nm and 660 nm. The spectra of narrow-band filter and notch filter are shown in Fig. 2(a). The reference SpO$_2$ values of subjects were measured by a clinical-grade pulse oximetry (Mindray BeneVision N17 oximetry). Besides, we added a clinical-grade wearable device (Apple Watch S7) to measure SpO$_2$ from the subject’s wrist simultaneously. The reason why using these two devices together is that the Apple Watch uses a reflectance manner to estimate SpO$_2$ while pulse oximetry uses a transmissive manner, and these two manners may produce a different estimate when SpO$_2$ is unstable. In order to acquire a reliable SpO$_2$ reference, we used them together. If the difference between SpO$_2$ measured by two devices is larger than 4%, the measurement will be discarded. A total of 20 healthy adult volunteers involved in this experiment. The subject was asked to sleep in the bed and remained stable without voluntary motion. The forehead skin area of each subject was recorded in a 300 sec video sequence. In the first 100 sec, the subject was asked to breath normally, and in the next 100 sec, hold the breath to decrease SpO$_2$ till she/he cannot hold any longer. And for the rest of time till the end of recording, subject was asked to breath freely again.
In Exp. II, we investigated the performance of notch RGB camera in a dim lighting condition, benchmarked with the narrow-band RGB camera. The purpose of this experiment is to validate if the amount of light transmittance of the notch RGB camera is sufficient for SpO$_2$ estimation. We anticipate that the notch RGB camera will be less sensitive than the regular RGB camera since a part of spectrum is blocked. Regular RGB camera has larger light transmittance and it will provide higher SNR of signal measurement than the notch camera in dim illumination. This is known in advance and thus was not compared for this experiment. In this experiment, only the Lamp Fluo. (B) (a dim and indirect illuminated fluorescent lamp) was turned on to mimic the environment for sleep monitoring. A total of 10 healthy adults were involved. The subject was asked to sleep in the bed for real sleep. Apple Watch attached to the wrist can label different sleep stages (REM sleep, light sleep and deep sleep). The forehead skin area of each subject was recorded in a 60-minutes video sequence.
In Exp. III, we investigated whether the wavelength 580 - 605 nm is the best choice of notch filter for SpO$_2$ estimation. A special illumination was used in this experiment: fluorescent light plus an additional LED illumination of 570 - 610 nm. A regular RGB camera was benchmarked to show how significant the impact of the wavelengths 580 - 605 nm is on SpO$_2$ estimation. Besides, we used a wider wavelength of additional illumination in 570 - 610 nm rather than 580 - 605 nm so as to investigate whether prohibiting 580 - 605 nm is sufficient for improving SpO$_2$ estimation. The experimental protocol was the same as Exp. I.
Finally, a clinical trial was conducted in the NICU of The Third People’s Hospital of Shenzhen to validate the functionality of the notch RGB camera setup. The dim illumination was a combination of a weak fluorescent light and non-directly illuminated daylight, and a total of 22 infants were recorded. Note that we did not modify the illumination setting in NICU in order to ensure that all clinical experiments were conducted in real clinical environments and use cases. The notch camera was used to record a 10 - 15 minutes video for each infant. The reference SpO$_2$ was recorded by a clinical-grade pulse oximetry (Mindray Benevision N17 patient monitor). Among 22 infants, 20 were in the state of sleep and showed a stable SpO$_2$ over at least 30 seconds, and the remaining 2 infants were awake and their SpO$_2$ varied much over the recording period.
All cameras used in our experiments were IDS UI-3860CP-C-HQ with the IMX290LQR-C CMOS sensor. The lens (brand: IT-C041216-2MP) were with 4 - 12 mm focal length, F1.2-C aperture range, and 2 mega pixels. The narrow-band camera was equipped with a band-pass filter centered at 550 nm and 660 nm. The notch camera was equipped with a notch filter blocking the wavelengths of 580 - 605 nm. All the cameras were externally triggered at a stable frame rate of 60 Hz. The image data was captured at a resolution of 484 $\times$ 274 pixels. The study was approved by the Institutional Review Board of Southern University of Science and Technology and the Institutional Review Board of The Third People’s Hospital of Shenzhen (IRB no.: 2022-07-02), and written informed consents were obtained from the test subjects or from the legal guardian of infants.
4.2 Video and signal processing
The algorithmic workflow for extracting the SpO$_2$ signal from a video is shown in Fig. 5. We first manually select the skin region of interest (ROI) that contains the PPG signal, and then average the pixel values of ROI and concatenate the spatially averaged skin-pixel values from consecutive frames as a raw signal trace in each channel. A sliding window with 10 sec length and 1 sec stride is applied. In each window, the raw signal is temporally normalized by dividing its mean value (DC normalization) and filtered by a 2nd order butter-worth band-pass filter with cut-off frequency [0.7, 3] Hz (adults) or [1.5, 5] Hz (preterm infants) to eliminate the out-band distortions. After that, the peak-valley distance (AC/DC) is measured from each cardiac cycle in the PPG signal. We have two means to extract the peak-valley distance: the first is to calculate the difference between the medians of peaks and valleys [15], and the second is to calculate the envelop of the signal by Hilbert Transform [35] and then take the median of the envelop height (i.e. the norm of the products of Hilbert Transform) as the peak-valley distance. We used both to determine a peak-valley distance: if the difference between the two means is less than 5%, the average of them will be used, otherwise abandoned as an invalid measurement. To derive RR, the peak-valley value of the R channel is divided by that of the G channel. Finally, in the training process, RR will be calibrated with a SpO$_2$ reference to derive the calibration coefficients; in the testing process, RR will be calibrated to an absolute SpO$_2$ value using the estimated calibration model.
Fig. 5. Block diagram of camera-SpO$_2$ extraction algorithm. The linear regression step highlighted by dashed box is only needed in the training process for determining the RR-SpO$_2$ calibration coefficients.
5. Results and discussion
In this section, we first discuss the results obtained in the lab feasibility study, and then the results obtained in the clinical trial.
5.1 Feasibility validation in the lab
According to section 4.1, three experiments (Exp. I-III) were conducted in the lab to benchmark three camera setups: regular RGB camera, narrow-band RGB camera and notch RGB camera. The results are presented and discussed in order. Figure 6 shows the results of Exp. I, where we compared the performance of notch, narrow-band and regular RGB cameras under the ideal illumination with bright and stable fluorescent lamp. It can be seen that most of the data points lie inside the 95% confidence interval and a linear fit can be implemented for all three setups. However, it is apparent that the 95% confidence interval of the notch RGB camera is narrower than that of narrow-band and regular RGB cameras, and more data points outside the 4% tolerance range were produced by narrow-band and regular setups. The R$^2$ values of narrow-band and regular setups are also lower than that of the notch setup, which are 0.63, 0.60 versus 0.77.
Fig. 6. The scatter plot and linear fit show the correlation between measured RR and reference SpO$_2$ for three benchmarked RGB camera setups in an ideal illumination condition (bright and stable fluorescent lamp).
We discuss the reasons for the performance degradation of narrow-band and regular camera setups one by one. (i) For the narrow-band RGB camera, the sensitive red wavelength is centered at 660 nm but fluorescent lamp provides limited illumination at this wavelength (see Fig. 2(b)). Thus, the PPG signal measured at 660 nm has low SNR. The absorption of hemoglobin at 660 nm is also low, which means that PPG measurement in the red channel is challenging for the narrow-band setup. This was not found by prior art that introduced the narrow-band filter [21], because their experiments used a LED designed exclusively for the used narrow-band filter (i.e. they used a narrow-band filter sensitive at 457/530/628 nm and a LED with energy concentrated at these wavelengths as well). So, we suggest that, as part of our unique perspective, notch filters, which only attenuates 25 - 30 nm spectra between R and G wavelengths that deteriorates the measurement most but keeps the boarder sensitivity of R and G channels on their respective sides, is a better choice. (ii) For the regular RGB camera, the worse performance was not expected because $S(\lambda )$ and $r(\lambda )$ were invariant for this experiment, which was not due to illumination source. To investigate the reasons, we further rewrite Eq. (8) as:
Secondly, we investigated the camera setups in the dim lighting condition (by a dim fluorescent) in Fig. 7, i.e. the results of Exp. II. This experiment was to mimic the practical scenario of monitoring sleeping patients in warding. We noted that the apple watch worn by the subjects was used to label sleep stages and only the results obtained at the deep sleep stage are presented, which has two main purposes: (i) to check if our experiments were conducted under the low-light illumination in which the subjects were able to sleep, which mimicked the warding and monitoring environment; (ii) to focus on the stages in which the subjects were unconscious and may be trapped into a case of short breathlessness that could reduce SpO$_2$ [36], resulting in a variation of SpO$_2$. We used the linear fit produced in Exp. I to calibrate RR to SpO$_2$ values in this experiment (for the notch RGB camera: SpO$_2$ = −36.5$\times$RR + 112.2; for the narrow-band RGB camera: SpO$_2$ = −35.9$\times$RR + 107.8, as shown in Fig. 6).
Fig. 7. The comparison between SpO$_2$ traces generated by notch and narrow-band RGB camera setups in the dim illumination condition. MAE [notch] = 1.63%, MAE [narrow] = 24.98%.
Figure 7 shows that under a dim lighting illumination, the narrow-band setup produced a SpO$_2$ estimate with larger deviation with the reference than the notch setup. Besides, it can be seen that decreasing SpO$_2$ cases happened to subject II and IV, and notch setup can still track these changes under the low-light condition. The quality drop of narrow-band camera can be explained by worse measurement SNR as we expected. For a dim lighting condition, due to the few amounts of light transmittance, the signal SNR of the narrow-band camera cannot even reach the least required SNR for a possible communication specified by Shannon Equation [37], which is specified in Eq. (18):
where $C$ is the real transferring rate of the signal, i.e. the signal generation frequency; in this case, the signal source is the skin, and the signal to be transmitted is the PPG signal, and the transmitting rate is exactly its frequency, i.e. pulse rate; $B$ is the bandwidth of the communication channel; in this case, it is the highest frequency that sensor can perceive, and based on Nyquist sampling theorem it is half of the sampling rate of camera. We examine the SNR of the signals obtained by the narrow-band RGB camera and notch RGB camera respectively, calculated by [37]: where the $H$ is the Fourier Transform of signals, i.e. SNR of both signals from R and G channels were calculated and the lower one was used to check the condition of Eq. (18). The results are shown in Fig. 8, which meet our expectation that the signal SNR of the notch RGB camera is higher than the least required SNR by Eq. (18), whereas for the narrow-band RGB camera, signal SNR is lower than the least required SNR and lower than that of the notch RGB camera.Fig. 8. Plot showing the comparative results between SNR of notch camera setup, SNR of narrow-band camera setup and least required SNR determined by Eq. (18) under low-light illumination (i.e. sleep lamp).
Then, we present the estimates of the notch and regular RGB camera setups under the illumination of bright fluorescent with additional LED lights in 570 - 610 nm in Fig. 9 (i.e, the results of Exp. III). We theoretically proved in section 3.1 that 580 - 610 nm is a major wavelength range that induces the measurement dependency on the illumination spectra for RGB camera setups, so in Exp. III, we used an additional LED illumination in this range to check whether it will apparently affect the estimation. Besides, in order to check whether a notch filter that attenuates 580 - 605 nm is sufficient for resisting the negative impact on SpO$_2$ estimation, the additional illumination of this experiment was extended to a wider range wavelength of 570 - 610 nm. We used the linear fit produced by Exp. I to calibrate RR to SpO$_2$ value in this experiment (for notch RGB camera: SpO$_2$ = −36.5$\times$RR + 112.2; for regular RGB camera: SpO$_2$ = −35.1$\times$RR + 115, as the Fig. 6 shows).
Fig. 9. The estimated results of the notch and regular RGB camera setup under an illumination of bright fluorescent with additional LED lights in 570 - 610 nm. MAE [notch] = 1.4194, MAE [regular] = 6.5404.
As Fig. 9 illustrates, SpO$_2$ estimated by the regular RGB camera setup has clear variance among subjects: for Subjects I-V, it produced a large deviation w.r.t. the reference and cannot track the SpO$_2$ changes; for Subject VI, it gave a decent measurement that is close to the reference; for Subject VII and VIII, it can track SpO$_2$ changes but still yielded a deviation w.r.t. the reference. On the contrary, the notch RGB camera setup can still perform generally well among all subjects when tracking the SpO$_2$ changes and giving outputs close to the reference. This result can be explained by Eqs. (16–17): the spectral distribution of incident light for Exp. III was different from Exp. I, so when applying the linear fit of Exp. I for calibrating RR of Exp. III to SpO$_2$, it is reasonable that some deviations occur in regular RGB camera. Besides, as Eq. (16) indicates, inside the integration sign, the spectral distribution of incident light $S(\overline {\lambda })$ is multiplied with the impact of non-blood components $I_{nb}$, non-pulsatile blood component $I_{b,np}$ and pulsatile blood component $I_{b,p}$, which are all subject-dependent, leading to variations among different subjects. For the notch RGB camera, it was less affected by the impact of $S(\overline {\lambda })$, $I_{nb}$ and $I_{b,np}$ (see Eqs. (9)–(10)), yielding a more accurate and stabler performance among different subjects. The experimental results indicate that illumination of 570 - 610 nm indeed have a negative impact on camera-SpO$_2$ estimation, and suppressing the wavelength range of 580 - 605 nm is beneficial for reducing the negative impact.
In summary, the notch RGB camera can indeed improve the performance of camera-SpO$_2$ measurement in low-light conditions. Attenuating wavelength range 580 - 605 nm is reasonable for resisting the impact of spectral distribution of incident light. Further confirmation on the performance of notch RGB camera was conducted in the NICU scenario, where the notch RGB camera is the focus of our investigation.
5.2 Clinical validation in the NICU
Following the feasibility study in the controlled lab, the notch RGB camera setup was further investigated in NICU for infant SpO$_2$ monitoring. We only evaluated the notch RGB camera in this clinical trial, for the following reason: the pulsation strength of infants is weak, so the distance between camera and infants cannot be too far, however, in a close distance, even with two cameras juxtaposed tightly, it is not possible for two cameras to record videos of forehead of infant simultaneously since the forehead of preterm infants is too small. Besides, we have proved that even under an ideal and controlled lab condition, the regular RGB camera and narrow-band RGB camera cannot reach similar level of performance as the notch RGB camera, so they are less likely to work properly in non-ideal and undetermined practical scenarios (e.g. hospital care units).
A total of 22 infants involved in this clinical trial and 22 videos of 10 - 15 minutes were recorded for all the infants. Segments of these video were divided into a couple of sets for training and testing. The details are shown in Table 3. For the training process, we used the data in DS I to obtain the calibration parameters A and B by fitting to a linear regression model, and the underline model is least squares regression. The definition of calibration coefficients is shown in Fig. 10(a), where all the data points lie inside the 4% tolerance range, giving the model SpO$_2$ = -29.0$\times$RR +116.9. For the testing process, its results are shown in Fig. 10(b), Fig. 11 and Fig. 12. Figure 10(b) presents a Bland-Altman plot of camera-SpO$_2$ and reference SpO$_2$ based on DS II, which shows that most of differences are between Mean$\pm$1.96 Std, and the two points outside the Mean$\pm$1.96 Std are still within $\pm$3% range. DS III involves the segments of the most stable parts which were manually selected by visual inspection, i.e. the parts in which the infants did not have a strong motion like head rotation. Figure 11 is generated based on DS III, which shows the continuous SpO$_2$ traces obtained by the notch RGB camera setup for sleeping infants. Figure 12 shows the SpO$_2$ traces obtained from awake infants. From Fig. 11, it is clear that there is no large continuous deviation between camera-SpO$_2$ and reference SpO$_2$ for 20 sleeping infants. Some short-term deviations appear, which were caused by abrupt motion during sleep. The overall MAE of these sleeping infants is 3.41%, which is less than 4% specified by the ISO standard (80601-2-61, 2011) [15] but larger than that of experiments conducted in lab upon adult. This may be caused by the weaker blood pulsation of preterm infants (i.e. undeveloped cardiorespiratory system) [38]. From Fig. 12(a) we can see a rapid decrease in SpO$_2$ of >10% and a rapid increase of >15% for the Infant 21 within a short-term period less than 40 seconds, which can be accurately tracked by the notch RGB camera. Figure 12(b) shows a continuous change of SpO$_2$ over 500 seconds for Infant 22, it is clear that the notch RGB camera can generally track the continuous changes in SpO$_2$, though with certain delay in time. The delay is suspected to be caused by the different measurement skin-spot by camera and pulse oximetry [17], i.e. camera measured infant’s forehead skin while pulse oximetry measured infant’s toe. The overall MAE of these awake infants is 3.17%, which is similar to that of sleeping infants, indicating the stability of notch RGB camera setup when measuring infants in different states.
Fig. 10. (a) The scatter plot and linear fit show the correlation between measured RR and referenced SpO$_2$ based on DS I ; (b) Bland-Altman plot of camera-SpO$_2$ and reference SpO$_2$ based on DS II.
Fig. 11. The continuous estimation results of notch RGB camera setup for sleeping infants (Infant 1-20, DS III). SpO2 MAE = 3.41%.
Fig. 12. The continuous estimation results of notch RGB camera setup for awake infants (Infant 21 (a) and Infant 22 (b), DS IV). SpO2 MAE = 3.17%.
Table 3. The video data sets divisiona
Note that during our experiments, we adjusted the camera to ensure that there was no direct mirror-like specular reflections from the incubator shield and the incubators were totally transparent. About 1/3 infants involved in our experiments were not recorded in an incubator, and we did not observe a major difference between these infants and the ones recorded in an incubator. So, the effect of incubator on camera-SpO$_2$ measurement is considered to be minimal for this study.
In summary, the results of this clinical trial show that SpO$_2$ measured by the notch RGB camera setup can be applied to track the continuous SpO$_2$ of infants in sleep or awake state with an accuracy meeting the ISO standard. For the next step, we will initiate a multi-center clinical trial for infant SpO$_2$ monitoring, to include data measured from larger populations from different hospitals with different environments to confirm the performance. Besides, motion distortion is still a major challenge for continuous SpO$_2$ monitoring, so our future work will also focus on improving the motion-robustness of SpO$_2$ monitoring in the NICU scenario.
6. Conclusions
This paper proposed a notch RGB camera setup to overcome the limitations of existing regular and narrow-band RGB cameras in estimating SpO$_2$. By using an optical notch filter, we can improve the independency of SpO$_2$ measurement on the illumination spectra while maintaining a high measurement SNR. Feasibility validations were conducted upon 20 healthy volunteers using three RGB camera setups (regular, narrow-band and notch) under different illumination conditions. A clinical trial was conducted upon 22 preterm inpatients in NICU to validate the functionality of the notch RGB camera in practical clinical use cases. The results show that the notch RGB camera setup can reach a performance of MAE<4% in estimating SpO$_2$ from both the sleeping and awake infants.
Funding
National Key Research and Development Program of China (2022YFC2407800); National Natural Science Foundation of China (62271241); Basic and Applied Basic Research Foundation of Guangdong Province (2023A1515012983); Shenzhen Science and Technology Innovation Program (JSGGKQTD20221103174704003); Shenzhen Science and Technology Innovation Program (JSGG20220606141001003); Shenzhen Fundamental Research Program (JCYJ20220530112601003).
Acknowledgments
The authors would like to thank the volunteers from the Southern University of Science and Technology for the involvement in the lab experiments and the patients from The Third People’s Hospital of Shenzhen for the participation in the clinical trial.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. L. O’Brien, V. Stebbens, C. Poets, et al., “Oxygen saturation during the first 24 hours of life,” Arch. Dis. Childhood-Fetal Néonatal Ed. 83(1), 35F–38 (2000). [CrossRef]
2. M. M. Hoeper, H.-A. Ghofrani, E. Grünig, et al., “Pulmonary hypertension,” Deutsches Ärzteblatt International 114, 73 (2017).
3. L. B. Ware and M. A. Matthay, “The acute respiratory distress syndrome,” N. Engl. J. Med. 342(18), 1334–1349 (2000). [CrossRef]
4. E. C. Eichenwald, K. L. Watterberg, S. Aucott, et al., “Apnea of prematurity,” Pediatrics 137(1), 1 (2016). [CrossRef]
5. C. Monfredini, F. Cavallin, P. E. Villani, et al., “Meconium aspiration syndrome: a narrative review,” Children 8(3), 230 (2021). [CrossRef]
6. Y. Zeng, X. Song, J. Yang, et al., “Time-domain features of angular-velocity signals for camera-based respiratory ROI detection: A clinical study in NICU,” in 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), (2023), pp. 1–6.
7. M. Villarroel, A. Guazzi, J. Jorge, et al., “Continuous non-contact vital sign monitoring in neonatal intensive care unit,” Healthcare Technol. Lett. 1(3), 87–91 (2014). [CrossRef]
8. J. G. Webster, Design of Pulse Oximeters (CRC Press, 1997).
9. M. Evans and L. Geddes, “An assessment of blood vessel vasoactivity using photoplethysmography,” Medical instrumentation 22(1), 29–32 (1988).
10. A. Moço, S. Stuijk, and G. de Haan, “Posture effects on the calibratability of remote pulse oximetry in visible light,” Physiol. Meas. 40(3), 035005 (2019). [CrossRef]
11. K. K. Tremper, “Pulse oximetry,” Chest 95(4), 713–715 (1989). [CrossRef]
12. J. W. Severinghaus, “Recent developments in pulse oximetry,” Anesthesiology 76(6), 1018–1038 (1992). [CrossRef]
13. F. P. Wieringa, F. Mastik, and A. van der Steen, “Contactless multiple wavelength photoplethysmographic imaging: A first step toward “SpO2 camera” technology,” Ann. Biomed. Eng. 33(8), 1034–1041 (2005). [CrossRef]
14. D. Shao, C. Liu, F. Tsow, et al., “Noncontact monitoring of blood oxygen saturation using camera and dual-wavelength imaging system,” IEEE Trans. Biomed. Eng. 63(6), 1091–1098 (2016). [CrossRef]
15. W. Verkruysse, M. Bartula, E. Bresch, et al., “Calibration of contactless pulse oximetry,” Anesth. analgesia 124(1), 136–145 (2017). [CrossRef]
16. T. Abay and P. Kyriacou, “Photoplethysmography for blood volumes and oxygenation changes during intermittent vascular occlusions,” J. Clin. Monit. Comput. 32(3), 447–455 (2018). [CrossRef]
17. L. Kong, Y. Zhao, L. Dong, et al., “Non-contact detection of oxygen saturation based on visible light imaging device using ambient light,” Opt. Express 21(15), 17464–17471 (2013). [CrossRef]
18. A. R. Guazzi, M. Villarroel, J. Jorge, et al., “Non-contact measurement of oxygen saturation with an RGB camera,” Biomed. Opt. Express 6(9), 3320–3338 (2015). [CrossRef]
19. P. S. Addison, D. Jacquel, D. M. Foo, et al., “Video-based physiologic monitoring during an acute hypoxic challenge: heart rate, respiratory rate, and oxygen saturation,” Anesth. Analgesia 125(3), 860–873 (2017). [CrossRef]
20. N. H. Kim, S.-G. Yu, S.-E. Kim, et al., “Non-contact oxygen saturation measurement using YCGCR color space with an RGB camera,” Sensors 21(18), 6120 (2021). [CrossRef]
21. M. Van Gastel and W. Verkruysse, “Contactless SpO2 with an RGB camera: experimental proof of calibrated SpO2,” Biomed. Opt. Express 13(12), 6791–6802 (2022). [CrossRef]
22. M. van Gastel, W. Wang, and W. Verkruysse, “Reducing the effects of parallax in camera-based pulse-oximetry,” Biomed. Opt. Express 12(5), 2813–2824 (2021). [CrossRef]
23. A. Moço and W. Verkruysse, “Pulse oximetry based on photoplethysmography imaging with red and green light: calibratability and challenges,” J. Clin. Monit. Comput. 35(1), 123–133 (2021). [CrossRef]
24. L. Tarassenko, M. Villarroel, A. Guazzi, et al., “Non-contact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiol. Meas. 35(5), 807–831 (2014). [CrossRef]
25. Y. Mendelson and B. D. Ochs, “Noninvasive pulse oximetry utilizing skin reflectance photoplethysmography,” IEEE Trans. Biomed. Eng. 35(10), 798–805 (1988). [CrossRef]
26. M. Van Gastel, S. Stuijk, and G. De Haan, “New principle for measuring arterial blood oxygenation, enabling motion-robust remote monitoring,” Sci. Rep. 6(1), 38609 (2016). [CrossRef]
27. S. Prahl, “Optical absorption of hemoglobin,” Oregon Medical Center News (1999).
28. L. Finlayson, I. R. Barnard, L. McMillan, et al., “Depth penetration of light into skin as a function of wavelength from 200 to 1000 nm,” Photochem. Photobiol. 98(4), 974–981 (2022). [CrossRef]
29. A. V. Moço, S. Stuijk, and G. de Haan, “New insights into the origin of remote ppg signals in visible light and infrared,” Sci. Rep. 8(1), 8501 (2018). [CrossRef]
30. D. Smith, “Calculating the emission spectra from common light sources,” COMSOL Multiphysics (2016).
31. C. D. Karakochuk, S. Y. Hess, D. Moorthy H. M. H. W. Group, et al., “Measurement and interpretation of hemoglobin concentration in clinical and field settings: a narrative review,” Ann. N. Y. Acad. Sci. 1450(1), 126–146 (2019). [CrossRef]
32. M. Sidstedt, J. Hedman, E. L. Romsos, et al., “Inhibition mechanisms of hemoglobin, immunoglobulin g, and whole blood in digital and real-time pcr,” Anal. Bioanal. Chem. 410(10), 2569–2583 (2018). [CrossRef]
33. M. Al-Othmani, N. Ghaddar, and K. Ghali, “A multi-segmented human bioheat model for transient and asymmetric radiative environments,” Int. J. Heat Mass Transfer 51(23-24), 5522–5533 (2008). [CrossRef]
34. W. Wang, A. C. Den Brinker, S. Stuijk, et al., “Algorithmic principles of remote PPG,” IEEE Trans. Biomed. Eng. 64(7), 1479–1491 (2017). [CrossRef]
35. D. Benitez, P. Gaydecki, A. Zaidi, et al., “The use of the Hilbert transform in ECG signal analysis,” Comput. Biol. Med. 31(5), 399–406 (2001). [CrossRef]
36. M. R. Mannarino, F. Di Filippo, and M. Pirro, “Obstructive sleep apnea syndrome,” Eur. J. Internal Med. 23(7), 586–593 (2012). [CrossRef]
37. C. E. Shannon, “A mathematical theory of communication,” The Bell Syst. Tech. J. 27(3), 379–423 (1948). [CrossRef]
38. P. Stanger, M. A. Heymann, H. Tarnoff, et al., “Complications of cardiac catheterization of neonates, infants, and children: a three-year study,” Circulation 50(3), 595–608 (1974). [CrossRef]
