1. Introduction

The smartphone steadily makes its way to become an indispensable tool in individual healthcare and living quality monitoring. This trend is made possible by the rapid developments of the sensing modules that are specifically tailored for smartphones, e.g., built-in cameras [1–3], microphone [4–6] and touch screen [7,8]. Among these sensing modules, the camera has experienced extensive technical innovation and nowadays can deliver a comparable imaging quality to specialized medical imagers. With assembled color filters in Bayer arrangement, smartphone cameras may be able to differentiate the spectral information from collected signals in RGB channels. Utilizing this ability, researchers have been exploring smartphone-enabled analysis to extract tissue chromophore information like hemoglobin [9], melanin [10], bilirubin [11], etc. However, the built-in architecture of Bayer filters in the smartphone camera inevitably limits spectral resolution that is required to evaluate the optical properties of the sample, largely due to the overlapped sensitivity ranges in the RGB channels [12].

For example, as a biomarker of liver functions, bilirubin has distinct absorption in the wavelength bands between 350 and 500 nm, which can be exploited to develop optical bilirubinometer for measuring blood bilirubin level (BBL) in people [13–15]. Aiming for a low cost and easy access, enormous effort has been paid to realize bilirubin detection with smartphone cameras. Previous studies reported some strategies by extracting raw signals from RGB channels in photographs [16,17]. While the results are promising, the measurement accuracy remains inadequate to inform clinical information. Some other studies adopted additional color calibration, image segmentation and feature extraction steps to preprocess the data to retrieve more spectral information from acquired color images [18–20]. Though accuracy has improved, the added operations often require professional interventions that need to be accomplished off-line. In this case, the smartphone is simply used as a data collection unit for experts rather than ready-to-use customer devices, which challenges its utilities to serve the general public.

Multispectral imaging is capable of maximally recording the spectral information of subjects, thus being widely used in conducting life science research, contributing to public healthcare services [21–33]. Realizing this technique on smartphones would create another space for exploitation to benefit our community, given the large user base, high usage frequency and low cost. Here, we describe a beta-version of mobile app, termed as SpeCamX, aimed to transform smartphones into multispectral imagers without any additional hardware attachments or internal modifications. With the SpeCamX, we enable the unmodified smartphone to provide a 27-channel multispectral data cube ranging from 420 to 680 nm by a single snapshot. From the multispectral data cube, the SpeCamX provides multiple functions to estimate corresponding chromophore levels, including hemoglobin, pigmentation, bilirubin, etc. To show the performance of this app and method, we installed SpeCamX on an unmodified smartphone and used it as a bilirubinometer to quantify the sclera pigmentation at the region of bulbar conjunctiva to predict BBL (Fig. 1). In brief, the multispectral data cube of sclera is captured to derive the reflectance spectra, which is used as the input of a hybrid prediction model trained using artificial neural network (ANN), support vector machine (SVM), k-nearest neighbors (KNN) and random forest (RF) algorithms. In the clinical imaging of 320 liver disease (LD) patients, the SpeCamX-enabled prediction demonstrated high correlation (R > 0.91, p < 0.0001) with the standard clinical testing (which is however invasive). To show the benefits, we compared the prediction using SpeCamX-enabled spectrally augmented learning (SAL) with RGB-enabled learning (RGBL) using RGB photographs captured by smartphone snapshots. Experimental results demonstrated that SpeCamX-enabled SAL delivers higher prediction quality, efficiency and stability than RGBL, especially when the data feeding was limited. Without additional customized hardware, our strategy has the potential to be widely distributed in an extremely cost-effective and easy-to-use mode. Providing tests of BBL and other bio-chromophores in this manner would be particularly useful for the users at both the resource-limited and homecare settings.

 

Fig. 1. Schematic of SpeCamX-augmented smartphone bilirubinometer. The housed flashlight in a smartphone illuminates the sclera region of the bulbar conjunctiva in patients. The SpeCamX converts RGB photographs into multispectral data cubes ranging from 420 to 680 nm that are used to derive the reflectance spectra of sclera, which is then used to predict the BBL in patient through an incorporated regression model hybridized by four machine learning algorithms. The predicted value is immediately displayed on the smartphone screen.

Download Full Size | PDF

2. Methods and materials

2.1 Development of SpeCamX

We calculated and embedded several transformation matrices (TMs) in SpeCamX as default options for different phone models, including those from Google, Apple, Samsung and Huawei. For this purpose, we acquired RGB photographs of X-rite ColorChecker Classic with corresponding smartphones under the illumination provided by their flashlights. When doing so, the color chart should be placed at a suitable distance with a smartphone to prevent overexposure (Maximum RGB value > 250) and under exposure (Maximum RGB value < 100). From the photographs, we sampled and calculated averaged RGB values of 24 color blocks. Then, we calculated the TM from the following equation:

SpeCamX was developed using an open-source integrated development environment (IDE) Android Studio (Google, CA). The Weiner estimation algorithm and default TMs developed on Matlab R2021b were incorporated into the SpeCamX. The “Imager” fragment can invoke the built-in camera and use it under default settings (Resolution: 2268 by 4032 pixels; f/1.7; shutter speed: 1/60; white balance: 5500 K).

2.2 Study design and population

To demonstrate its utility, we employed SpeCamX as a bilirubinometer to measure and quantify the blood bilirubin levels in patients with liver diseases. In this study, a total of 320 patients with liver diseases were enrolled between September 2021 and April 2022. Their diagnostic information was shown in Table 1. Anonymized and de-identified sclera images at the anterior segment of the eye were collected from the First Hospital of Jilin University (Jilin, China) using SpeCamX. The sample size was selected based on previous experience using a smartphone bilirubinometer [16–20], which is larger than the required size derived from a power analysis so we can investigate the learning efficiency of SAL and RGBL in different sample sizes. We randomly selected the patients with liver diseases and included all the samples with accessible images of sclera regions. Blinding was applied during all data analysis. This study adhered to tenets of the Declaration of Helsinki and was performed in accordance with the Health Insurance Portability and Accountability Act. This study was reviewed and approved by the Ethics Committee of the First Hospital of Jilin University. Informed consent was obtained from the subject prior to the start of each study session.

Table 1. Diagnosis profile of subjects in clinical assessment.

View Table

Before imaging, the patient was asked to lie down in the bed and keep their eyes open. The operator held the smartphone that was pointing to the sclera tissue in the front eye segment. Then, the patient was asked to blink before taking each photo of the targeted region on the sclera. The photographing was conducted in a dark ward and the subject was illuminated by the smartphone flashlight, so no further color calibration is needed. Occasionally, some trials were conducted with some residue ambient light or room light. In this case, the smartphone was recalibrated on-site with a standard color chart. With the SpeCamX, we enable the unmodified smartphone to provide a 27-channel multispectral data cube ranging from 420 to 680 nm from each single snapshot. Ten snapshots were taken for each participant. For each snapshot, ten ROIs were selected. The final BBL value was averaged from these one hundred measurements. To show the benefits, we compared the prediction using SAL with RGBL using RGB photographs captured by smartphone snapshots. Within six hours after imaging, the participants were subjected to standard clinical blood sampling to obtain their BBL.A 3 mL blood sample was drawn from subcutaneous veins in the arm. Blood samples were analyzed by the diazo method using the Beckmann biochemical analysis system (Beckman Coulter Inc., CA) at the clinical pathology laboratory.

2.3 Machine learning model

We compared SAL and RGBL in four commonly used machine learning regression algorithms, including ANN, SVM, KNN and RF. We applied these regression algorithms by invoking functions in the Statistics and Machine Learning Toolbox in Matlab R2021b. In ANN algorithm, we used fitrnet to train a ten-layer, feedforward, fully connected neural network to predict BBL. Function fitrsvm fits a support vector regression model. Function kNNeighborsRegressor estimates the values of a continuous variable (target) based on one or more independent variables (predictors) based on KNN algorithm. In the RF algorithm, we used TreeBagger to combine the results of many decision trees to provide predictions. While designed with different rationale and architectures, these four algorithms were selected because they represent commonly used machine learning methods and are all appropriate for training regression models. In this way, the generalizability of SpeCamX-enabled augmentation can be tested. Afterwards, a hybrid regression model was built by linearly combining these models and incorporated in SpeCamX since hybrid machine learning has the advantages in reducing biases and increasing accuracy. The output of the hybrid model can be previewed as the “BILI”‘ value in the “Analysis” fragment of SpeCamX.

2.4 Statistical analysis

To assess the prediction quality, Pearson’s correlation coefficients were calculated between the blood test bilirubin levels and the prediction using SAL and RGBL. Correlations were considered significant if p < 0·001. 95% prediction bands were computed for the correlation plots. The bias (MD) and 95% limits of agreement (LOA) were computed for the Bland-Altman plots. LOA was computed as 1.96 times the standard deviation of the error. To create ROC curves, 320 cases were classified into positive and negative groups using 17.1 µmol/L as the threshold. All the plots and curves were generated using Origin 2021b.

To compare the prediction performance of models using SAL and RGBL, default settings and parameters of functions were used in both methods to keep the consistency of conditions. In RGBL, averaged RGB values from the pixels in the ROI were used as the input dataset. In SAL, the averaged reflectance spectra were used as the input. To avoid overfitting, the prediction of each group was obtained by ten-fold cross-validation. 320 cases were randomly split into ten sets. In each round, the model was trained using nine of ten sets, and then tested on the remaining one set. In ten rounds, every data set needed to be tested for once. The final prediction was averaged from predictions of ten rounds. To challenge the prediction models with reduced data feeding, the sample subjects were randomly selected by an algorithm developed on Matlab R2021b. We tested sample sizes with data resampling percentage changing from 12.5% (n = 40) to 100% (n = 320) with a step width at 6.25% (n = 20). Under each sample size, RGBL and SAL models were trained with the same dataset. To avoid overfitting, the prediction of each group was obtained by ten-fold cross-validation as well.

3. Results

3.1 Development of SpeCamX

Many proof-of-concept studies were attempted where the strategy was to develop additional hardware attachments to enable the smartphone to acquire multispectral information [34]. In these attachments, diverse wavelength selection units can be designed, like light sources at appropriate wavelengths or tunable optical filters. However, asynchronous data acquisition at different wavelengths would inevitably cause co-registration error in the data cube which consequently contaminates the analysis outcomes. More importantly, the required investments in these attachments may also pose an obstacle on their practical applications.

In our previous study, we described a strategy to reconstruct multispectral information directly from skin photographs in RGB format [10]. Before imaging, the RGB camera was calibrated to obtain a transformation matrix (TM) using Wiener estimation algorithm. Then, using the TM, any acquired RGB photographs can be reconstructed into multispectral images. Following this strategy, we herein integrated the workflow into SpeCamX to implement a multispectral imager on an unmodified smartphone. The app was developed on Android 11 platform (Google Inc., CA). Currently, we installed it in the Google Pixel 4 smartphone (Google Inc., CA) for demonstration, but any type of smartphone can be used.

SpeCamX can automatically convert a built-in RGB camera in a smartphone into a multispectral imager. The whole app platform consists of four functional fragments, with their interfaces illustrated in Fig. 2. The “Imager” fragment (Fig. 2(A)) allows the users to set up required functions before imaging. The users can choose a default setting by selecting the model of their smartphones or enter the customized model to select the type of color chart for recalibration if the smartphone is not in the default list. Fig. 2(B) shows the drop-down menu of options in the “Imager” fragment. Fig. 2(C) shows the recalibration page. After the setting, users can launch the imager by tapping the camera icon in Fig. 2(A). All the recorded datasets can be searched in the “Records” fragment (Fig. 2(D)). The “MSI” fragment (Fig. 2(E)) allows the user to scrutinize the reconstructed spectral images of the subject if needed. Embedded algorithms in SpeCamX automatically process the spectral data cubes to extract the features of interest and present the results in the “Analysis” fragment (Fig. 2(F)). For example, when the smartphone is used as a bilirubinometer, the “BILI” represents the prediction value made from the region of interest (ROI) in the “Bili” page. Users can acquire these predictions either in the real-time preview mode or after imaging. Except for BBL, other functions like blood perfusion and pigmentation, can also be accessed in the drop-down menu on the top of the interface. The workflow details of these fragments can be found in Supplement 1 (Section 1, Figs. S1 to S6, and associated Visualization 1, Visualization 2, and Visualization 3). Overall, the SpeCamX possesses five key features: 1) No external attachments or internal modifications to the smartphone are necessary, thus SpeCamX can be distributed to the users just like other apps in Google shops for example; 2) No further calibration is required when used in the default mode, but there is an option provided for users to calibrate their specific smartphones in a customized mode, which is particularly useful under un-controlled illumination conditions; 3) The customized recalibration can be simply realized by imaging commercialized standard color charts; 4) No offline operation is required, the imaging, processing and analysis functions are all integrated in the SpeCamX; 5) Except for the BBL detection, other functions, like mapping of blood perfusion and pigmentation, can be provided as well. These characteristics would ensure the compatibility and practicability of SpeCamX in mobile health applications.

 

Fig. 2. Interface and functions of SpeCamX which consists of four fragments. (A) “Imager” fragment to set up the camera in the smartphone. (B) Supported functions in the setup options. (C) Recalibration page to generate a customized TM with a standard color chart. (D) Interface of the “Records” fragment, where users can check the list of acquired data. (E) “MSI” fragment for the presentation of the acquired spectral images of subjects. (F) “Analysis” fragment to present the quantitative information of the extracted feature.

Download Full Size | PDF

3.2 Spectral imaging using SpeCamX

The retrieval of spectral information from an RGB camera relies on the TMs that are obtained during the calibration process and pre-stored in SpeCamX. Here we describe the performance evaluation for the spectral reconstruction realized by the pre-stored matrices. In this method, the default TM for the smartphone (Google Pixel 4, Google Inc., CA) was obtained from 24 Macbeth classic color blocks in the calibration step. We used a X-rite ColorChecker Digital SG (note that this was not the one used for calibration) as the target for evaluation. After the smartphone took the color photo of the ColorChecker, the app automatically reconstructed the spectra for both classic colors and skin tones. The comparisons of the standard and reconstructed spectra were shown in Fig. 3(A). The reconstructed spectra highlighted in red match very well with the references in both classic and skin tone colors. The root mean square error (RMSE) of each color block was measured to quantify the reconstruction accuracy (Supplement 1 (Section 2, Table S1)). The average RMSE of all color blocks is less than 0.04, indicating an accurate reconstruction of reflectance spectra from 420 nm to 680 nm from a single snapshot RGB image.

 

Fig. 3. Performance of spectral imaging using SpeCamX. (A) Spectral reconstruction test on a standard color checker (X-rite ColorChecker Digital SG). The pre-stored TM was applied on 24 classic colors and also 14 skin tone colors to estimate the reflectance. The reconstructed spectra were compared with the standard spectra on the right. (B, C) Spectral reconstruction of color block #7D under different ISO and color temperature settings. The color temperature is increased from 2500 K to 9000 K with a step width of 500 k. The ISO is set to be 880, 840, 800, 720, 640, 570, 500, 450, 400, 360, 320, 285, 250 and 200. (D, E) Standard deviations of signals in each channel of 96 color blocks with the alternation of color temperature and ISO settings. RGB values of color block #7D shows much higher standard deviations than the reconstructed spectra in this procedure. (F) Spectral reconstruction test on bilirubin phantoms. The concentrations of bilirubin are 0.00, 0.23, 0.47, 0.94, 1.88, 3.75, 7.50, 15.00, 30.00 mg/dL in phantom one to nine, respectively. The inserted figure shows the RGB photographs of phantoms which show progressively darker yellowish pigmentation from phantoms one to nine. The curves present the reconstructed reflectance spectra accordingly. The dotted line curves present the reflectance spectra measured by a standalone spectrometer. The reflectance at wavelengths around 460 nm shows gradual decrease from phantom one to nine because of the increased absorption of bilirubin. (G) The linear relationship between the reflectance reduction measured by SpeCamX and the bilirubin concentrations of phantoms. The inserted table shows the fitted equation and fitting errors.

Download Full Size | PDF

RGB values can be influenced by different illumination conditions and channel sensitivities, which may lead to inconsistent responses under different camera settings. To address this issue, SpeCamX provided both default TMs and recalibration options to stabilize the quality of spectral imaging. Here, we simulated some extreme conditions by adjusting the color temperature and ISO of the camera to challenge this stability. The X-rite ColorChecker Digital SG was again used in this evaluation and imaged under different settings in camera. The color temperature was increased from 2500 K to 9000 K with a step width of 500 k.The ISO was set to be 880, 840, 800, 720, 640, 570, 500, 450, 400, 360, 320, 285, 250 and 200, respectively. The RGB values and reflectance spectra of all color blocks in these procedures were recorded by SpeCamX. As an example, the results of color block #7D were shown in Fig. 3, B and C. The horizontal axis includes both RGB and multispectral imaging (MSI) channels, while the vertical axis represents the settings in camera. The RGB signals were normalized into the same scale as the MSI channels and mapped in the same figure. With the color temperature increased from 2500 K to 9000 K, we observed the signals in G and B channels remained relatively stable, but that in the R channel increased proportionally from the RBG images. In contrast, the SpeCamX provided relatively stable signals in all reconstructed reflectance spectral channels despite the change in the color temperature. A similar result can be observed in Fig. 3(C), where RGB values are increased with the increase of ISO but the SpeCamX provided consistent spectral reconstruction in all the channels. To quantify the consistency, we calculated the standard deviations of signals in each channel for all color blocks, shown in Fig. 3(D) and 3(E), respectively. After normalizing RGB values into the same scale as MSI signals, the averaged standard deviations in RGB channels were calculated to be ∼0.045 and ∼0.070 when changing the color temperature and ISO, respectively. The corresponding standard deviation values in MSI channels were calculated to be ∼0.015 and ∼0.013. Compared with the RGB values, the signals in MSI channels perform much lower standard deviations. These experiments demonstrate that the SpeCamX can reconstruct accurate spectral information of the sample with a high consistency under different device conditions and settings.

Next, we prepared a set of phantoms to further test the performance of SpeCamX to recover the spectral information of bilirubin. The bilirubin levels were 0.00, 0.23, 0.47, 0.94, 1.88, 3.75, 7.50, 15.00, 30.00 mg/dL in the phantoms one to nine. The acquired RGB photographs of these phantoms are shown in the inset of Fig. 3(F), deeper yellowish pigments with the increase of the bilirubin concentration. Their reflectance spectra were obtained by the SpeCamX and presented as the curves in Fig. 3. These spectra were normalized by the reflectance at 680 nm because the absorbance of bilirubin at this wavelength band is negligible. Compared with phantom one without bilirubin, other phantoms give lower reflectance around 460 nm and the rate of reduction is similar to that of the concentration. We calculated the values of rate reduction at 460 nm and mapped the points with their bilirubin concentrations, accordingly, shown in Fig. 3. There is a significant linear relationship between these two variables, verifying that SpeCamX can be used to detect and quantify the optical absorption of bilirubin.

3.3 Sclera imaging using SpeCamX to enable bilirubinometer

Figure 4 shows results of SpeCamX-enabled imaging on the sclera in the anterior segment of the eye (bulbar conjunctiva region [35]) in two representative clinical cases. Clinically, BBLs in the patients were measured at 27.0 and 368.9 µmol/L, respectively. From RGB photographs in Fig. 4, A and B, we can clearly observe darker yellowish color in the sclera of patients with higher BBL. By the side of the photographs, we presented every frame of the spectral images obtained by SpeCamX. In the wavebands from 420 to 480 nm, two cases show distinct signal strength differences in the sclera due to different levels of bilirubin concentration. With a further increase of the wavelength, the difference gradually decreases because the absorbance of bilirubin becomes negligible at the longer wavelengths. In the red bands above 650 nm, no significant absorption can be observed in both cases. Fig. 4, C and D illustrate the generation of the reflectance spectra averaged from two imaging trials. In each trial, we acquired ten snapshots at different regions of the sclera. From each snapshot, we calculated an averaged spectrum from the selected ROI. The reflectance spectra also support our above observation that the sclera tissue of the patient with higher BBL shows lower reflectance in wavebands from 420 to 480 nm. SpeCamX predicted the BBL of these two cases to be 30.5 and 380.5 µmol/L, respectively, agreeing well with the clinical testing results (see the prediction method in the next section).

 

Fig. 4. Examples of clinical imaging using SpeCamX as a bilirubinometer. (A, B) RGB and spectral images of cases with BBLs at 27.0 and 368.9 µmol/L, respectively. (C, D) Ten samplings on different regions of the sclera and the reflectance spectrums of cases in (A) and (B), respectively. The case with high BBL shows darker yellowish in the RGB photograph and lower reflectance in wavebands from 420 to 480 nm than the case with low BBLs.

Download Full Size | PDF

3.4 Prediction of blood bilirubin levels with machine learning

Machine learning is increasingly applicable in the medical context because of its excellent ability to recognize subtle pattern features on datasets [36]. The ability of algorithms to extract features which may not be sensitive to the human observer has been extensively explored for a wide variety of predictions including blood tests [37,38], Alzheimer's disease [39] and cardiovascular diseases [40], etc. Here, the rich but subtle information due to the chromophores embedded within the multispectral images [41–45] acquired by a smartphone would provide an excellent opportunity to develop a machine learning method to predict the concentration of that chromophore. Besides, as we can observe from Fig. 4, A and B, except for the yellow pigmentation, the sclera tissue region also shows redness because it is covered by a highly vascularized conjunctiva layer. This multiple-chromophore structure of human tissue is more complicated than the bilirubin phantom, which may be less interpretable using linear regression. We therefore developed a machine learning strategy embedded in the SpeCamX for the prediction of BBLs. Below, we present the measures of accuracy for the models constructed to create this prediction and contrast the interpretability of different algorithms.

320 patients with LD were enrolled in the study. The subjects show diversities in the gender, age and diagnosis of disorders (Table 1). RGB photographs and reflectance spectra of sclera, paired with BBL results obtained from clinical blood testing were included in the data set. Four established machine learning algorithms, including ANN, SVM, KNN and RF, were used to train regression models. While designed with different rationale and architectures, these four algorithms were selected because they represent commonly used machine learning methods and are all appropriate for training regression models. In this way, the generalizability of SpeCamX-enabled augmentation can be tested. Afterwards, a hybrid regression model was built by linearly combining these models and incorporated in SpeCamX since hybrid machine learning has been reported to be capable of reducing biases and increasing accuracy [46,47]. The output of the hybrid model can be previewed as the “BILI” value in the “Analysis” fragment of SpeCamX.

Figure 5, A-C shows the predictions obtained from the hybrid regression model incorporated in SpeCamX. In 320 cases, the model provided an excellent correlation between SpeCamX prediction and clinical BBL measurements with a R value above 0.90 (p < 0.0001). From the Bland-Altman plots (Fig. 5(B)), we can also observe small limits of agreement (LOA) (+119.90/-117.45 µmol/L) and bias (1.23 µmol/L). Fig. 5(C) shows the receiver-operating characteristic (ROC) curve of SpeCamX-enabled BBL prediction. The area under the ROC curve (AUC) was calculated to be 0.97(0.94-1.00), indicating that SpeCamX and its built-in prediction model can provide a reliable measurement of the BBL by simply taking color photos of the sclera tissue using a smartphone.

 

Fig. 5. Quantification and improvements of SpeCamX-enabled SAL prediction using the hybrid regression model. (A) The relationship between BBL and predictions using SAL. Red point: prediction using SAL; red area: 95% prediction band of prediction using SAL. (B) Bland-Altman plots of predictions using SAL. Red dotted line: 1.96 limits of agreement; red line: MD of prediction. (C) The ROC analysis of prediction using SAL. A bigger AUC indicates a better diagnostic. (D) The comparison between predictions using SAL and RGBL in different sample sizes (resampling percentages from left to right: 100% (n = 320); 75% (n = 240); 50% (n = 160); 25% (n = 80)). Here RGBL was conducted using the same hybrid model as SpeCamX. Red point: prediction using SAL; red area: 95% prediction band of prediction using SAL. Black point: RGBL prediction; black area: 95% prediction band of RGBL prediction; (E) Corresponding Bland-Altman plots of predictions in d. Black dotted line: 1.96 limits of agreement of RGBL prediction; red dotted line: 1.96 limits of agreement of prediction using SAL; black line: MD of RGBL prediction; red line: MD of predictions using SAL. (F-H) Quantification of R (F), MD (G) and STD of MD (H) between BBLs and prediction using SAL/RGBL along different resampling percentages from 12.5% (n = 40) to 100% (n = 320). Black curve: RGBL prediction; red curve: predictions using SAL; blue curve: difference between RGBL prediction and predictions using SAL.

Download Full Size | PDF

To validate our assumption that SpeCamX-enabled multispectral imaging can better predict BBL than conventional smartphone imaging, we compared the quality of prediction using SpeCamX-enabled SAL and RGBL. Besides, given richer information with higher spectral resolution, SAL should also learn quicker than the RGBL model. To validate this point, we also tested and compared the prediction performance while reducing the data feeding. Fig. 5, D and E show the predictions and Bland-Altman plots using SAL and RGBL, respectively. In the whole set of 320 cases, RGBL produced visually similar plots with SAL (Fig. 5(D)), but its Bland-Altman plots indicate a wider LOA (+136.63/-126.57 µmol/L) and bigger bias (5.03 µmol/L) (Fig. 5(E)). Further, when the sample size was reduced (Fig. 5(D)), the SAL prediction remains relatively stable and constant, whilst a deterioration of the RGBL prediction is observed (where the regression curve is seen gradually deviating and the prediction band is becoming wider). The corresponding Bland-Altman plots in Fig. 5(E) validated this observation. With the sample size decreased to 25% (n = 80), the LOA of SAL prediction is +114.22/-107.87 µmol/L, but the LOA of RGBL prediction is +166.69/-148.82 µmol/L. We tested more sample sizes with data resampling percentage changing from 12.5% (n = 40) to 100% (n = 320) with a step width at 6.25% (n = 20) and summarized the R values, mean difference (MD) and standard deviation (STD) of MD in Fig. 5, F and G. Overall, the SAL prediction shows higher R, lower MD and STD than RGBL in all groups.

The enhancement of SAL prediction over RGBL is benefited from the multispectral information brought by the SpeCamX, rather than by a specific design of the prediction model. To demonstrate this point, we further separately investigated the quality of predictions using SAL and RGBL implemented by individual ANN, SVM, KNN and RF algorithms. The input of SAL and RGBL are the spectra saved in SpeCamX and corresponding RGB values of ROIs. The predictions were detailed in Supplement 1 (Section 3, Fig. S7 and Section 4, Fig. S8). To summarize these comparisons, SAL improved the prediction quality to varying degrees, especially with less data feeding. To illustrate this point clearer, we quantified the prediction performance of SAL and RGBL with data resampling percentage ranging from 12.5% (n = 40) to 100% (n = 320) with a step width at 6.25% (n = 20). The R, MD and STD were then measured and presented as curves in Fig. 6. The evolution curves showed that the R of SAL always remained at high levels of around 0.90 (p < 0.0001), even when only 12.5% (n = 40) of the data was used to train the model. On the contrary, the R value of RGBL can be even lower than 0.6. In all methods except for SVM, the prediction biases of SAL are close to zero, smaller or at least comparable to RGBL predictions. In SVM, the bias of SAL is increased, but still averagely 55% (16.87 ± 3.27 µmol/L) smaller than that of RGBL. The STD of MD in SAL slightly increases with smaller sample size but overall lower than 75 µmol/L, which is almost the best level the RGBL can achieve. Fig. 6, E and F show that the AUC value of SAL in all models is above 0.94, which outperforms that of RGBL in all algorithms. Besides the enhancements exist in absolute values, the red SAL curves in Fig. 6 seem to perform higher stability in all groups than RGBL curves. We quantified the standard deviations of these four indices in all groups with different sample sizes and algorithms (Supplement 1 (Section 5, Fig. S9)). The standard deviations of R, MD, STD and AUC in RGBL prediction are 72% (0.06), 50% (6.80 µmol/L) and 66% (11.74 µmol/L) higher than those in SAL prediction. All these merits demonstrate that SpeCamX-enabled SAL can significantly augment the BBL prediction quality regardless of the learning algorithms used, which indicates that its superiority originated from the multispectral imaging capacity in SpeCamX. In this case, it is reasonable to expect that any machine learning augmentation can be added when deploying SpeCamX for applications other than bilirubin level detection as demonstrated here.

 

Fig. 6. Quantification and improvements of SpeCamX-enabled SAL prediction using single machine learning models. (A-D) Quantification of R (left), MD (middle), and STD (right) of the BBLs predicted with SAL/RGBL when data feeding varied from 12.5% (n = 40) to 100% (n = 320) into different machine learning models: (A) ANN; (B) SVM; (C) KNN; (D) RF. Black curve: RGBL prediction; red curve: SAL prediction; blue curve: difference between RGBL prediction and SAL prediction. (E-F) the receiver-operating characteristic (ROC) analysis of prediction using RGBL (E) and SAL (F) using the single and hybrid machine learning model. Regardless of single or hybrid models, SAL achieves a better AUC than RGBL, which indicates a better diagnostic value for SAL-based SpeCamX. All the groups with different sample sizes are randomly selected from the data pool of 320 patients.

Download Full Size | PDF

4. Discussions

The development of mobile health based on smartphones facilitates daily monitoring of many vital signs and body tissue compositions, some of them have the potential to indicate disease conditions that are difficult to detect/diagnose without physically visiting healthcare providers [48]. Exploring and developing such techniques can clearly promote and benefit public healthcare. For example, the global incidence of LD is estimated at 1.5 billion, which leads to about 2 million deaths each year [49,50]. Close monitoring of the at-risk population is believed to be an effective strategy to control its progression and spread [51]. However, frequent testing through visiting the clinical labs imposes an inevitable burden to the patients, both psychologically and economically, impacting their compliance to seek for medical services. To improve the clinical compliance and promote their willingness to accept the monitoring of liver health conditions, one solution is to noninvasively detect bilirubin levels in the serum, preferably that can be performed in a non-clinical environment. The balance of BBL in the circulation relies on a normal liver metabolism, which makes it a suitable biomarker of liver functions [52,53]. At different severity stages of LD, bilirubin dysbolism accumulates and eventually causes different levels of hyperbilirubinemia, which usually appears as the yellowish pigmentation in body tissue [54,55]. With distinct optical spectral properties, bilirubin-induced pigmentation is suitable to be noninvasively measured using optical sensors to estimate the BBL and finally predict the liver condition. Some of these sensors, like transcutaneous bilirubinometer, equip spectral illumination for the detection of the light absorption to estimate BBL [56]. We also see some portable versions of these sensors that are developed to save people from the frequent clinical visits [57], however, substantial investment is still required to acquire these dedicated devices, which eventually prevents them from wider spread use.

Herein, we provided a strategy by developing a mobile application termed SpeCamX to transform smartphones into spectral imagers and tested this strategy in BBL prediction. SpeCamX can acquire multispectral data cubes through a single snapshot using a smartphone without external attachments. Besides, SpeCamX provides a set of pre-stored TMs for some popular smartphones and standard room conditions. This provision would save users from the requirements on the expertise of complicated color calibration and off-line processing. Moreover, after spectral imaging, SpeCamX uses the “Analysis” fragment in the app to analyze the chromophore information from acquired multispectral data cubes. Except for the bilirubin prediction discussed in this study, the SpeCamX also provides options to map the blood perfusion, melanin pigmentation and other chromophores within human tissue. Given that smartphones are already integrated terminals equipped with detector, processor and display units, the installation of SpeCamX will enable them to conduct selfie health screening and monitoring with a high independence and almost cost to nothing.

In this study, as a proof of concept, the output data cube of SpeCamX was set to be in 27 channels covering from 420 to 680 nm with a step width at 10 nm. We assumed that bilirubin and hemoglobin dominated the color of scleral tissue in the bulbar conjunctiva region (which is easily accessible by smartphone), thus the selected wavelength bands covered both the absorption peaks and troughs of these two chromophores. Moreover, the format of output data, including the spectral resolution and channel number, can be adjusted according to specific sample compositions and processing methods. For instance, the user can set up a more refined spectral resolution (i.e. to have denser spectral channels) around bilirubin’s absorption peak to maximally minimize the effect from hemoglobin content. Another example is in developing machine learning algorithms, a setting with more channels and higher spectral resolutions would be particularly useful to enable potential features to be fully learned. Furthermore, SpeCamX can also support non-learning algorithms by imaging at typical wavelengths, like 460 and 500 nm used in transcutaneous bilirubinometer. Overall, SpeCamX-enabled medical imaging possesses a high technical flexibility for further optimization and exploitation.

Through the clinical imaging of patients with LD, we demonstrated the enhancements on predictions realized by SpeCamX. Encouraged by this study, we believe SpeCamX may also act as an effective tool to monitor other bilirubin-related diseases, like neonatal jaundice. In fact, both infants and the guardians should benefit from the real-time, non-contact and non-invasive evaluation mode of this SpeCamX-augmented smartphone bilirubinometer. Further, if we see the bilirubin prediction described here as evidence, it should be reasonable to expect similar prediction quality when evaluating other chromophores. For example, SpeCamX should conceivably better describe hemoglobin behaviors than regular camera apps, which would strengthen the monitoring of blood and vascular abnormalities. In a word, SpeCamX has the potential to contribute to conventional smartphone-enabled medical imaging methods and improve their performance.

There are still some limitations in our study. Although we maintained the diversity of patients in gender and ages in the demonstration of SpeCamX augmented bilirubinometer, only Asian patients were enrolled and imaged. Being dominated by bilirubin and hemoglobin, the color of scleral tissue is not correlated with the complexions in subjects, but further investigation is needed. Besides this, the property of SpeCamX to estimate targets from more complicated backgrounds remains untested. Some human tissues may contain chromophores with more diversity and less predictability. Aiming to spread the usage of SpeCamX, we need to further explore its usability in these scenarios.

Looking forward, though our current study was conducted in clinics, a wider application of SpeCamX is expected for the public in their daily lives. In the future, we will keep working on the optimization, packaging and approval of this app and method to provide its open access. Through this way, we may conduct studies on the health care of a larger population with more diversity in regions, races, and complexions. Meanwhile, we will keep updating the embedded “Analysis” fragment by uploading more functions to empower this platform for the screening and monitoring of more health issues.

5. Conclusions

In conclusion, we have presented a mobile app and an augmented smartphone bilirubinometer that hold great promise for advancing medical diagnostics. The development of this smartphone-based system highlights the potential of using smartphones as multispectral imagers for bilirubin measurement, offering a convenient and cost-effective solution. Our clinical study has shown a strong correlation between the measurements obtained from the smartphone-based bilirubinometer and traditional laboratory methods. This demonstrates the accuracy and reliability of the system. Additionally, the use of SpeCamX for prediction has shown higher accuracy, efficiency, and stability compared to prediction based on regular RGB photographs. This indicates the potential of the SpeCamX-based approach for providing accurate healthcare assessments in various settings. The accessibility and affordability of smartphones make this technology highly promising for widespread adoption. By leveraging the existing hardware and computational power of smartphones, our solution has the potential to improve access to bilirubin monitoring, particularly in resource-limited areas.