1. Introduction

Due to its non-invasive property and frequently label-free nature, harmonic optical potential related techniques such as optical tweezers and optical waveguides have become increasingly attractive in numerous biological applications ranging from the manipulation of living cells [1] and individual virus/extracellular vesicles [2,3], to real-time observation of the dynamic processes of trapped particles by accurately measuring their precise displacement or piconewton trapping force. Furthermore, through high-speed observations of the diffusive Brownian motion of trapped particles and calculating their stiffness and time evolution of the mean square displacement (MSD), morphological properties below the optical diffraction limit such as the hydrodynamic diameter [4], as well as related physical quantities such as the refractive index [5,6] could be characterized. In addition, joint morpho-chemical information provided by combining laser tweezers with other optical detection modes (such as Raman [7], fluorescence [8], and others) are capable of achieving precise, quantitative chemical characterization of heterogeneous nanoparticles at the individual-particle level. All the techniques above require a precise and high-bandwidth measurement of the trapped particle positions.

The position of trapped particles has traditionally been measured using either position sensitive detectors [9] (such as quadrant photodiode (QPD) and position sensitive diode (PSD), etc.) or sensitive high-speed video cameras (such as sCMOS) [10] with particle tracking algorithms. Compared to the former that only provide only location information from a single detection spot, the spatially resolved high-speed camera offers a more comprehensive solution by observing the constrained Brownian motion process of single/multiple trapped nanoparticles simultaneously. Moreover, once their sizes were determined, refractive index distributions could also be extracted by inverting the scattering intensity recorded in images based on homogeneous or core-shell sphere Mie scattering models [11]. However, common high-speed cameras only offer the measurement rates up to a maximum of a few kHz, which leads to a conflict. Namely, strong stiffness traps are typically required to stably trap low index contrast, nanometer-sized particles such as liposomes or extracellular vesicles, yet the stronger the trap stiffness, the faster the instrument measurement speed must be to ensure that the measurement captures primarily ballistic motion. It has been proven that few kHz acquisition rates do not meet the full sampling measurement conditions of the diffuse Brownian motion for stable trapped particles under strong stiffness [4,12,13], which leads to overestimation of particle size due to inaccurate MSD measurement, and ultimately affecting the characterization accuracy of related physical properties such as refractive index [14].

Various methods have been explored for solving the sizing distortion problem in camera-based measurements, which can be mainly divided into three approaches. First, since the frequency of overdamped oscillatory motion is inversely proportional to the trapping stiffness, relatively weak stiffness traps could be applied for trapping and observing the diffusive Brownian motion of the nanoparticle, such that full-sampling lies within the acquisition bandwidth of camera [15,16]. However, this might sacrifice the trapping stability and scattering intensity of nanoparticles, and may be incompatible with other measurement modes demanding a strong laser (e.g. Raman spectroscopy). A second approach is to establish the calibrated constrained Brownian motion model by considering both the blurring and aliasing effect caused by the limited bandwidth of the camera measurement, and obtain a corrected size by fitting the experimental measurements to an adjusted model [17]. However, as the constrained model is, in general, dependent on particle size, the corrected fitting quality relies on the deviation between a guessed value for the particles and their actual size. This leads to a lack of precision when applying such models for the characterization of bio-nanoparticles with highly unknown and heterogeneous characteristics, which is typically the case for biological nanoparticles such as extracellular vesicles. Finally, deep learning algorithms such as the convolutional neural network (CNN) provide an efficient technical means to solve the sizing regression problem. It has been reported that by using a CNN, physical properties such as particle size, refractive index [18] or even fluid viscosity around particles [19] can be successfully predicted from time resolved displacement/force series of polystyrene and silica nanoparticles. Despite to improve the diffusion coefficient and size calculation accuracy of nanoparticles under free Brownian motion [20,21], as well as the autocorrelation function accuracy [22] under constrained Brownian motion, when considering size prediction under constrained Brownian motion, deep learning has been applied only to fully simulated datasets, and its exploration on experimental measurements (particularly for biological nanoparticles) remains to be demonstrated.

To address the above gaps, here we describe a residual neural network (ResNet) to characterize the size of optically trapped nanoparticles based on temporally under-sampled position measurements acquired by a camera with a limited detection bandwidth. Our proposed network was trained using simulated trapped nanoparticles position series under strong stiffness traps. We experimentally demonstrate that our simulated data-driven network enables size prediction of standard nanoparticles as well as extracellular vesicles (EVs), resulting in more accurate size prediction results than other algorithms such as adjusted Lorentzian fitting method or CNN-based networks. We further explored the effect of the time series length on the networks performance and demonstrate that accurate particle sizes could be predicted even with short data lengths, providing substantial improvements in measurement speeds for real experiments. Lastly, by expanding the network to human platelet-free plasma (PFP)-derived vesicles, we found the predicted sizing results are in close agreement with gold-standard nanoparticle tracking analysis (NTA) characterization. To highlight the unique capacity of high-speed camera based optical tweezers measurements compared with traditional QPD measurements, we further quantify the refractive index of vesicles separated by different purification procedures by the combination of our proposed network and individual scattering images from the video series. Overall, this study offers a promising ResNet-based sizing method for studying the morphological characterization of optical trapped nanoparticles on single-particle level, obtaining more accurate results than other state-of-art algorithms on experimental measured datasets such as standard nanoparticles and EVs, and can be generalized to many other optical potential trapping measurements.

2. Materials and methods

2.1 Laboratory-built optical tweezers system and its data processing

A laboratory-built optical tweezers system was developed previously [13], with its detailed optical configuration description in the Fig. 1(a). Briefly, a 785 nm single-mode continuous wave Gaussian laser (TA Pro, Toptica, Germany, output power of 300 mW in front of objective) is used to create a tightly confined laser trap via a high NA water immersion objective (60x / NA = 1.27, Nikon Plan Apo SR IR), and a gradient attenuator was applied for power reduction during trapping measurement. Elastically scattered light is collected and directed towards a high-speed sCMOS detector (zyla 4.2, Andor, Belfast, UK) placed in a conjugate plane to the sample plane. An imaging lens (f = 500 mm) is coupled with the camera and the pixel to micron conversion factor of the sCMOS image is 0.0443 micron/pixel. The region of interest (ROI) was restricted to image the motion of trapped individual nanoparticle, which was selected to 20 × 60 pixels (corresponding to 0.89 µm × 2.66 µm in the sample plane), allowing images to be taken at 6500 Hz.

 

Fig. 1. Optical tweezers measurement system and data preprocessing. (a) Schematic of the optical tweezers system setup. A 785 nm laser (red) is used to trap nanoparticles and excite elastic and Raman scattering. A bright-field imaging path is shown in green, while an sCMOS camera collects the backward elastic scattering of trapped nanoparticles. (b) Acquired high-speed image of a weak-stiffness trapped single PS 200 nm nanobead with an overlaid trajectory recorded across ∼1s. The extracted 2D and 1D displacement are shown in (c), and then converted into variance vs. time-lag plots for size extraction as shown in (d). Black and Blue dots represent for measurement under weak and strong stiffness, respectively.

Download Full Size | PDF

To simplify the network training process, preprocessing, including extraction of 1D particle position from 2D images, is necessary. Firstly, we determine the sub-pixel level centroid of each particle in each frame and then re-link the individual nanoparticles between frames. Figure 1(b) shows an example of extracting 2D displacement from high-speed images of a weak-stiffness trapped individual 200 nm polystyrene (PS) nanoparticle. Next, the obtained time stack of 2D position was further reduced into a 1D position series along the direction of weakest trap stiffness (the blue-line axis shown in Fig. 1(c)) and use it as an input to a pre-trained network. As the particle’s motion is least constrained in this direction, it yields the particle motion characteristics that most closely approaches free Brownian motion. Comparison of size predictions along both axes are shown in Figure S1 in Supplement 1. To validate the performance of size prediction neural network on experimental data, size information was extracted from the same nanoparticle measured with reduced trap stiffness (10%∼20% of the above strong stiffness) and used as ground truth of training dataset. Generally, the relationship between time lag $\Delta t$ and position autocorrelation time ${\tau}$ were used determine the dynamic regime of trapped nanoparticles, the latter is written as,

2.2 Sample preparation and data processing of nanoparticles

NIST standard nanoparticles including 100 nm and 200 nm polystyrene (PS, Thermo Fisher, Massachusetts, USA) and 200 nm silica (Nanocomposite, San Diego, USA) nanoparticles were diluted with ultrapure water, then sonicated for 5 min and filtered using polycarbonate filters (Millipore) with 220 nm pore size before use. Additionally, EVs extracted from HN4 cell supernatant and human PFP were prepared by size exclusion chromatography (SEC) and ultracentrifugation, respectively. The detailed preparation steps of the EV samples are shown in the Supplement 1.

2.3 Generation of simulated time tracks of trapped nanoparticles

Considering the challenge of obtaining large datasets by single-beam optical tweezers with limited throughput, here we chose to instead train our network using purely simulated data. Based on the Monte-Carlo method [23], we simulated a large dataset of one-dimensional constrained Brownian motion of trapped nanoparticles to utilize as a training dataset. Since the nanoparticle moves during the recording of its position, its measured position is significantly affected by motion blur when the temporal camera bandwidth of the measurement is limited. Here we incorporate this blurring effect and subsequent positioning error into our simulations, with the simulated time series of individual trapped nanoparticle positions P(t) could be written as:

 

Fig. 2. Establishment of a simulated training dataset and the size prediction neural network. (a) The simulated training dataset under strong-stiffness (right) was established based on the Monte-Carlo method, including the camera’s blurring effect by time-averaging of position in each 10 frames (left), as well as the positioning error measured by tracking an individual 500 nm nanoparticle fixed on coverslip (middle). (b) The ResNet-50 architecture for particle size prediction. During training, the input to the training is the time series displacement under strong stiffness and the “ground truth” size information. The numbers (×3, × 4, × 6) under each residual block represents the number of times the three convolutional layers are repeated.

Download Full Size | PDF

Since size of nanoparticles in this work are much smaller than laser wavelength, stiffness K is primarily determined by the gradient force when ignoring the extremely weak scattering force (Taking PS200nm nanoparticle as an example, the calculated ${F_{sca}} = 6.1 \times {10^{ - 4}}$pN and ${F_{gra}}$=0.43pN respectively, ${F_{sca}}$ is ∼3 orders of magnitude smaller than ${F_{gra}}$), which can be written based on dipole approximation [27]:

After following the above steps, a simulated constrained Brownian motion time series data was generated as shown in Fig. 2(a). In our case, a 20000-series-large database was constructed to perform training of each ResNet network, each time series were composed initially of 6000 frames (corresponding to an experimental measurement about 1 second in length). The a priori known size of nanoparticles and their corresponding simulated time series of strongly-constrained Brownian motion are regraded as the output and input for the neural network training, respectively.

2.4 ResNet-based size prediction network for strongly trapped nanoparticles

We propose a single-output one dimensional ResNet50-based network [31] for size prediction, as shown in Fig. 2(b). Briefly, our proposed network is composed of five main stages (1×conv at the beginning, 16 × 3 conv in the residual blocks, and 1×fully connected layer at the end, making up the 50 layers), with shortcut connections between the input and output of every residual block. A convolutional block is added in the shortcut path such that input and output have the same dimension. The shortcut or skip connections result in great improvement in both performance and training time in deeper networks, and help to mitigate the diminishing performance observed in deep neural networks by allowing the gradient information to pass through the layers. Briefly, the initial convolution layer has 64 filters, with the kernel size of 5, which is followed by the batch normalization, ReLU activation layer, and a max pooling layer of size 1 × 3, with a stride of 2. Next are the different groups of residual blocks with default filters and kernel sizes as indicated by the different colors in Fig. 2(b). The loss function is the mean squared error loss, with L₂ regularization and dropout [32] applied to avoid overfitting.

We apportioned 90% of the simulated constrained Brownian motion of nanoparticles for the network training and the remaining 10% for validation, and employed a 10-fold cross-validation method to validate our model. The networks are trained using the Adam optimizer [33] for 50 epochs. The batch size equal to 32 was found to be empirically optimal and learning rate is set to 5e-5, The input to the network is the strong stiffness time series, and the ground truth size information. The network is constructed using the Tensorflow framework and Keras library, and training is accelerated with NVIDIA Geforce RTX-2080 GPU 8 GB RAM. Further, the network performance was evaluated by the root-mean-squared error of size prediction (RMSEP) on both simulated and experimental nanoparticles and extracellular vesicles datasets. An example of the training-testing loss curve during the network training is shown in Supplement 1 Figure S3.

3. Results and discussion

3.1 Simulation study and experimental validation of the size characterization network

Following training, simulated data of constrained Brownian motion from particles with different size and refractive index distributions (including 100 nm and 200 nm polystyrene (PS) standard nanobeads, 200 nm silica standard nanobeads and extracellular vesicles (EVs)) were generated as described above and used to evaluate the performance of our size prediction deep network. For each of these three different particle types studied, the validation dataset consists of 2000 time series examples that were generated independently of the data used for the training.

The performance for predicting size of different nanoparticles is shown in Fig. 3. As the camera bandwidth is not high enough to accurately sample diffusive motion, directly applying the Stokes-Einstein equation to the raw camera data yields very poor sizing results. These can be markedly improved by correcting the data and fitting with the adjusted Lorentzian fitting [17] (detail shown in Supplement 1 Figure S4). However, even better results can be obtained by both CNN-based deep learning (trained by the same simulated data, detail shown in Supplement 1 Figure S5), as well as our proposed ResNet, which has the optimum performance. It’s clearly shown that the ResNet achieved accurate precision and maintained a good degree of linear agreement in predicting the particle size across a wide range of sizes and nanoparticle types. The detailed root mean squared error of prediction (RMSEP) and mean absolute percentage error (MAPE) values are summarized in Table 1, including results from multiple different lengths of input time series. Following simulation validation, measurement of nanoparticles with the same sizes and materials as the simulation data were used to validate whether our proposed network training based on simulated datasets could generate accurate predictions from experimental data. It can be seen from Fig. 3(b) and Table 1 that experimental prediction error is slightly larger than simulated data in all cases, which points to the existence of data bias between simulated and measured data. This might be caused by the environmental flicker noise caused by instability of the laser source. Interestingly, we noticed from Fig. 3(b) that the ResNet performance on EVs seems to be degraded by the addition of a deterministic offset of the sizing predictions, which could be reduced by a simple linear regression (as shown in the Supplement 1 Figure S6 a). Therefore, despite the slightly increased error values, the size predictions are all highly accurate, with MAPE < 5% for both standard nanobeads and complex, highly heterogeneous EVs (shown in the second and fifth column in Table 1), which are close to the simulated results. Echoing the performance on experimental data, adjusted Lorentzian fitting and CNN-based deep networks showed worse performance. Detailed results of their fitting performance are shown in the Supplement 1.

 

Fig. 3. Single-particle level ResNet size prediction performance on both simulated (a) and experimental (b) datasets of polystyrene (PS) and silica standard nanobeads, and extracellular vesicles (EVs). Red lines represent lines of perfect agreement. (c) Averaged RMSEP for raw R-NTA, adjusted Lorentzian fitting, CNN and our ResNet sizing network. (error bars = mean ± STD).

Download Full Size | PDF

Table 1. RMSEP and MAPE of the particle size predictions by the ResNet on the simulated and experimental datasets considering the different time series length.^a

View Table

3.2 Exploring the effect of time series length on sizing performance

To further validate our ResNet prediction network, we evaluate its performance on reduced data lengths. To match the reduced input dimension, here we retrained network by using the same 20000 time series with either 3000 or 1000 frames extracted from the original full 6000-frame data sequences used in the original training. Figure 4(a) and (b) shows the effect of time series length on the ResNet prediction error using the experimental datasets. Meanwhile, linear regression model established on full frames data is extended to short-data-length conditions, the RMSEP versus time series length can be depicted in Fig. 4(c), with their specific values reported in Table 1. Traditionally, a certain number of frames (∼1s acquisition time) is required to obtain robust size predictions of nanoparticles under the diffusive-regime of constrained Brownian motion, the prediction accuracy decreases significantly with fewer data points (as shown in the Supplement 1 Figure S7). It can be seen from Fig. 4(c) that the EVs size estimation errors of our ResNet network have increased by 0.6 and 1.4 times when using only 1/2 or 1/6^th of the full frames, respectively, indicating that the network is difficult to maintain performance after reducing the number of training data points. However, despite this performance degradation, our purposed network output still obtains substantially more accurate results compared to adjusted Lorentzian analysis and CNN, as well as the weak-stiffness trapping measurement. Furthermore, shortening the data lengths from, say, 6000 to 1000 also may allow us to increase the amount of training data, as now the original 20000 simulated tracks with 6000- frames each could be converted to 120000 tracks with 1000- frames each. These vectors could then be independently used as training data. The experimental results indicate that the RMSEP/ MAPE of EVs size prediction could be modestly improved from 18.33/8.6% to 17.2/6.5% respectively by increasing the amount of training data in this fashion, with the detail shown in Supplement 1s Figure S8.

 

Fig. 4. (a-b) Effect of time series length on ResNet prediction performance using experimental datasets of polystyrene (PS), silica standard nanobeads, and EVs using half and 1/6^th of the full 6000 frames. (c) RMSEP versus time series length on EVs experimental data using different sizing methods. (error bars = mean ± STD).

Download Full Size | PDF

3.3 Size and refractive index characterization of human PFP-derived Vesicles

To validate the generalization ability, our proposed network is further applied to the analysis of vesicles derived from identical human PFP samples, but prepared with different purification methods (size exclusion chromatography, SEC and ultracentrifugation, UC). SEC and UC are both common purification methods for EVs [34], yet yield purified populations with slightly different size and refractive index distributions due to SEC selecting EVs based on size, and UC selecting based on density, as well as the “gentler” SEC purification limiting formation of particle aggregates. Further, a highlight of the high-speed camera-based optical tweezers measurement is its ability to extract multi-parametric information, such as refractive index of nanoparticles by fusing quantitative size and scattering images together. Similar to our previous work [13], by making the simplifying assumption that vesicles are structurally homogenous nanoparticles, Mie theory could be employed to determine the refractive index of each single vesicle, as details described in Figure S9 in the Supplement 1. Size distributions of SEC and UC-derived vesicles predicted by our proposed ResNet network were shown in Fig. 5(a), demonstrating extremely good agreement with results from gold-standard NTA measurements. The detailed average sizes are summarized in Table S2, we note that the UC derived vesicles have a wider distribution with larger mean sizes compared with SEC, which may be due to the aggregation effect caused by UC preparation [35]. In addition, the average refractive index analysis results of EVs yield distributions generally in the recognized EVs distribution range of 1.4-1.42. However, there are still around 20% of vesicles with refractive index above 1.45. These vesicles are likely not EVs but residual lipoprotein contaminants (such as very-low density lipoprotein, VLDL particles), which generally have higher refractive index than EVs [36] and are commonly observed in SEC purified PFP samples [37]. In contrast, these high refractive index vesicles are rarely present in UC preparation samples, which provides higher purification efficiency.

 

Fig. 5. Morpho-chemical characterization of PFP-derived EVs (total of 66 and 50 for SEC and UC respectively) based on ResNet size predictions. (a) Histogram of particle sizes predicted by ResNet (bins) and NTA (curves) for SEC (blue) and ultracentrifugation methods (black), respectively. (b) RI distributions vs. size, determined by the joint combination of scattering images and ResNet size predictions. Inset shows the RI distribution of SEC (blue) and UC (black) purified vesicles.

Download Full Size | PDF

4. Conclusions

In this paper, we have presented a ResNet-regression-enabled size prediction model based on imaging measurements of nanoparticles in optical tweezers, with results showing that the network accurately outputs the sizes of both simulated and experimental nanoparticles under strong trapping forces, which is challenging or impossible to achieve using standard solutions such as via a constrained Brownian model. The strong-stiffness trap regime is particularly important for measurements of small nanoparticles with low refractive-index contrast such as EVs. However, physics-based models do not perform well in these regimes, and data-driven methods such as deep learning offer a more promising approach for these complex non-linear regression problems. To demonstrate the utility of our method in these difficult situations, we explored the performance of our proposed network on experimentally measured bio-nanoparticles such as EVs, showing that our method substantially outperforms other state-of-art algorithms such as CNN and adjusted Lorentzian fitting methods.

Importantly, the training dataset was generated completely using simulated data, which enables the network to be rapidly and easily trained for different experimental settings and expected particle size and refractive index distributions. Additionally, experiments on EVs show that the ResNet network possesses a good generalization capability, where vesicles from different biological sources and from different preparation protocols were accurately predicted using a single trained network. Importantly, considering the sensitive sCMOS detection of weakly scattered light as well as the combination of size prediction network with scattering images measured by camera, optical tweezers measurement enables the ability to determine the refractive index of individual trapped vesicles according to Mie theory, which is challenging for methods based on position-sensitive sensors. Experimental results indicate that >100 plasma derived vesicles predicted by ResNet show strong agreement with measurements determined from gold-standard NTA, yielding both size and refractive index distributions in the range of 80-250 nm and 1.35-1.59, respectively. Considering the differences in physical characteristics between impurities (lipoproteins etc.) and EVs in PFP derived samples, simulated training datasets with larger sample numbers and more widely varying physical parameters could be employed in the future to reduce generalization error and obtain more accurate and reliable size predictions.

The essence of this work is aimed at solving the under-sampling problem in camera-based particle tracking through a ResNet network, which confirms the superiority of deep learning in solving complex nonlinear fitting issues. Wide potential exists for our network to be integrated with various optical tweezers-related characterization techniques, such as laser tweezer Raman spectroscopy (LTRS), etc. Our network could also possibly be applied for simultaneous multiplex sizing when using a high-speed camera and an integrated multi-focal tweezers system. Ultimately, then, we anticipate that further improvements in our proposed methods will find wider applications in any area where characterization of small, low-RI-contrast nanoparticles plays an important role, especially in EVs, liposomal nanomedicines etc.