Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

Optical projection tomography reconstruction with few views using highly-generalizable deep learning at sinogram domain

Open Access Open Access

Abstract

Optical projection tomography (OPT) reconstruction using a minimal number of measured views offers the potential to significantly reduce excitation dosage and greatly enhance temporal resolution in biomedical imaging. However, traditional algorithms for tomographic reconstruction exhibit severe quality degradation, e.g., presence of streak artifacts, when the number of views is reduced. In this study, we introduce a novel domain evaluation method which can evaluate the domain complexity, and thereby validate that the sinogram domain exhibits lower complexity as compared to the conventional spatial domain. Then we achieve robust deep-learning-based reconstruction with a feedback-based data initialization method at sinogram domain, which shows strong generalization ability that notably improves the overall performance for OPT image reconstruction. This learning-based approach, termed SinNet, enables 4-view OPT reconstructions of diverse biological samples showing robust generalization ability. It surpasses the conventional OPT reconstruction approaches in terms of peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics, showing its potential for the augment of widely-used OPT techniques.

© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement

1. Introduction

The accurate visualization of structures plays a crucial role in advancing biomedical research. Among the range of imaging modalities available, computed tomography (CT) is renowned for its ability to reconstruct the three-dimensional (3D) structure of a target using a series of 2D image projections acquired from multiple viewing angles. Similar to X-ray computed tomography [13], OPT is particularly designed for imaging mesoscale biological specimens containing specific light-absorptive or fluorescently-labelled structures and has been widely applied to the 3D imaging of various model organisms, such as mouse embryo, zebrafish larva, and C.elegans [46]. In conventional OPT setups, achieving accurate tomographic reconstructions with high spatial resolution and minimal artifacts needs the acquisition of a substantial number of views, often in the range of several hundreds [79]. These views capture an extensive array of light rays from a wide angular range, typically spanning 180° or even wider. The acquisition of such a comprehensive dataset is crucial to ensure precise three-dimensional imaging, enabling researchers to obtain detailed information about the sample under investigation without significant visual distortions or noticeable artifacts. However, this process is obviously accomplished with long acquisition time, which cause low imaging speed and phototoxicity. In order to address this issue, one direct approach is to reduce the number of acquisition views, but employing a limited number of views in the filtered back projection (FBP) reconstruction method often leads to the emergence of streaking artifacts [10,11]. Compressed sensing (CS) based tomographic reconstruction has become a popular choice of recovering images from incomplete measurements [12,13], but the algorithm consumes a large amount of time. Furthermore, although CS can reduce the number of required acquisition views, it is typically limited to approximately fifty views, which still necessitates a considerable acquisition time. To further reduce the number of views, we introduce deep learning image enhancement algorithm [1416]. There are three main approaches: 1. end-to-end image enhancement in spatial domain, such as CNOPT,DRONE [17,18]; 2. completion of sinogram in sinogram domain, followed by reconstruction using traditional algorithms, such as LDCT [19,20]; 3. Simultaneous optimization of both domains, such as DRONE [21,22]. These three approaches can effectively reduce the streaking artifacts, but it has not been discussed which approach is optimal considering generalization and performance.

Here we propose an evaluation method based on deep learning to analyze the characteristics of the image domain and sinogram domain. We found the data complexity in the sinogram domain is lower, which is suitable for network to fit the mapping function and enabling robust reconstruction. Recognizing this advantage, we employ a neural network with a feed back initialization method to handle sinogram completion, called SinNet. To demonstrate its unique advantages, we quantitatively validated SinNet on both synthetic and experimental data. As demonstrations, SinNet is capable of performing high-quality reconstruction of zebrafish embryos with only four views, reducing the acquisition time by an impressive factor of 50.

2. Principle

In the field of biological imaging, significant variations in features are often observed among diverse samples, posing a challenge to the generalizability of deep learning-based reconstruction algorithms [23,24]. These disparities can impede the effective application of learned models across different domains. From the perspective of domain adaptation, the variances in image features across various domains play a crucial role in determining the generalization capabilities of deep learning models. Inspired by the field of computer image recognition, we first introduce a classification network (ResNet50) into this biomedical analysis [25]. We generated a training dataset containing 5290 2D slices from 22 nuclear-labeled zebrafish hearts, which were labeled from 1 to 22. During the training phase, the labeled training data was fed into the network. The network first computed the probability of these input sample, which was then used to calculate the complementary information entropy. Higher entropy values indicated instances that could not be reliably identified. Through a series of learning iterations, the network reduced the complementary information entropy of the training data from 1.0 to approximately 0.3. This reduction served as a metric for determining whether network has been successfully trained. We then applied the well-trained classifier to diverse samples, including simulated data from cardiomyocyte-labeled zebrafish hearts, simulated data from blood vessel-labeled zebrafish, and experimental OPT data from auto-fluorescence zebrafish embryos, to assess their similarity. We found the entropy was >0.9 at spatial domain, which indicated a huge similarity gap. In sharp contrast, when we translated them into sinogram domain, the entropy drastically decreased to ∼0.4, indicating a much smaller gap. Additionally, we performed cosine similarity calculations on the biological sample images in both the spatial and sinogram domain (Fig. 1). The results revealed a significantly higher cosine similarity among the different biological samples at sinogram domain compared to the spatial domain. This finding provides compelling evidence for a smaller gap within the sinogram domain.

 figure: Fig. 1.

Fig. 1. Evaluation of disparities between training and other samples. The top row displays spatial domain images, while the bottom row shows sinogram domain images. Complementary information entropy analysis reveals a substantial disparity among these samples in the spatial domain. However, upon transformation into the sinogram domain, this disparity diminishes significantly. Furthermore, the cosine similarity measurements between these samples corroborate this observation. (ZH: Cardiomyocytes-labeled zebrafish heart, BZ: Blood vessel-labeled zebrafish, AZ: Auto-fluorescence zebrafish embryo). Scale bar: 50 µm.

Download Full Size | PDF

Following our analysis, we choose to solve sparse view OPT as sinogram domain to achieve a high generalization reconstruction cause its high data similarity [27,28]. We have proposed a feed-back pre-processing sinogram completion algorithm in OPT reconstruction. By integrating the feed-back pre-processing step into the pipeline, SinNet exhibits superior optimization capabilities, which is shown in Fig. S1. The pre-processing algorithm plays a crucial role in refining the sinogram data prior to input into SinNet, effectively addressing any existing gaps or inconsistencies. The principle is illustrated in Fig. 2, which comprises three steps: data preprocessing, network training, and reconstruction. In part one, we first translate sparse-view OPT data into low-resolution sinogram and use BP-FP initialization method to up-sampling the low resolution sinogram. In part two, we send high resolution sinogram and the up-sampling sinogram to SinNet. With iteratively optimization, SinNet can effectively fit the complex mapping function between them. In the final part, SinNet demonstrates its exceptional generalization ability and performance by successfully completing the low-resolution sinogram from various samples. And the completed sinogram will be reconstructed by FBP to obtain high resolution slice. This capability highlights the network's capacity to handle diverse data sets and produce reliable and accurate sinogram completions. The robust generalization ability of SinNet plays a crucial role in achieving high-quality reconstructions and enables its broader applicability in the field of OPT imaging.

 figure: Fig. 2.

Fig. 2. Pipeline of SinNet. (a) Data pre-preparation and SinNet training. The samples are densely captured within 180°. Each slice is reconstructed by filtered back-projections and forward-projections to obtain the high resolution sinogram working as the label data of SinNet. Then the samples are captured at 45° intervals. The process of back-projection-projection is repeated to obtain the low resolution sinogram as the input of SinNet, calculating mean squared error of the SinNet output data and the label data, optimizing the SinNet parameters by gradient descent algorithm. (b) Data reconstruction. The sample outside the training library are captured at 45° intervals. The low resolution sinogram is obtained by back-projection-projection algorithm, which is input into the well-trained SineNet to obtain the high resolution sinogram. Then high quality image is reconstructed from the SinNet output data by FBP.

Download Full Size | PDF

SinNet is based on the U-net architecture, as shown in Fig. 3. U-net is a widely adopted neural network in biological applications for image segmentation, owing to its strong performance. With the aid of data augmentation, U-net showcases its rapid training speed and superior performance when training on limited amount of label data.

 figure: Fig. 3.

Fig. 3. SinNet’s architecture. The SinNet architecture we used included a contracting path of four blocks(encoders), with each block containing a 3 × 3 convolutional layers followed by a max pooling layer. Starting from 128, the number of features were doubled after each max pooling step. The expanding path also included four blocks(decoders), with each block up-scaling the features by a factor of two. And we connect the corresponding encoder and decoder to improve the ability of SinNet.

Download Full Size | PDF

3. Methods

3.1 OPT setup

A commercial white light source composed of a 5 × 6 LED lighting array and a slab diffuser (WorldView, Beijing, China) were used to provide uniform illumination over the entire sample. The fixed zebrafish embryo rotated along its vertical axis with a custom rotation stage. The transmitted light was collected using a 1/2” CMOS monochrome camera (EO-5012 M, Edmund Optics, New Jersey, USA) and a 2X telecentric lens (REV 02, Edmund Optics, New Jersey, USA). The exposure time was 30 ms.

3.2 Forward projection (FP) and filter back projection (FBP)

If the sample is within the depth of focus of a linear imaging system, OPT data can be analyzed using parallel projection in a manner analogous to X-ray CT data. The Forward projection (FP) model is expressed as follows:

$$Y = RX$$
where Y represents the measured 2D sinogram, R denotes the Radon transform, and X signifies the 2D fluorescence distribution of a slice. The determination of X can be achieved analytically through FBP. We performed the FP and FBP using the Astra Toolbox, a highly efficient and versatile open-source toolkit designed for tomographic projection and reconstruction [26].

3.3 Data preprocessing

Deep learning is a data-driven approach that relies heavily on training data for its performance. To assess the efficacy of the algorithm, we conducted experiments on both synthetic and real OPT data. Synthetic OPT data was generated by forward projecting high-quality 3D stacks captured by light-sheet microscopy to simulate the OPT imaging process. Dense angular sampling was performed to each sample, with typically 200-300 projections being captured within 180 degrees. Then sparse angular sampling was performed to obtain a low sampling-rate dataset. Then we rearrange the OPT data (view, x, z) to the sinogram (z, view, x). We used images from dense and sparse views as training image pairs. Similar to spatial-domain super-resolution methods, sinogram-domain super-resolution also requires up-sampling the low-resolution inputs. However, traditional methods using linear interpolation for angle up sample introduce artifacts and are not suitable for low sampling rates. To overcome these limitations, we propose a backward projection-forward projection (BP-FP) method that directly obtains the up sample sinogram from the sparse 2D projections without introducing artifacts. The BP-FP method reconstructs a low-quality 3D stack from the sparse 2D projections and performs dense projections on each slice to obtain the sinogram at the target resolution. In addition, we use the optimization idea of Simultaneous Iterative Reconstruction Technique (SIRT) to optimize this process iteratively, make it more close to HR-sinogram.

3.4 Domain gap evaluating

To evaluate the complexity of data from different domains, we employed the ResNet50 convolutional neural network as a classifier. To determine the training completion, we assessed the network's performance by evaluating its ability to correctly identify and classify the slices of the training images. Subsequently, we utilized the trained network to evaluate the data of interest and obtain a probability distribution that reflects the similarity of the data to the training dataset. This probability distribution served as a measure of the data's similarity and allowed us to compute the domain gap between the data and the training dataset.

To compute the domain gap and assess the similarity, we leveraged the concept of complementary information entropy. Specifically, we analyzed the level of entropy in the obtained probability distribution. High entropy indicated that the trained classifier struggled to recognize and classify the data, implying a low similarity between the data and the training dataset.

$$entropy = \mathop \sum \limits_1^m {p_i}lo{g_2}({{p_i}} )/\mathop \sum \limits_1^m ({1/\textrm{m}} )lo{g_2}({1/\textrm{m}} )$$

3.5 Network training

During the training process, we perform max-value normalization on the input images. The CNN extracts feature maps from low-quality images through convolutional layers, and predicts high-quality images based on them. We use the L2 norm as the loss function of the network to measure the difference between the output image and the ground-truth, and optimize the network parameters using gradient descent to effectively reduce the difference between the output and label images.

$${l_2} = \frac{1}{n}\mathop \sum \limits_{i = 1}^n {({{I_{i0}} - {I_i}} )^2}$$
where i is the pixel index. The L2-norm is averaged over each batch of 32 slices and back-propagated through the network to update the neuron weights. The Adam algorithm for stochastic gradient descent was used with the default parameters in PyTorch. The training of the convolutional neural network is performed on a computer equipped with a Nvidia 1080Ti and usually takes 4-6 hours. The trained Network can be generalized. For example, the network trained on 4-views still shows significantly better resolution and less artefacts in the reconstruction results compared to the FBP results as shown in Fig. S2. But the CNN needs to be trained separately for different sampling views to maximize the network performance

3.6 Evaluation methods

The normalized root mean square error (NRMSE) was used to evaluate the pixel-wise difference between the predicted image and the ground truth (GT). These two images were first normalized to about the same intensity range [0,1] by a percentile normalization, which ensured the difference of background would not affect the quantification. The NRMSE was computed as

$$\textrm{NRMSE} = {\; }\sqrt {\frac{1}{N}\mathop \sum \limits_{i = 1}^N {{({v_i} - {u_i})}^2}} {\; } \times 255$$
where N is the pixel number of the image (i.e., width × height) and ${v_i}$ and ${u_i}$ are the ith pixel intensity of the prediction image and the ground truth, respectively.

4. Results

4.1 High-quality OPT reconstruction with few views using SinNet

Imaging of the zebrafish heart is crucial for studying cardiac dynamics in biomedical imaging. To evaluate our proposed method, we conducted experiments on simulated zebrafish heart OPT data. We previously captured high-resolution 3D images of beating zebrafish heart hemodynamics using post-gated light-sheet microscopy technique [29]. Based on these 3D images, we generated synthetic LR and HR sinogram pairs using a simulated projection. After an image augment, the total amount of generated sinogram pairs for network training was 4290. We had four beating zebrafish hearts in total, with three of them being used for training and the last one being used for validation. After training, we use SinNet to artificially reconstruct images from 10, 8, 6, and 4 viewpoints, as illustrated in Fig. 4. Figure 4(c) shows the 3D reconstruction results of a simulated beating zebrafish heart obtained using SinNet and we compared them with the data reconstructed using the traditional FBP algorithm (Visualization 1). The results indicate that the images reconstructed using SinNet exhibit higher resolution and fewer artifacts in comparison to those obtained using the traditional FBP algorithm. Furthermore, the reconstruction quality is comparable to that obtained using the traditional algorithm with 200 viewpoints.

 figure: Fig. 4.

Fig. 4. SinNet reconstruction for simulated fluorescent zebrafish heart OPT data. (a) The reconstruction data of traditional FBP algorithm on 200 views,10 views, 8 views, 6 views and 4 views. The top row shows the single slice results, and the bottom row shows the corresponding sinogram. The reconstruction result from 200-views projection images is used as the ground truth. (b) The prediction results of the SinNet on 10 views, 8 views, 6 views and 4 views. (c) The 3D reconstruction from 200 views, 8 views and 8 views (SinNet). Scale bar: 50 µm

Download Full Size | PDF

4.2 Transfer learning ability of SinNet and other state-of-the-art methods

In order to demonstrate the transfer learning ability of our proposed method, we compared the recovery results of sparse-view OPT data using SinNet and other state-of-the-art methods including DRONE (simultaneous optimization of both domains), CNOPT and FBP-ConvNet (end-to-end image enhancement in spatial domain). To investigate the performance of SinNet in cross-modality recovery, we utilized the sparse view data and fed it into the pre-trained SinNet model. The model has been pre-trained on nuclear-labeled zebrafish hearts, allowing it to effectively learn and recover information across different modalities. The reconstructed results obtained using SinNet were compared with other algorithm. As shown in Fig. 5, our proposed SinNet provided remarkable enhancement for all types of input data. Qualitative analysis of the reconstruction quality showed that the SinNet results displayed spatial resolution similar to the ground truth FBP results obtained from dense view data. It is not surprising that CNOPT and FBP-ConvNet demonstrate limited generalization potential, likely because of the significant disparity within the image domain. While DRONE exhibits generalization capability close to our SinNet approach, it still requires sequential optimization in both sinogram domain and image domain, which is a suboptimal strategy as compared to simpler and more robust optimization in pure sinogram domain. Therefore, we recommend the application of suitable initialization in the sinogram domain due to their superior generalization ability and convenient training process. Additionally, SinNet also shows better performance than other sinogram method as shown in Fig. S3. Meanwhile the result verifies the rationality of our domain evaluation method.

 figure: Fig. 5.

Fig. 5. Cross-modality restoration by SinNet and other state-of-the art methods. The first column is the ground truth. The PSNR and SSIM compared to GT have been annotated. Scale bar: 50 µm

Download Full Size | PDF

4.3 Limited sampling learning ability of SinNet and other state-of-the-art methods

Neural network training often demands a substantial amount of training data, which can be challenging to obtain for OPT reconstruction of various types and labeled biological samples. Sparse-view OPT reconstruction poses the additional challenge of achieving training results with a smaller sample set. In this study, we demonstrated that data in the sinogram domain exhibits higher data similarity compared to data in the image domain. This observation suggests that training the network in the sinogram domain needs fewer training pairs. To verify this, we assembled a training set comprising 22 3D zebrafish heart images, which resulted in a total of 5290 training pairs. We conducted network training in both the image and sinogram domains, gradually reducing the number of training samples to examine the impact on network performance. To assess the impact of reducing the training data, we conducted experiments by reducing the number of training samples to one-eighth of the complete training set. Under such a condition, a significant quality degradation occurred in all the reconstructions by those image domain networks. In sharp contrast, our SinNet domain network exhibited stable performance even with the dramatically reduced training data (Fig. 6).

 figure: Fig. 6.

Fig. 6. Limited sampling learning ability of SinNet and other state-of-the-art methods. (a) Comparative evaluation of restored results using SinNet and other state-of-the-art methods under varying training data parameters. (b) Comparison of PSNR and SSIM against the GT across different training data parameters. Scale bar: 50 µm

Download Full Size | PDF

4.4 SinNet reconstruction for experimental data of zebrafish embryo

We conducted vascular staining experiments on zebrafish embryos to validate the effectiveness of the SinNet based method for processing OPT data. The experimental data were obtained by imaging 200 views of three wild-type zebrafish embryos at 3-days post-fertilization using an OPT system. Two of the samples were utilized for training, while the remaining one is used to validate our method. We generated low-resolution (LR) data as network inputs by resampling these 200 views to four views with an angle interval of 45 degrees. After augmenting the dataset, the total amount of training pairs is 4000. Figure 7 shows the reconstruction results and compares them with the results obtained using conventional FBP method (200 views), conventional FBP method (4 views) and dual-domain deep-learning approach DRONE. The reconstruction results clearly demonstrate that our SinNet reconstruction not only preserves the original high-quality signal but also delivers superior resolution with fewer artifacts. It should be also noted that DRONE exhibits a much larger parameter space than SinNet, requiring a substantial volume of training data for optimal performance. Given the constraints of our available training data, SinNet emerges as the more advantageous approach. These results demonstrate the effectiveness of the SinNet-based method in processing OPT data.

 figure: Fig. 7.

Fig. 7. Reconstruction for experiment OPT data of a zebrafish embryo. (a) 3D reconstruction by SinNet. (b) Comparison between GT, FBP (4views), DRONE and SinNet. Scale bar: 50 µm

Download Full Size | PDF

5. Conclusion

In conclusion, we have presented a generalizable sparse-view image reconstruction algorithm to address the limitations associated with the number of views in high-resolution volume imaging of diverse specimen types. While OPT is a valuable technique for three-dimensional imaging, its time-consuming and high-bleaching data acquisition method has impeded its application in dynamic processes and imaging of photosensitive samples. Our deep learning-based method can avoid the degradation of three-dimensional reconstruction images caused by sparse view projection images. Our method capitalizes on the inherent advantages of the sinogram domain, which exhibits lower data complexity and lends itself well to sparse-view OPT reconstruction algorithms with high generalization performance. Moreover, SinNet makes a substantial contribution to robust reconstruction, as demonstrated by its successful reconstruction of high-quality 3D stacks using only four views. In summary, this work represents a significant breakthrough in the acquisition and analysis of three-dimensional data from a wide range of specimens, particularly those with dynamic processes or photosensitivity. By introducing SinNet and its robust reconstruction capabilities, we have overcome the limitations of sparse view projection imaging and achieved remarkable results in terms of image quality.

Funding

National Natural Science Foundation of China (21874052, 21927802, 61860206009, T2225014); National Key Research and Development Program of China (2022YFC3401102).

Acknowledgments

The authors thank Zhaofei Wang (School of Optical and Electronic Information, Huazhong University of Science and Technology) for the help on the code.

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Supplemental document

See Supplement 1 for supporting content.

References

1. W. A. Kalender, “X-ray computed tomography,” Phys. Med. Biol. 51(13), R29–R43 (2006). [CrossRef]  

2. A. Momose, T. Takeda, Y. Itai, and K. Hirano, “Phase–contrast X–ray computed tomography for observing biological soft tissues,” Nat. Med. 2(4), 473–475 (1996). [CrossRef]  

3. D. J. Brenner and E. J. Hall, “Computed tomography—an increasing source of radiation exposure,” N. Engl. J. Med. 357(22), 2277–2284 (2007). [CrossRef]  

4. J. Sharpe, U. Ahlgren, P. Perry, B. Hill, A. Ross, J. Hecksher-Sørensen, R. Baldock, and D. Davidson, “Optical projection tomography as a tool for 3D microscopy and gene expression studies,” Science 296(5567), 541–545 (2002). [CrossRef]  

5. T. Alanentalo, A. Asayesh, H. Morrison, C. E. Lorén, D. Holmberg, J. Sharpe, and U. Ahlgren, “Tomographic molecular imaging and 3D quantification within adult mouse organs,” Nat. Methods 4(1), 31–33 (2007). [CrossRef]  

6. M. J. Boot, C. H. Westerberg, J. Sanz-Ezquerro, J. Cotterell, R. Schweitzer, M. Torres, and J. Sharpe, “In vitro whole-organ imaging: 4D quantification of growing mouse limb buds,” Nat. Methods 5(7), 609–612 (2008). [CrossRef]  

7. C. Vinegoni, L. Fexon, P. F. Feruglio, M. Pivovarov, J.-L. Figueiredo, M. Nahrendorf, A. Pozzo, A. Sbarbati, and R. Weissleder, “High throughput transmission optical projection tomography using low cost graphics processing unit,” Opt. Express 17(25), 22320–22332 (2009). [CrossRef]  

8. M. Rieckher, U. J. Birk, H. Meyer, J. Ripoll, and N. Tavernarakis, “Microscopic optical projection tomography in vivo,” PLoS ONE 6(4), e18963 (2011). [CrossRef]  

9. P. Fei, Z. Yu, X. Wang, P. J. Lu, Y. Fu, Z. He, J. Xiong, and Y. Huang, “High dynamic range optical projection tomography (HDR-OPT),” Opt. Express 20(8), 8824–8836 (2012). [CrossRef]  

10. S. Singh, M. K. Kalra, J. Hsieh, P. E. Licato, S. Do, H. H. Pien, and M. A. Blake, “Abdominal CT: comparison of adaptive statistical iterative and filtered back projection reconstruction techniques,” Radiology 257(2), 373–383 (2010). [CrossRef]  

11. M. J. Willemink and P. B. Noël, “The evolution of image reconstruction for CT—from filtered back projection to artificial intelligence,” Eur Radiol 29(5), 2185–2195 (2019). [CrossRef]  

12. E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm Pure Appl Math 59(8), 1207–1223 (2006). [CrossRef]  

13. X. Zhang, T. Bai, H. Meng, and J. Chen, “Compressive sensing-based ISAR imaging via the combination of the sparsity and nonlocal total variation,” IEEE Geosci. Remote Sensing Lett. 11(5), 990–994 (2014). [CrossRef]  

14. Y. Zhao, M. Zhang, W. Zhang, Y. Zhou, L. Chen, Q. Liu, P. Wang, R. Chen, X. Duan, and F. Chen, “Isotropic super-resolution light-sheet microscopy of dynamic intracellular structures at subsecond timescales,” Nat. Methods 19(3), 359–369 (2022). [CrossRef]  

15. Z. Wang, L. Zhu, H. Zhang, G. Li, C. Yi, Y. Li, Y. Yang, Y. Ding, M. Zhen, and S. Gao, “Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning,” Nat. Methods 18(5), 551–556 (2021). [CrossRef]  

16. M. Weigert, U. Schmidt, T. Boothe, A. Müller, A. Dibrov, A. Jain, B. Wilhelm, D. Schmidt, C. Broaddus, and S. Culley, “Content-aware image restoration: pushing the limits of fluorescence microscopy,” Nat. Methods 15(12), 1090–1097 (2018). [CrossRef]  

17. S. P. Davis, S. Kumar, Y. Alexandrov, A. Bhargava, G. da Silva Xavier, G. A. Rutter, P. Frankel, E. Sahai, S. Flaxman, and P. M. French, “Convolutional neural networks for reconstruction of undersampled optical projection tomography data applied to in vivo imaging of zebrafish,” J. Biophotonics 12(12), e201900128 (2019). [CrossRef]  

18. Y. Guo, T. T. Hormel, S. Pi, X. Wei, M. Gao, J. C. Morrison, and Y. Jia, “An end-to-end network for segmenting the vasculature of three retinal capillary plexuses from OCT angiographic volumes,” Biomed. Opt. Express 12(8), 4889–4900 (2021). [CrossRef]  

19. M. U. Ghani and W. C. Karl, “Deep learning-based sinogram completion for low-dose CT,” in 13th Image, Video, and Multidimensional Signal Processing Workshop (IEEE, 2018), 1–5.

20. N. Awasthi, G. Jain, S. K. Kalva, M. Pramanik, and P. K. Yalavarthy, “Deep neural network-based sinogram super-resolution and bandwidth enhancement for limited-data photoacoustic tomography,” IEEE Trans. Ultrason., Ferroelect., Freq. Contr. 67(12), 2660–2673 (2020). [CrossRef]  

21. W. Wu, D. Hu, C. Niu, H. Yu, V. Vardhanabhuti, and G. Wang, “DRONE: dual-domain residual-based optimization network for sparse-view CT reconstruction,” IEEE Trans. Med. Imaging 40(11), 3002–3014 (2021). [CrossRef]  

22. C. Zhang, J. Fu, and G. Zhao, “Learning from Projection to Reconstruction: A Deep Learning Reconstruction Framework for Sparse-View Phase Contrast Computed Tomography via Dual-Domain Enhancement,” Appl. Sci. 13(10), 6051 (2023). [CrossRef]  

23. C. Oare, S. Sun, K. Dusenbery, M. Reynolds, D. Koozekanani, B. Gerbi, and C. Ferreira, “Analysis of dose to the macula, optic disc, and lens in relation to vision toxicities–A retrospective study using COMS eye plaques,” Physica Medica 101, 71–78 (2022). [CrossRef]  

24. S. Khan, B. Song, S. Mallidi, S. Li, H. Liu, M. Bilal Hussain, S. Siddiqui, A. P. Khan, K. Akhtar, and S. A. Siddiqui, “Clinical assessment of a low-cost, hand-held, smartphone-attached intraoral imaging probe for 5-aminolevulinic acid photodynamic therapy monitoring and guidance,” J. Biomed. Opt. 28(08), 082809 (2023). [CrossRef]  

25. X. Liu and S. Zhang, “Who is closer: A computational method for domain gap evaluation,” Pattern Recognition 122, 108293 (2022). [CrossRef]  

26. W. Van Aarle, W. J. Palenstijn, J. De Beenhouwer, T. Altantzis, S. Bals, K. J. Batenburg, and J. Sijbers, “The ASTRA Toolbox: A platform for advanced algorithm development in electron tomography,” Ultramicroscopy 157, 35–47 (2015). [CrossRef]  

27. J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” J. Mach. Learn. Res. 7, 1–30 (2006).

28. A. P. Bradley, “The use of the area under the ROC curve in the evaluation of machine learning algorithms,” Pattern Recognition 30(7), 1145–1159 (1997). [CrossRef]  

29. Z. Guan, J. Lee, H. Jiang, S. Dong, N. Jen, T. Hsiai, C.-M. Ho, and P. Fei, “Compact plane illumination plugin device to enable light sheet fluorescence imaging of multi-cellular organisms on an inverted wide-field microscope,” Biomed. Opt. Express 7(1), 194–208 (2016). [CrossRef]  

Supplementary Material (2)

NameDescription
Supplement 1       Supplementary information
Visualization 1       The OPT image of cardiac dynamics are reconstructed by 200 views (FBP), 4 views (SinNet), 4 views (FBP).

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (7)

Fig. 1.
Fig. 1. Evaluation of disparities between training and other samples. The top row displays spatial domain images, while the bottom row shows sinogram domain images. Complementary information entropy analysis reveals a substantial disparity among these samples in the spatial domain. However, upon transformation into the sinogram domain, this disparity diminishes significantly. Furthermore, the cosine similarity measurements between these samples corroborate this observation. (ZH: Cardiomyocytes-labeled zebrafish heart, BZ: Blood vessel-labeled zebrafish, AZ: Auto-fluorescence zebrafish embryo). Scale bar: 50 µm.
Fig. 2.
Fig. 2. Pipeline of SinNet. (a) Data pre-preparation and SinNet training. The samples are densely captured within 180°. Each slice is reconstructed by filtered back-projections and forward-projections to obtain the high resolution sinogram working as the label data of SinNet. Then the samples are captured at 45° intervals. The process of back-projection-projection is repeated to obtain the low resolution sinogram as the input of SinNet, calculating mean squared error of the SinNet output data and the label data, optimizing the SinNet parameters by gradient descent algorithm. (b) Data reconstruction. The sample outside the training library are captured at 45° intervals. The low resolution sinogram is obtained by back-projection-projection algorithm, which is input into the well-trained SineNet to obtain the high resolution sinogram. Then high quality image is reconstructed from the SinNet output data by FBP.
Fig. 3.
Fig. 3. SinNet’s architecture. The SinNet architecture we used included a contracting path of four blocks(encoders), with each block containing a 3 × 3 convolutional layers followed by a max pooling layer. Starting from 128, the number of features were doubled after each max pooling step. The expanding path also included four blocks(decoders), with each block up-scaling the features by a factor of two. And we connect the corresponding encoder and decoder to improve the ability of SinNet.
Fig. 4.
Fig. 4. SinNet reconstruction for simulated fluorescent zebrafish heart OPT data. (a) The reconstruction data of traditional FBP algorithm on 200 views,10 views, 8 views, 6 views and 4 views. The top row shows the single slice results, and the bottom row shows the corresponding sinogram. The reconstruction result from 200-views projection images is used as the ground truth. (b) The prediction results of the SinNet on 10 views, 8 views, 6 views and 4 views. (c) The 3D reconstruction from 200 views, 8 views and 8 views (SinNet). Scale bar: 50 µm
Fig. 5.
Fig. 5. Cross-modality restoration by SinNet and other state-of-the art methods. The first column is the ground truth. The PSNR and SSIM compared to GT have been annotated. Scale bar: 50 µm
Fig. 6.
Fig. 6. Limited sampling learning ability of SinNet and other state-of-the-art methods. (a) Comparative evaluation of restored results using SinNet and other state-of-the-art methods under varying training data parameters. (b) Comparison of PSNR and SSIM against the GT across different training data parameters. Scale bar: 50 µm
Fig. 7.
Fig. 7. Reconstruction for experiment OPT data of a zebrafish embryo. (a) 3D reconstruction by SinNet. (b) Comparison between GT, FBP (4views), DRONE and SinNet. Scale bar: 50 µm

Equations (4)

Equations on this page are rendered with MathJax. Learn more.

Y = R X
e n t r o p y = 1 m p i l o g 2 ( p i ) / 1 m ( 1 / m ) l o g 2 ( 1 / m )
l 2 = 1 n i = 1 n ( I i 0 I i ) 2
NRMSE = 1 N i = 1 N ( v i u i ) 2 × 255
Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.