This PDF file contains the front matter associated with SPIE Proceedings Volume 11634 including the Title Page, Copyright information, and Table of Contents.

Welcome and Introduction to SPIE Conference 11634: Multimodal Biomedical Imaging XVI

We present a bi-modal bench-top system combining OCT with broadband, single-fiber reflectance spectroscopy. This combination aims to address the limited molecular sensitivity of standard OCT imaging in order to obtain co-registered morphological and molecular information. We present various technical innovations for this work, including an all-reflective scanner head with adaptive optic components for focus scanning and reduction of field curvature. Furthermore, we demonstrate the use of specialty fiber components to obtain multiple illumination schemes for the spectroscopic channel and enhance the spatially resolved reconstruction of optical properties.

Early identification of the margins and location of choroidal neovascularization (CNV) is critical for the precise diagnosis and treatment of numerous neovascular eye diseases, including age-related macular degeneration (AMD). Integration of multimodal photoacoustic microscopy (PAM) and optical coherence tomography (OCT) imaging has been developed to complement the strengths of each modality. A major challenge remains in selectively distinguishing CNV from native microvasculature due to the high optical absorption of hemoglobin. To overcome such limitations, RGD targeting peptides conjugated with gold nanorods (GNR-RGD) was used as multimodal contrast agents to increase the sensitivity of PAM and OCT, allowing for enhanced visualization of CNV due to RGD’s selective binding to integrins in neovascularization. The ability of GNR-RGD enhanced PAM and OCT imaging was evaluated in three New Zealand White rabbits with CNV models. The CNV model was created at day 28 post laser-induced retinal vein occlusion. In vivo color fundus photography, fluorescein angiography, PAM, and OCT imaging was acquired before and after intravenous injection of 400 μL GNR-RGD at concentration of 5 mg/mL at days 1, 3, 5, 7, and 14. Longitudinal studies show that GNR accumulated at CNV sites and led the PAM and OCT signal increased by 27.2-fold in PAM and 171.4 % in OCT peaking at 48 h post-injection and decreased at day 14. Histological analysis, TUNEL assay, and liver and kidney function tests show no systemic toxicity of GNR in the retina or vital organs. The above approaches can provide a potential multimodal molecular imaging tool for precise evaluation of CNV in AMD and other neovascular diseases.

We present a combination of sub-nanosecond two-photon microscopy (TPM) and megahertz-rate optical coherence tomography (MHz-OCT) via a double ferrule for future endoscopic setups. The double ferrule combines a Hi1060 fiber of the OCT system and a double cladding (DC) fiber of the TPM setup. The use of sub-nanosecond pulses for TPM simplifies the setup substantially as no dispersion management is required. The inner cladding of the DC fiber collects the fluorescent light. The double ferrule was tested in a microscope setup. We characterized our system and collected first imaging data.

We demonstrate that the lasing emission spectra of nanowire lasers internalized by progenitor retinal pigmented epithelial cells (RPE) can be exploited as unique “identifiers” to label each individual cell during long-time in vivo observation. Since nanowires could provide a 25 dB signal enhancement in optical coherence tomography (OCT) and green emission in fluorescence microscopy (FM), we utilized OCT and FM concurrently to track the 3D trajectories of RPE cells in rabbit retina in vivo migrating towards the laser-induced wounds. Our study confirms the feasibility of nanowire lasers as novel probes in single progenitor cell tracking, which could potentially facilitate the fundamental research in regenerative medicine.

Cherenkov light is emitted in response to therapeutic x-ray beam delivery for the treatment of breast cancer, and serves as a passive, non-contact approach for measuring optical signal that is intrinsically linear with dose. However, the intensity of emitted light is attenuated due to absorbers in the tissue (blood, pigment, radiodensity, etc.). If correction for this attenuation were possible, then absolute dose imaging would be feasible. In this study, the planning CT scan was spatially sampled over the area emitting Cherenkov, and the attenuation of the signal was corrected for, using CT radiodensity. There was a linear correlation between presence of fibroglandular (high HU) versus adipose (low HU) and the emitted Cherenkov light. This relationship was used to generate scale factors to normalize out existing tissue variability in images recorded during fractionated radiotherapy, which reduced patient-to-patient variability to under 10%.

The voxel-based Monte Carlo method (VMC) offers efficiency in modeling light transport in complex bio-tissues, but is known to produce erroneous results due to its terraced boundaries. We present a significantly improved VMC by incorporating mesh-based boundary information in a hybrid modeling approach. A fast preprocessing step first extracts surface meshes from an arbitrary voxelated domain using the marching-cubes algorithm. An extended voxel format is developed to encode oblique surface information while keeping the data structure efficient for parallel processing. This enables modeling of subvoxel boundaries, resulting in significantly improved accuracy in benchmarks, including an MRI human brain atlas.

Breast cancer is a highly heterogeneous disease comprising a variety of genotypes and phenotypes of varying levels of aggressiveness. This presents significant challenges to clinical management of early-stage cancers. In this paper, we describe the use of multimodal optical technologies including near-infrared (NIR) spectroscopy, diffuse correlation spectroscopy (DCS) and indocyanine green (ICG) fluorescence imaging to evaluate the aggressiveness and progression of two patient-derived xenograft models of human breast cancer. Optical markers reveal distinctive features between low- and high-aggressiveness tumors that could potentially be translated for clinical use.

Label-free multimodal nonlinear optical imaging and optical biopsies of fresh, unstained, resected tissue specimens offer a wealth of new biomarkers for assessing the tumor microenvironment and diagnosing disease. By developing widely coherent supercontinuum from photonic crystal fibers, new excitation wavelengths can be generated to tailor the light stimulus in new ways. As a result, Simultaneous Label-free Auto-fluorescence Multi-harmonic (SLAM) microscopy can visualize the rich intrinsic molecular, metabolic, and structural information in cells and tissues. Results suggest broad potential of this stain-free, slide-free, imaging technology and methodology for real-time point-of-procedure applications, including the histopathological assessment of living biopsy specimens.

Diffuse optical tomography, including fluorescence molecular tomography (FMT) have been greatly facilitated by the implementation of structured illumination (SI) strategies in recent years. In this work, we investigate the inverse problem in k-space reflectance fluorescence tomography. This in silico investigation leverages MCX, a Monte Carlo based platform, to generate large data sets for comparison between dAUTOMAP, a deep learning architecture, and commonly employed iterative solvers. We show that the proposed dAUTOMAP-based technique outperforms the traditional reconstruction algorithms. This new image formation approach is expected to facilitate imaging of sub-cutaneous tumors in live animals with enhanced resolution compared to the current gold standard.

Geographic atrophy (GA) is late-stage dry age-related macular degeneration (AMD). Improved predictors of GA progression would be useful in clinical trial design and may be relevant for clinical practice. The purpose of this study was to accurately predict GA progression over time from baseline fundus autofluorescence (FAF) images (Heidelberg Engineering, Inc., Germany) using deep learning. Study eyes of patients (n = 1312) enrolled in the Lampalizumab trials^{1, 2} (NCT02479386, NCT02247479, NCT02247531) were included. The dataset was split by patient into training (n = 1047) and holdout sets (n = 265). GA progression, defined as GA lesion growth rate, was derived by a linear fit on all available measurements of GA lesion area (mm2 , measured from manually graded FAF images). The model performance was evaluated using 5-fold cross-validation (CV). Coefficient of determination (R2 ) computed as the square of Pearson correlation coefficient was used as the performance metric. Multiple modeling approaches were implemented, and the best performance was observed using cascade learning. In this approach, pre-trained weights on ImageNet were finetuned to predict GA lesion area followed by further fine-tuning to predict GA growth rate. The 5-folds had an average CV R² of 0.44, and the holdout showed R² of 0.50 (95% confidence interval: 0.41 - 0.61). In comparison, a linear model using only baseline GA lesion area in the same holdout showed an R² of 0.18. Further investigation with visualization techniques might help understand the pathophysiology behind the predictions. The predictions may be improved by combining with imaging modalities like near-infrared and/or optical coherence tomography.

The FLS program is being embraced internationally for training and credentialing surgeons. Manual skills assessment is performed during a proctored exam. In spite of its growing popularity, there are several major problems with the manual skills component of the FLS. Herein, we will summarize our efforts in establishing functional Near InfraRed Spectroscopy (fNIRS) as a fast and robust method to support this segment of the FLS program, especially leveraging recent development in Machine Learning. Moreover, we will report on the potential of using concurrent video stream towards enhanced multimodal surgical skill assessment.

Despite the technological evolutions that transform the operating rooms nowadays, a major clinical need remains: surgeons need to distinguish healthy from diseased tissues while performing a procedure. Tissue status assessment procedures such as blood perfusion monitoring require objective input that can potentially be obtained with fluorescence imaging and oxygenation imaging. We developed a multimodal imaging platform for performing widefield quantitative oxygenation imaging and fluorescence imaging in a clinical environment. We demonstrate in-vivo the impact of widefield quantitative oxygenation imaging on blood perfusion assessment. Fluorescence imaging provided by the system is used in complement to confirm the outcome of oxygenation imaging.

Establishing adequate resection margins during colorectal cancer surgery is challenging. Currently, in up to 30% of the cases the tumor is not completely removed, which emphasizes the lack of a real-time tissue discrimination tool that can assess resection margins up to multiple millimeters in depth. Therefore, we propose to combine spectral data from diffuse reflectance spectroscopy (DRS) with spatial information from ultrasound (US) imaging to evaluate multi-layered tissue structures. First, measurements with animal tissue were performed to evaluate the feasibility of the concept. The phantoms consisted of muscle and fat layers, with a varying top layer thickness of 0-10 mm. DRS spectra of 250 locations were obtained and corresponding US images were acquired. DRS features were extracted using the wavelet transform. US features were extracted based on the graph theory and first-order gradient. Using a regression analysis and combined DRS and US features, the top layer thickness was estimated with an error of up to 0.48 mm. The tissue types of the first and second layers were classified with accuracies of 0.95 and 0.99 respectively, using a support vector machine model.

Multimodal imaging systems offer the opportunity to scrutinize different properties of biological samples. Optical coherence microscopy (OCM) is a non-invasive and high-resolution imaging technique capable of generating threedimensional images of tissue. In this work, a multimodal imaging system interleaving OCM with a dual-channel fluorescence microscopy (DC-FM) system was developed to add functional imaging capabilities to OCM. The combined system was able to simultaneously acquire both reflectance and fluorescence data from the same location of the sample at the speed of 250 kHz, and with a lateral resolution of ~ 2.1 μm. An axial resolution of 2.4 μm in sample over the imaging depth of 1 mm was achieved with OCM. The performances of the combined system were evaluated by imaging a multi-layer tape as well as a gel containing green and red fluorescent microspheres. While OCM enabled the depth localization of all fluorescent microspheres, it was not able to discriminate between green and red fluorophores, a feature that was achieved with DC-FM. Hence, the interleaved system has the potential of assessing structural as well as cellular level functional changes in biological samples. This system will be applied toward longitudinal studies in small animal models.

Early assessment of sub-surface bladder tumor extension is challenging as there are no acceptable imaging modalities to determine sub-surface three-dimensional (3D) tumor extension. Current existing cystoscopy guided transurethral resection technique is limited on direct visualization of tumor surface. In this paper, a multi-modal optical imaging modality combing high-resolution optical coherence tomography (OCT) and depth-resolved high-sensitivity fluorescence laminar optical tomography (FLOT) for structural and molecular imaging was developed. Bladder tissues from UPII-SV40T mice model were imaged by the multi-modal system ex vivo and sub-surface tumor extension was reconstructed. Algorithm was developed to further quantified 3D bladder tumor morphology and molecular alterations.

The brain is composed of the cerebrum, cerebellum, diencephalon and brainstem. The cerebrum is the superlative part of the central nervous system and also the main part of the brain. There are differences and similarities between humans and mouse. The study of mouse brain model is helpful to understand the process in clinical trials and also has reference significance for the study of human brain. Therefore, the study of mouse brain is particularly important. As the skull has a large scattering effect on light, it is difficult for us to image the brain through the skull directly. Therefore, we often use methods such as optical clearing or thin skull to reduce or remove the influence of the skull on imaging. In this paper, the transmission of photons in mouse brain was studied using Monte Carlo method. In the study of photon trajectories, the photon distribution without intact skull went farther in both longitudinal and transverse directions compared with that of with intact skull. In terms of the optical absorption density and fluence rate. On the condition of with intact skull, the distribution of optical absorption density and fluence rate was fusiform and rounder on the whole. The radial distribution range of optical absorption density and fluence rate was 0.25 cm, which was approximately 2.5 times of that of with intact skull. In the depth direction, due to the strong scattering and absorption of the scalp and skull, the optical absorption density dropped sharply from 0.890 cm^-1 to 0.415 cm^-1 . When the photons arrived at the gray matter layer, only a few photons were reserved. Due to the strong absorption and scattering effect of the gray matter layer, only a few photons left, the optical absorption density increased from 0.415 cm^-1 to 0.592 cm^-1 , and then decreased again. When the depth was 1.35 cm, the optical absorption density dropped to 0 cm-1 . After removing the skull, due to the weak absorption and scattering effect of normal saline and cerebrospinal fluid, the optical absorption density was low (0.119 cm^-1 ) and dropped slowly. When the photons arrived at the gray matter layer, most of the photons were reserved. Due to the strong absorption and scattering effect of the gray matter layer, the optical absorption density increased from 0.117 cm^-1 to 0.812 cm^-1 , then the optical absorption density decreased to 0 cm^-1 at a depth of 1.35 cm. The distribution of radiant fluence rate is similar to that of optical absorption density. This study will provide reference and theoretical guidance for the optical imaging of mouse brain and the study of the mouse and human brain.

The current coronavirus disease (COVID-19) outbreak has demonstrated the need to develop rapid and accurate diagnostic tools in order to assess the different types of respiratory diseases that affect COVID-19 patients that are significant and critical for the patient care and treatment. X-Ray imaging is one of the most important radiological examinations for screening and diagnosis of lung diseases. In this work, we describe a deep-learning based approach appropriate to potentially diagnose automatically patterns and characteristics from medical X-Ray images associated with known respiratory diseases as well as COVID-19 related ones. We propose a Convolutional Neural Network (CNN) framework for multi-label classification of the fourteen respiratory diseases and that of healthy patients. Here, we note the dataset shows disproportionate representations of the diseases. We have trained the CNN model on large multidisease, physician-diagnosed X-ray images available on an NIH open database source. We tested several loss functions commonly recommended for multi-classification training, and determined that Multi-Labeled Margin Soft loss function shows a rather smooth optimization with an apparent exponential behavior with training epoch. Following a transfer learning approach, we extend the parameters obtained from the training on the large data set to train and assess newly acquired X-ray images from COVID-19 infected patients but not labeled for any particular respiratory disease. Although additional work on validation and testing of the CNN model is needed, we have identified several parameters relevant for accurate classification

Multimodal imaging is an advantageous method to increase the accuracy of disease classification. As an example, we and others have shown that optical coherence tomography images and fluorescence spectroscopy contain complementary information that can increase the sensitivity and specificity for cancer detection. A common challenge in multimodal imaging is image co-registration. The different images are often taken with separate imaging setups, making it challenging to precisely image the same tissue area or co-register the images computationally. To solve this problem, we have developed a co-registered multimodal imaging system that images the same tissue location with reflectance, multi-photon, and optical coherence microscopy. The co-registration mechanism is a dual-clad fiber that integrates with a scanning microscope or scanning endoscope, collecting all three signals using the same optical path. In the current implementation, optical coherence tomography utilizes a 1300 nm super luminescent diode, multi-photon signals are excited by a custom femtosecond 1400 nm fiber laser producing two- and three-photon signals in the 460-900 nm band, and reflectance imaging operates at 561 nm. The system separates the different signals using fiber wavelength division multiplexers, a dual-clad fiber coupler, and dichroic mirrors to deliver the different signals to the corresponding detector. This wavelength selection enables the system to work passively, meaning that there is no need for devices such as filter wheels. Using the scanning microscope configuration, we have obtained multimodal images of ex-vivo ovine ovary tissue.

An important point-of-care diagnostic technology for COVID-19 is x-ray imaging of the lungs. Here we present a novel deep learning training method which combines both supervised and reinforcement learning methodologies which allows transfer learning in a convolutional neural network (CNN). The method integrated hill-climbing techniques and stochastic gradient descent with momentum to train the CNN architectures without overfitting on small datasets. The model was trained using the Kaggle COVID-19 Chest Radiography dataset. The dataset consists of 219 COVID-19 positive images, 1341 normal images, and 1345 viral pneumonia images. Since training of a CNN can be affected by bias and depends on the limitations of available computing power, the data set was reduced to 219 images for each class. From each of the classes, 150 random images were used for training the CNN algorithm and the model was tested with 69 independent images. Transfer training was done on three models, namely, VGG-19, DenseNet-201, and NASNet. The DenseNet-201 architecture performed the best in terms of accuracy achieving an accuracy of 96.1%. The VGG-19 and DenseNet-201 had sensitivity of 91.3 % while NASnet had a slightly higher sensitivity of 92.8%. This shows that we can have high confidence of the classification results achieved by these models. These results show that deep learning methodologies can be used for identifying COVID-19 patients quickly and accurately.

Using the multimodal combination of optical coherence tomography (OCT) with Raman spectroscopy (RS) for obtaining morpho-molecular tissue characteristics of suspicious bladder cancer lesions for improved diagnostics and signal origin characterization. We present endoscopically acquired co-localized OCT-RS data on bladder cancer biopsies. The correlated OCT and RS data enables a new way of interpretation and understanding of signal origin for both imaging modalities. Histopathological findings serve as ground truth. Molecular signal contributions can be directly correlated to identified morphological features from the OCT and lead to a better understanding of underlying biological structures.

Photoacoustic imaging is a promising technique that combines optical contrast with ultrasonic detection to map the distribution of the absorbing pigments in biological tissues. Photoacoustic microscopy with large depth of focus is significant to the biomedical research. The conventional optical-resolution photoacoustic microscope (OR-PAM) suffers from limited depth of field (DoF) since the employed focused Gaussian beam only has a narrow depth range in focus, little details in depth direction can be revealed. Here, we developed a synthetic large volumetric optical-resolution photoacoustic microscopy using morphological pyramid fusion. A self-made optical-resolution photoacoustic microscope was used to obtain source images of the same sample with different focus. Firstly, morphological smoothing was performed on the source images, then source images were decomposed using morphological pyramid, and finally the fusion image was obtained by performing inverse morphological pyramid transform. Simulation was performed to test the performance of our method, different focused images were used to verify the feasibility of the method. Performance of our method was analyzed by calculating Entropy, Average gradient, Mean Square Error (MSE) and Edge strength. The result of simulation shown that this method can extend the depth of field of PAM two times without the sacrifice of lateral resolution. And the in vivo imaging of the mouse cerebral vasculature with intact skull further demonstrates the feasibility of our method.

Optical coherence tomography is widely used in the non-invasive detection of skin cancer. We are proposing an image guiding system integrated in a full-field OCT (FF-OCT) system and an image matching algorithm to register the guiding image onto a dermoscopic image. This guiding system takes the objective optics of FF-OCT system to observe the same vision with OCT. The FOV of the guiding image and FF-OCT is 2*2mm² and 500*500μm² respectively. The spatial relation between OCT and the guiding image system is fixed, the OCT imaging location can also be identified. The guiding image is then matched into the clinically used dermoscopic image which has a 20*20 mm² FOV by an image matching algorithm. The FF-OCT image can be registered on the dermoscopic image through the guiding image. This allows physicians to target the scanning area precisely in the lesion, and record the scanned point to confirm the full lesion. This invention improves the efficiency of the entire examination and allows to follow-ups the lesion at different time.

We propose a novel Time-Resolved Mesoscopic Fluorescence Molecular Tomography (TR-MFMT) system based around a supercontinuum laser across visible to NIR range and a customized Single-Photon Avalanche Diode (SPAD) array. The system characterization is performed within the settings of multi-spectral fluorescence imaging. Especially, the characterization is achieved with a challenging application: NIR FRET quantification. The characterization results demonstrate the utility of narrow instrument response function systems and provide the foundation for high-resolution imaging multiplexed molecular imaging in the mesoscopic regime.