This PDF file contains the front matter associated with SPIE Proceedings Volume 11596 including the Title Page, Copyright information and Table of Contents

This talk will examine the challenges of designing and translating new biophotonics technologies to solve real clinical needs, drawing from examples to improve early detection of cervical cancer for women in Texas and Latin America, to improve point-of-care diagnosis of COVID-19, and to improve newborn survival in African hospitals. The talk will summarize lessons learned to increase the diversity of innovation teams, and to increase the impact and sustainability of the resulting innovations.

Welcome and Introduction to SPIE Medical Imaging conference 11596: Image Processing

There is great bodily and cognitive injury occurring to patients cared for in the ICU, much of which is preventable. ICU Survivors live with the post-intensive care syndrome or PICS, and it alters their lives in dramatic ways. I will review the data on PICS, discuss delirium and acquired-dementia, and evidence-based ways to improve patient safety and recovery.

The single-image super-resolution (SISR) network based on deep learning is dedicated to learning the mapping between low-resolution (LR) images and high-resolution (HR) images. The optimal parameters of these networks often require extensive training on large-scale external image databases. For medical magnetic resonance (MR) images, there is a lack of large data sets containing high-quality images. Some deep networks that perform well on natural images cannot be fully trained on MR images, which limits the super-resolution (SR) performance. In traditional methods, the non-local self-similarity has been verified as useful statistical prior information for image restoration. The inherent feature correlation not only exists between pixels, but some patches also tend to be repeated at different positions within and across scales of MR images. Therefore, in this paper, we propose a mixed self-similarity attention network (MSAN) to explore the long-range dependencies of different regions fully. In the feature map of the entire input MR image, the prior information of self-similarity is divided into two scales: point-similarity and patch-similarity. We use points and patches that are highly similar to the current area to restore a more detailed structural texture. The internal correlation items can be used as an essential supplement to the limited external training dataset. Besides, the large number of less informative background in MR images will interfere with practical self-similarity information. A dual attention mechanism combining first-order attention and second-order attention gives more weight to salient features and suppresses the activation of useless features. Comprehensive experiments demonstrate that the proposed achieves significantly superior results on MR images SR while outperforming state-of-the-art methods by a large margin quantitatively and visually.

Artificial intelligence (AI) models are used in medical image processing and analysis tasks like organ segmentation, anomaly detection, image reconstruction, and so on. Most often these models are trained on specific type of source domain images (non-contrast or contrast enhanced, specific field-of-view (FOV), dosage, demography, etc). It is desirable to adapt these models to a different but similar target domain, through unsupervised or semi-supervised learning methods. This paper describes a framework to re-purpose trained organ segmentation models for a target domain on which the model was not trained. The adaptation is proposed using an additional autoencoder network as a post-processing step to improve accuracy of predicted segmentation mask. Unsupervised and semi-supervised versions of adaptation are tested on contrast enhanced Computed Tomography (CT) liver, cardiac and lung exams. Experiment results show adaptability of a trained network from non-contrast to contrast enhanced scans with improved accuracy in contrast enhanced volume segmentation. A domain adaptation case study on lung disease exams with bacterial pneumonia or COVID-19 pathology also shows the effectiveness of proposed methodology.

Deep learning plays a critical role in medical image segmentation. Nevertheless, manually designing a neural network for a specific segmentation problem is a very difficult and time-consuming task due to the massive hyperparameter search space, long training time and large volumetric data. Therefore, most designed networks are highly complex, task specific and over-parametrized. Recently, multiobjective neural architecture search (NAS) methods have been proposed to automate the design of accurate and efficient segmentation architectures. However, they only search for either the macro- or micro-structure of the architecture, do not use the information produced during the optimization process to increase the efficiency of the search, and do not consider the volumetric nature of medical images. In this work, we propose EMONAS, an Efficient MultiObjective Neural Architecture Search framework for 3D medical image segmentation. EMONAS is composed of a search space that considers both the macro- and micro-structure of the architecture, and a surrogate-assisted multiobjective evolutionary based algorithm that efficiently searches for the best hyperparameters using a Random Forest surrogate and guiding selection probabilities. EMONAS is evaluated on the task of cardiac segmentation from the ACDC MICCAI challenge. The architecture found is ranked within the top 10 submissions in all evaluation metrics, performing better or comparable to other approaches while reducing the search time by more than 50% and having considerably fewer number of parameters.

Object detection in medical images using deep learning is a challenging task, due to the imbalance often present in the data. Deep learning algorithms require large amount of balanced data to achieve optimal performance, as well as close monitoring and ne-tuning of hyper parameters. For most applications, such performance monitoring is done by simply feeding unseen data trough the network, and then using the loss function for evaluation. In the case of small or sparse objects, the loss function might not able to describe the features needed, but such features can be hard to capture in a loss function. In this paper we introduce a lesion-wise whole volume validation tool, which allows more a more accurate performance monitoring of segmentation of small and sparse objects. We showcase the efficacy of our tool by applying it to the task of microbleed segmentation, and compare the behaviour of lesionwise-whole volume validation compared to well known segmentation loss functions. Microbleeds are visible as small (less than 10 mm), ovoid hypo-intensities on T2*-weighted and susceptibility weighted magnetic resonance images. Detection of microbleeds is clinically relevant, as microbleeds can indicate the risk of recurrent stroke, and are used as imaging biomarker for various neurodegenerative diseases. Manual detection or segmentation is time consuming and error prone, and suffers from high inter- and intraobserver variability. Due to the sparsity and small size of the lesions, the data is severely imbalanced.

Classification of glomerular lesions is a routine and essential task in renal pathology. Recently, machine learning approaches, especially deep learning algorithms, have been used to perform computer-aided lesion characterization of glomeruli. However, one major challenge of developing such methods is the naturally imbalanced distribution of different types of lesions. In this paper, we propose CircleMix, a novel data augmentation technique, to improve the accuracy of classifying globally sclerotic glomeruli with a hierarchical learning strategy. Different from the recently proposed CutMix method, the CircleMix augmentation is optimized for the ball-shaped biomedical objects, such as glomeruli. 8,841 glomeruli with five classes (normal, periglomerular fibrosis, obsolescent glomerulosclerosis, solidified glomerulosclerosis, and disappearing glomerulosclerosis) were employed to develop and evaluate the proposed methods. From five-fold cross validation, the proposed methods achieved superior performance of classification (F1 = 71.2%), compared with the MobileNetV2 baseline (F1 = 68.7%).

Deep learning based quantitative assessment of digital pathology images and understanding the underlying reasons for a specific clinical decision is challenging, and automatic histology pattern classification and tumor localization in whole-slide pathology images are critical for interpretable learning systems. In this study, we propose an end-to-end deep learning framework for automatic detection and localization of tumors directly from non-stained whole slide prostate core biopsy images (WSI). We use a previously described Generative Adversarial Network (GAN)-based model from our laboratory for computational Hematoxylin and Eosin (HE) staining of native non-stained pathology images. A convolutional neural network to detect and classify tumor regions in 1024×1024 virtually stained HE pixel patches, and a concurrent deep weakly supervised (WSL) model that provides localization of predominant histologic patterns used for tumor classification without the need for pixel-level annotations are reported in this study for the first time. The end-to-end system was evaluated on a 17K hold out set of 1024×1024 non-stained patches extracted from 13 whole slide prostate biopsy images. Experimental results yielded 86.37% patch-level classification accuracy with 85.05% precision, and achieved a Dice index of 65.07±1.99 (compared to 70.24±1.86 Dice index in the U-Net reference model for pixel-level segmentation). The end-to-end deep learning framework thus automates digital pathology image workflow from tissue staining to interpretable prostate tumor classification and can be valuable for accurate grading of prostate cancer and generalized to other whole-slide image classification tasks.

4D cardiac CT angiography (CCTA) images acquired for transcatheter aortic valve implantation (TAVI) planning provide a wealth of information about the morphology of the heart throughout the cardiac cycle. We propose a deep learning method to automatically segment the cardiac chambers and myocardium in 4D CCTA. We obtain automatic segmentations in 472 patients and use these to automatically identify end-systolic (ES) and end-diastolic (ED) phases, and to determine the left ventricular ejection fraction (LVEF). Our results show that automatic segmentation of cardiac structures through the cardiac cycle is feasible (median Dice similarity coefficient 0.908, median average symmetric surface distance 1.59 mm). Moreover, we demonstrate that these segmentations can be used to accurately identify ES and ED phases (bias [limits of agreement] of 1.81 [-11.0; 14.7]% and -0.02 [-14.1; 14.1]%). Finally, we show that there is correspondence between LVEF values determined from CCTA and echocardiography (-1.71 [-25.0; 21.6]%). Our automatic deep learning approach to segmentation has the potential to routinely extract functional information from 4D CCTA.

We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. Our model consists of an encoder, a decoder and a classifier. The encoder learns a non-linear subspace shared between the input data modalities. The classifier and the decoder act as regularizers to ensure that the low-dimensional encoding captures predictive differences between patients and controls. We use a learnable dropout layer to extract interpretable biomarkers from the data, and our unique training strategy can easily accommodate missing data modalities across subjects. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data. Using 10-fold cross validation, we demonstrate that our model achieves better classification accuracy than baseline methods, and that this performance generalizes to a second dataset collected at a different site. In an exploratory analysis we further show that the biomarkers identified by our model are closely associated with the well-documented deficits in schizophrenia.

Longitudinal information is important for monitoring the progression of neurodegenerative diseases, such as Huntington's disease (HD). Specifically, longitudinal magnetic resonance imaging (MRI) studies may allow the discovery of subtle intra-subject changes over time that may otherwise go undetected because of inter-subject variability. For HD patients, the primary imaging-based marker of disease progression is the atrophy of subcortical structures, mainly the caudate and putamen. To better understand the course of subcortical atrophy in HD and its correlation with clinical outcome measures, highly accurate segmentation is important. In recent years, subcortical segmentation methods have moved towards deep learning, given the state-of-the-art accuracy and computational efficiency provided by these models. However, these methods are not designed for longitudinal analysis, but rather treat each time point as an independent sample, discarding the longitudinal structure of the data. In this paper, we propose a deep learning based subcortical segmentation method that takes into account this longitudinal information. Our method takes a longitudinal pair of 3D MRIs as input, and jointly computes the corresponding segmentations. We use bi-directional convolutional long short-term memory (C-LSTM) blocks in our model to leverage the longitudinal information between scans. We test our method on the PREDICT-HD dataset and use the Dice coefficient, average surface distance and 95-percent Hausdor distance as our evaluation metrics. Compared to cross-sectional segmentation, we improve the overall accuracy of segmentation, and our method has more consistent performance across time points. Furthermore, our method identifies a stronger correlation between subcortical volume loss and decline in the total motor score, an important clinical outcome measure for HD.

Although high resolution isotropic 3D medical images are desired in clinical practice, their acquisition is not always feasible. Instead, lower resolution images are upsampled to higher resolution using conventional interpolation methods. Sophisticated learning-based super-resolution approaches are frequently unavailable in clinical setting, because such methods require training with high-resolution isotropic examples. To address this issue, we propose a learning-based super-resolution approach that can be trained using solely anisotropic images, i.e. without high-resolution ground truth data. The method exploits the latent space, generated by autoencoders trained on anisotropic images, to increase spatial resolution in low-resolution images. The method was trained and evaluated using 100 publicly available cardiac cine MR scans from the Automated Cardiac Diagnosis Challenge (ACDC). The quantitative results show that the proposed method performs better than conventional interpolation methods. Furthermore, the qualitative results indicate that especially finer cardiac structures are synthesized with high quality. The method has the potential to be applied to other anatomies and modalities and can be easily applied to any 3D anisotropic medical image dataset.

Robust groupwise registration methods are important for the analysis of large medical image datasets. We build upon the concept of deforming autoencoders that decouples shape and appearance to represent anatomical variability in a robust and plausible manner. In this work we propose a deep learning model that is trained to generate templates and deformation fields. It employs a joint encoder block which provides latent representations for both shape and appearance and is followed by two independent shape and appearance decoder paths. The model achieves image reconstruction by warping the template provided by the appearance decoder with the estimated warping field provided by the shape encoder. By restricting the embedding to a low-dimensional latent code, we are able to obtain meaningful deformable templates. Our objective function ensures smooth and realistic deformation fields. It contains an invertibility loss term, which is novel for deforming autoencoders and induces backward consistency. This should ensure that warping the reconstructed image with the deformation field ideally results in in the template. In addition, warping the template with the reversed deformation field should ideally produce the reconstructed image. We demonstrate the potential of our approach for application to two- and three-dimensional medical image data by training and evaluating it on labeled MRI brain scans. We show that adding the inverse consistency penalty to the objective function leads to improved and more robust registration results. When evaluated on unseen data with expert labels for accuracy estimation our three-dimensional model achieves substantially increased Dice scores by 5 percentage points.

Due to the spontaneous nature of resting fMRI (rs-fMRI) signals, cross-subject comparison and group studies of rs-fMRI are challenging. Existing group comparison methods typically reduce the fMRI time series either to lower-dimensional connectivity features or use ICA to reduce dimensionality. We previously developed BrainSync, an orthogonal transformation that allows direct comparison of fMRI time-series across subjects.¹ This orthogonal transform performs a temporal alignment of time-series at homologous locations across subjects allowing a direct comparison of scans. In contrast with existing fMRI analysis methods, this transform does not involve dimensionality reduction and preserves the rich functional connectivity information in fMRI data. BrainSync Alignment (BSA) is an extension of this approach that jointly synchronizes fMRI data across time-series data for multiple subjects.² Point-wise distance measures, or Pearson correlations, can be computed between the reference and synchronized time-series as measures of inter-subject differences in functional connectivity at each location in the brain. In group studies, especially in the case of spectrum disorders, distances to a single atlas do not fully reflect the differences between subjects that may lie on a multi-dimensional spectrum. Here we describe an approach that measures the distances between pairs of subjects instead of to a single reference point.³ We present novel pairwise statistical methods for fMRI that can be used for regression and also for identifying group differences. We demonstrate the effectiveness of our method in two studies: (i) pairwise comparisons of fMRI data in subjects for performing regression to an ADHD index, and (ii) an F-test using pairwise statistical analysis to compare traumatic brain injury (TBI) subjects that develop post-traumatic epilepsy (PTE) to those that do not.

Mass-univariate association analyses aim at mapping the brain regions associated with a trait/disorder. The correlation between vertices may cause a true association to spread locally (cluster of association) or distally (false positive cluster). We previously showed that controlling for all vertices in the model (using a linear mixed model: LMM), could greatly reduce the probability of false positive and improve mapping precision. Here, we investigated a new LMM called MOMENT which reduces false positive rate in methylome-wide association studies. Compared to LMM, MOMENT had enhanced power and mapping precision but failed at reducing the rates of false positives clusters.The increasing sample sizes from neuroimaging studies should allow detection of image measures are associated with phenotypic traits with smaller effect sizes, which will advance progress in the mapping of the brain regions associated with traits and diseases [1]. The UK Biobank (UKB) is one of the best example of this new generation of samples [2]. Multimodal Brain MRI collection is currently ongoing, with tens of thousands of individuals already imaged out of a target of 100,000 [2]. The large sample size, together with the breadth of phenotyping (incl. self-reports, in lab assessments, prescription and medical history), should allow new insights into the factors contributing to brain differences between older adults.

Total intracranial volume (TIV) is the volume enclosed inside the cranium, inclusive of the meninges and the brain. TIV is extensively used to correct variations in inter-subject head size for the evaluation of neurodegen- erative diseases. In this work, we present an automatic method to generate a TIV mask from MR images while synthesizing a CT image to be used in subsequent analysis. In addition, we propose an alternative way to obtain ground truth TIV masks using a semi-manual approach, which results in significant time savings. We train a conditional generative adversarial network (cGAN) using 2D MR slices to realize our tasks. The quantitative evaluation showed that the model was able to synthesize CT and generate TIV masks that closely approximate the reference images. This study also provides a comparison of the described method against skull stripping tools that output a mask enclosing the cranial volume, using MRI scan. In particular, highlighting the deficiencies in using such tools to approximate the volume using MRI scan.

In this paper, we propose a new module called cascaded multi-scale feature interaction (CMSI) for choroidal atrophy segmentation in fundus images. The proposed CMSI module makes full use of multi-scale features, including using cascaded pooling and convolution to complete feature interactions at different scales and using strip pooling to capture long-distance features. Based on the U-shape network, we use the ResNet as the backbone to extract hierarchical feature representations. The proposed CMSI module is added at the top of the encoder path. In summary, our main contributions are summarized in two aspects as follows: (1) The CMSI module is proposed for multi-scale feature ensembling by cascading multi-scale pooling and strip pooling. (2) The Dice coefficients of our proposed network on choroidal atrophy segmentation increased by 4.15% compared to U-Net.

Detection and segmentation of primary and secondary brain tumors are crucial in Radiation Oncology. Significant efforts have been dedicated to devising deep learning models for this purpose. However, development of a unified model for the segmentation of multiple types of tumors is nontrivial due to high heterogeneity across different pathologies. In this work, we propose BrainTumorNet, a multi-task learning (MTL) scheme for the joint segmentation of high-grade gliomas (HGG) and brain metastases (METS) from multimodal magnetic resonance imaging (MRI) scans. We augment the state-of-the-art DeepMedic1 architecture using this scheme and evaluate its performance on a highly unbalanced hybrid dataset comprising 259 HGG and 58 METS patient-cases. For the HGG segmentation task, the network produces a Dice score of 86.74% for whole tumor segmentation, which is comparable to 87.35% and 87.19% by the task-specific and single-task joint training baselines, respectively. For the METS segmentation task, BrainTumorNet produces an average Dice score of 62.60% thus outperforming the scores of 19.85%, 57.99%, 59.74%, and 44.17% by the two transfer-learned, task-specific, and single-task joint training baseline models, respectively. The trained network retains knowledge across segmentation tasks by exploiting the underlying correlation between pathologies. At the same time, it is discriminative enough to produce competitive segmentations for each task. The hard parameter sharing in the network reduces the computational overhead compared to training task-specific models for multiple tumor types. To our knowledge, this is the first attempt towards developing a single overarching model for the segmentation of different types of brain tumors.

Analysis of longitudinal changes in imaging studies often involves both segmentation of structures of interest and registration of multiple timeframes. The accuracy of such analysis could benefit from a tailored framework that jointly optimizes both tasks to fully exploit the information available in the longitudinal data. Most learning- based registration algorithms, including joint optimization approaches, currently suffer from bias due to selection of a fixed reference frame and only support pairwise transformations. We here propose an analytical framework based on an unbiased learning strategy for group-wise registration that simultaneously registers images to the mean space of a group to obtain consistent segmentations. We evaluate the proposed method on longitudinal analysis of a white matter tract in a brain MRI dataset with 2-3 time-points for 3249 individuals, i.e., 8045 images in total. The reproducibility of the method is evaluated on test-retest data from 97 individuals. The results confirm that the implicit reference image is an average of the input image. In addition, the proposed framework leads to consistent segmentations and significantly lower processing bias than that of a pair-wise fixed-reference approach. This processing bias is even smaller than those obtained when translating segmentations by only one voxel, which can be attributed to subtle numerical instabilities and interpolation. Therefore, we postulate that the proposed mean-space learning strategy could be widely applied to learning-based registration tasks. In addition, this group-wise framework introduces a novel way for learning-based longitudinal studies by direct construction of an unbiased within-subject template and allowing reliable and efficient analysis of spatio-temporal imaging biomarkers.

Deformable medical image registration is vital for doctor's diagnosis and quantitative analysis. In this paper, we propose a novel unsupervised learning model (denoted as BSADM) for 3D diffeomorphic medical image registration. Inspired by spatial attention module, we propose a new network architecture BSAU-Net by introducing a novel Binary Spatial Attention Module (BSAM) into skip connection, which can take full advantages of the spatial information extracted from the encoding path and corresponding decoding path. In addition, from variational method in differential geometry, monitor function f is used to control the Jacobian determinant (JD) of registration field ɸ. So, we also propose a novel orientation-consistent regularization loss to penalize the local regions with negative Jacobian determinant, which further encourages the diffeomorphic property of the transformations. We verify our method on two datasets including ADNI and PPMI dataset, and obtain excellent improvement on magnetic resonance (MR) image registration with higher average Dice scores and better diffeomorphic registration.

In order to alleviate the risk of radiation induced cataract in patients undergoing head CT examinations, the guidelines published by the American Association of Physicists in Medicine (AAPM) link the optimal scan angle to particular anatomic landmarks in the skull. In this paper, we investigated the use of a foveal fully-convolutional neural network (F-Net) for the segmentation-based detection of three head CT landmarks, with the final objective of an automatic scan quality control. Three individual networks were trained using ground-truth (GT) from three different readers to investigate the detection accuracy compared to each reader. The experiments were performed using 119 head CT scans and the three-fold cross-validation set up. For the evaluation, two performance measures were employed: the Euclidean distance between the detected landmarks and GT, and the distance of the detected landmarks to the plane generated from the GT landmark positions. For three readers, the median values of the Euclidean and point-to-plane distance obtained using F-Net were in the range of 1.3 - 2.8 mm and 0.3 - 0.8 mm, respectively. The presented method outperformed a previously published approach using image registration and achieved results comparable to the inter-observer variability between three readers. Further improvements were achieved by training a similar network which combined GT information from all readers.

Landmark detection is an essential step in the diagnosis of bone pathologies and pelvis morphometry. Hence, we propose a Deep Learning based method for automatic landmark detection on multi-modality hips magnetic resonance (MR) scans. Our method is based on a synergistic analysis of appearance and shape information by using deep networks for the detection of landmark candidate locations and then adjusting these locations using inter-landmark spatial properties. Our best model gives an average of 1.74 mm over all the landmarks, where 67% of the proposed landmarks are within the spatial matching error of at most 2mm.

Establishing accurate spatial correspondences is the main purpose for deformable medical image registration. Although many unsupervised learning-based methods have been proposed in this field and achieved fairly good results, because of their strong feature extraction ability and no need of the ground truth, they are often limited by relying on similarity of the spatially adjacent pixels which could not fully utilize geometrical feature for robust registration. To address this limitation, we propose a new graph loss function to represent the non-adjacent geometrical similarity. We divide the algorithm into two branches. The first branch takes pairs of medical images as input directly and obtains the loss term ^LCNN by typical convolution operation. In the second branch, we convert the images to forms of graph represents, and then obtain the loss term ^LGCN by graph convolution operation. Finally, the sum of the two loss terms constitute the total loss function. We verify our method on two datasets including LPBA40 and ADNI, and the experimental results demonstrate a marked improvement, with higher average Dice and lower registration errors of MSE compared with state-of-the-art methods.

Medical image registration over the past many years has been dominated by techniques which rely on expert an- notations, while not taking advantage of the unlabelled data. Deep unsupervised architectures utilized this widely available unlabelled data to model the anatomically induced patterns in a dataset. Deformable Auto-encoder (DAE), an unsupervised group-wise registration technique, has been used to generate a deformed reconstruction of an input image, which also subsequently generates a global template to capture the deformations in a medical dataset. DAEs however have significant weakness in propagating global information over range long dependencies, which may affect the registration performance on quantitative and qualitative measures. Our proposed method captures valuable knowledge over the whole spatial dimension using an attention mechanism. We present Deformable Auto-encoder Attention Relu Network (DA-AR-Net), which is an exquisite integration of the Attention Relu(Arelu), an attention based activation function into the DAE framework. A detachment of the template image from the deformation field is achieved by encoding the spatial information into two separate latent code representation. Each latent code is followed by a separate decoder network, while only a single encoder is used for feature encoding. Our DA-AR-Net is formalized after an extensive and systematic search across various hyper-parameters - initial setting of learnable parameters of Arelu, the appropriate positioning of Arelu, latent code dimensions, and batch size. Our best architecture shows a significant improvement of 42% on MSE score over previous DAEs and 32% reduction is attained while generating visually sharp global templates.

Mild traumatic brain injury (mTBI) is a complex syndrome that affects up to 600 per 100,000 individuals, with a particular concentration among military personnel. About half of all mTBI patients experience a diverse array of chronic symptoms which persist long after the acute injury. Hence, there is an urgent need for better understanding of the white matter and gray matter pathologies associated with mTBI to map which specific brain systems are impacted and identify courses of intervention. Previous works have linked mTBI to disruptions in white matter pathways and cortical surface abnormalities. Herein, we examine these hypothesized links in an exploratory study of joint structural connectivity and cortical surface changes associated with mTBI and its chronic symptoms. Briefly, we consider a cohort of 12 mTBI and 26 control subjects. A set of 588 cortical surface metrics and 4,753 structural connectivity metrics were extracted from cortical surface regions and diffusion weighted magnetic resonance imaging in each subject. Principal component analysis (PCA) was used to reduce the dimensionality of each metric set. We then applied independent component analysis (ICA) both to each PCA space individually and together in a joint ICA approach. We identified a stable independent component across the connectivity-only and joint ICAs which presented significant group differences in subject loadings (p<0.05, corrected). Additionally, we found that two mTBI symptoms, slowed thinking and forgetfulness, were significantly correlated (p<0.05, corrected) with mTBI subject loadings in a surface-only ICA. These surface-only loadings captured an increase in bilateral cortical thickness.

Misalignment of teeth or jaws can impact the ability to chew or speak, increase the risk of gum disease or tooth decay, and potentially influence a person’s (psychological) well-being. Orthodontic treatments of misaligned teeth are complex procedures that employ dental braces to apply forces in order to move the teeth or jaws to their correct position. Photographs are typically used to document the treatment. An automatic analysis of those photographs could support the decision making and monitoring process. In this paper, we propose an automatic model-based end-to-end 3-D reconstruction approach of the teeth from five photographs with predefined viewing directions (i.e. the photographs used in orthodontic treatment documentation). It uses photo- or view-specific 2-D coupled shape models to extract the teeth contours from the images. The shape reconstruction is then carried out by a deformation-based reconstruction approach that utilizes 3-D coupled shape models and minimizes a silhouette-based loss. The optimal model parameters are determined by an optimization which maximizes the overlaps between the projected 2-D outlines of individual teeth of the 3-D model and the contours extracted from the corresponding photograph. After that the point displacements between the projected outline and segmented contour are used to iteratively deform the 3-D shape model of each tooth for all five views. Back-projection into shape space ensures that the 3-D coupled shape model consists of (statistically) valid teeth. Evaluation on 22 data sets shows promising results with an average symmetric surface distance of 0.848mm and an average DICE coefficient of 0.659.

Speech is generated through complex contacts of the tongue with the palate and teeth. Evaluation of the tonguepalate contact can be beneficial in studies of linguistics, diagnosis and treatment of speech disorders, and speech synthesis. In this paper, we propose a method of tongue-palate contact assessment based on cine MR images during speech. We use a 2D U-Net to segment the space between the top of the tongue and the palates on the sagittal slices of the cine images. Then a series of MR palatograms are generated by computing the vertical thickness of the segmented space on all the sagittal slices and projecting onto the axial plane. Compared to static palatography and electropalatography, the proposed method assesses the tongue-palate contact information as well as the tongue-to-palate distances over time. We generate a sequence of MR palatograms for two healthy subjects, from three uttered phrases. During pronunciation of the selected phrases, the tongue-palate contact points and the relative tongue-to-palate distances were similar between the subjects.

Cochlear implants (CIs) are surgically implanted neural prosthetic devices to treat severe-to-profound hearing loss. Accurately localizing the CI electrodes relative to the intracochlear anatomy structures (ICAS) in the post-implantation CT (Post-CT) images of the CI recipients can help audiologists with the post-programming of the CIs. Localizing the electrodes and segmenting the ICAS in the Post-CT images are challenging due to the limited image resolution and the strong artifacts produced by the metallic electrodes. Currently, the most accurate approach to determine the physical relationship between the electrodes and the ICAS is to localize the electrodes in the Post-CT image, segment the ICAS in the pre-implantation CT (Pre-CT) image of the CI recipient, and register the two images. Here we propose a 3D multi-task network to remove the artifacts, segment the ICAS, and localize the electrodes in the Post-CT images simultaneously. Our network is trained with a small image set and achieves comparable segmentation results and encouraging electrode localization results compared to the current state-of-the-art methods. As our method does not require the Pre-CT images, it provides the audiologist with information that guides the programming process even for patients for whom these images are not available.

This paper addresses the problem of reconstructing an image from low-count Positron Emission Tomography (PET) data. We build on recent advances combining deep neural networks with expectation-maximization algorithms. More specifically, we extend the recent DIPRecon approach [1] to handle various challenges linked to natively low-count Yttrium 90 data. To this end, we rely on the interpretation of the Deep Image Prior (DIP) in the light of approximate Bayesian inference. By introducing a stochastic gradient Langevin dynamics (SGLD) optimizer, we reduce the tendency of the algorithm to overfit the noisy maximum likelihood estimate while improving the contrast recovery figures. Moreover, as a by-product of the SGLD optimization, the method recovers an uncertainty value associated with every voxel in the estimated image. We qualitatively and quantitatively evaluate the proposed method on data acquired with the NEMA IEC body phantom achieving high-quality results.

The subcortical structures of the brain are relevant for many neurodegenerative diseases like Huntington’s disease (HD). Quantitative segmentation of these structures from magnetic resonance images (MRIs) has been studied in clinical and neuroimaging research. Recently, convolutional neural networks (CNNs) have been successfully used for many medical image analysis tasks, including subcortical segmentation. In this work, we propose a 2-stage cascaded 3D subcortical segmentation framework, with the same 3D CNN architecture for both stages. Attention gates, residual blocks and output adding are used in our proposed 3D CNN. In the first stage, we apply our model to downsampled images to output a coarse segmentation. Next, we crop the extended subcortical region from the original image based on this coarse segmentation, and we input the cropped region to the second CNN to obtain the final segmentation. Left and right pairs of thalamus, caudate, pallidum and putamen are considered in our segmentation. We use the Dice coefficient as our metric and evaluate our method on two datasets: the publicly available IBSR dataset and a subset of the PREDICT-HD database, which includes healthy controls and HD subjects. We train our models on only healthy control subjects and test on both healthy controls and HD subjects to examine model generalizability. Compared with the state-of-the-art methods, our method has the highest mean Dice score on all considered subcortical structures (except the thalamus on IBSR), with more pronounced improvement for HD subjects. This suggests that our method may have better ability to segment MRIs of subjects with neurodegenerative disease.

We used orthonormal projective non-negative matrix factorization (OPNMF) to identify distinct spatial components of voxel-wise volume loss in the brains of patients with Alzheimer's disease relative to age-matched normal controls. Non-negative matrix factorization (NMF) is a dimension reduction approach which can be used to extract spatially localized components of volume loss from 3D T1-weighted MR images. OPNMF is a variant of NMF with a potential for providing more biologically meaningful components to help better assess the patterns of brain atrophy. Once components have been defined, non-negative coefficients representing each component's contribution to volume loss can be defined in subjects not included in the group used for component discovery. These coefficients are subject-specific quantitative attributes of atrophy, and we studied their partial dependence network using a Gaussian copula graphical model. We also verified their ability to distinguish diagnosis groups. We performed a clustering approach to investigate possible imaging subtypes of Alzheimer's Disease patients. Our results are the first attempt to apply OPNMF to Alzheimer's Disease and we detect different subtypes based on components of gray matter atrophy which correlate with previous work.

In pre- and post-surgical surface shape analysis, establishing shape correspondence is necessary to investigate the postoperative surface changes. However, structural absence after the operation accompanies focal non-rigid changes, which leads to challenges in existing surface registration methods. In this paper, we present a fully automatic particle-based method to establish surface correspondence that can handle partial structural abnormality in the temporal lobe resection. Our method optimizes the coordinates of points which are modeled as particles on surfaces in a hierarchical way to reduce a chance of being trapped in a local minimum during the optimization. In the experiments, we evaluate the effectiveness of our method in comparison with conventional spherical registration (FreeSurfer) on two scenarios: cortical thickness changes in healthy controls within a short scan-rescan time window and patients with temporal lobe resection. The postsurgical scan is acquired at least 1 year after the presurgical scan. In region of interest-wise (ROI-wise) analysis, no changes on cortical thickness are found in both methods on the healthy control group. In patients, since there is no ground truth available, we instead investigated the disagreement between our method and FreeSurfer. We see poorly matched ROIs and large cortical thickness changes using FreeSurfer. On the contrary, our method shows well-matched ROIs and subtle cortical thickness changes. This suggests that the proposed method can establish a stable shape correspondence, which is not fully captured in a conventional spherical registration.

Although multi-modal imaging tends to improve the segmentation and classification performance in the field of medical image processing, lacking certain modalities at test time limits its clinical applicability. In this paper, we explored the ability of cross-modal distillation for increasing the performance of T_1w MRI-based brain tumor segmentation. More specifically, we considered having high resolution T_1w and T_2w MRI sequences available for training while having only a high resolution T_1w MRI sequence available at test time. We investigated the efficacy of the proposed method to improve the whole tumor segmentation using the BRATS 2018 dataset. Both cross-modal knowledge distillation and cross-modal feature distillation approaches were confirmed to enrich the representation of the T_1w MRI sequence by learning from the representation of the more informative T_2w MRI sequence during training, thereby improving the mean Dice scores by 6.14 % and 7.02 %, respectively.

Deep Convolutional Neural Networks (CNNs) i.e. Residual Networks (ResNets) have been used successfully for many computer vision tasks, but are difficult to scale to 3D volumetric medical data. Memory is increasingly often the bottleneck when training 3D Convolutional Neural Networks (CNNs). Recently, invertible neural networks have been applied to significantly reduce activation memory footprint when training neural networks with backpropagation thanks to the invertible functions that allow retrieving input from its output without storing intermediate activations in memory to perform the backpropagation. Among many successful network architectures, 3D Unet1 has been established as a standard architecture for volumetric medical segmentation. Thus, we choose 3D Unet as a baseline for a non-invertible network and we then extend it with the invertible residual network. In this paper, we proposed two versions of invertible Residual Network, namely Partially Invertible Residual Network (Partially-InvRes) and Fully Invertible Residual Network (Fully-InvRes). In Partially-InvRes, the invertible residual layer is defined by a technique called additive coupling2 whereas in Fully-InvRes, both invertible upsampling and downsampling operations are learned based on squeezing (known as pixel shuffle).³ Furthermore, to avoid the overfitting problem because of less training data, a variational auto-encoder (VAE) branch is added to reconstruct the input volumetric data itself. Our results indicate that by using partially/fully invertible networks as the central workhorse in volumetric segmentation, we not only reduce memory overhead but also achieve compatible segmentation performance compared against the non-invertible 3D Unet. We have demonstrated the proposed networks on various volumetric datasets such as iSeg 2019⁴ and BraTS 2020.⁵

Purpose: Computed tomography perfusion (CTP) is used to diagnose ischemic strokes through contralateral hemisphere comparisons of various perfusion parameters. Various perfusion parameter thresholds have been utilized to segment infarct tissue due to differences in CTP software and patient baseline hemodynamics. This study utilized a convolutional neural network (CNN) to eliminate the need for non-universal parameter thresholds to segment infarct tissue. Methods: CTP data from 63 ischemic stroke patients was retrospectively collected and perfusion parameter maps were generated using Vitrea CTP software. Infarct ground truth labels were segmented from diffusion-weighted imaging (DWI) and CTP and DWI volumes were registered. A U-net based CNN was trained and tested five separate times using each CTP parameter (cerebral blood flow (CBF), cerebral blood volume (CBV), time-to-peak (TTP), mean-transit-time (MTT), delay time). 8,352 infarct slices were utilized with a 60:30:10 training:testing:validation split and Monte Carlo crossvalidation was conducted using 20 iterations. Infarct volumes were reconstructed following segmentation from each CTP slice. Infarct spatial and volumetric agreement was compared between each CTP parameter and DWI. Results: Spatial agreement metrics (Dice coefficient, positive predictive value) for each CTP parameter in predicting infarct volumes are: CBF=(0.67, 0.76), CBV=(0.44, 0.62), TTP=(0.60, 0.67), MTT=(0.58, 0.62), delay time=(0.57, 0.60). 95% confidence intervals for volume differences with DWI infarct are: CBF=14.3±11.5 mL, CBV=29.6±21.2 mL, TTP=7.7±15.2 mL, MTT=-10.7±18.6 mL, delay time=-5.7±23.6 mL. Conclusions: CBF is the most accurate CTP parameter in segmenting infarct tissue. Segmentation of infarct using a CNN has the potential to eliminate non-universal CTP contralateral hemisphere comparison thresholds.

Accurately segmenting organs in abdominal computed tomography (CT) scans is crucial for clinical applications such as pre-operative planning and dose estimation. With the recent advent of deep learning algorithms, many robust frameworks have been proposed for organ segmentation in abdominal CT images. However, many of these frameworks require large amounts of training data in order to achieve high segmentation accuracy. Pediatric abdominal CT images containing reproductive organs are particularly hard to obtain since these organs are extremely sensitive to ionizing radiation. Hence, it is extremely challenging to train automatic segmentation algorithms on organs such as the uterus and the prostate. To address these issues, we propose a novel segmentation network with a built-in auxiliary classifier generative adversarial network (ACGAN) that conditionally generates additional features during training. The proposed CFG-SegNet (conditional feature generation segmentation network) is trained on a single loss function which combines adversarial loss, reconstruction loss, auxiliary classifier loss and segmentation loss. 2.5D segmentation experiments are performed on a custom data set containing 24 female CT volumes containing the uterus and 40 male CT volumes containing the prostate. CFG-SegNet achieves an average segmentation accuracy of 0.929 DSC (Dice Similarity Coefficient) on the prostate and 0.724 DSC on the uterus with 4-fold cross validation. The results show that our network is high-performing and has the potential to precisely segment difficult organs with few available training images.

The unsupervised segmentation is an increasingly popular topic in biomedical image analysis. The basic idea is to approach the supervised segmentation task as an unsupervised synthesis problem, where the intensity images can be transferred to the annotation domain using cycle-consistent adversarial learning. The previous studies have shown that the macro-level (global distribution level) matching on the number of the objects (e.g., cells, tissues, protrusions etc.) between two domains resulted in better segmentation performance. However, no prior studies have exploited whether the unsupervised segmentation performance would be further improved when matching the exact number of objects at micro-level (mini-batch level). In this paper, we propose a deep learning based unsupervised segmentation method for segmenting highly overlapped and dynamic sub-cellular microvilli. With this challenging task, both micro-level and macro-level matching strategies were evaluated. To match the number of objects at the micro-level, the novel uorescence-based micro-level matching approach was presented. From the experimental results, the micro-level matching did not improve the segmentation performance, compared with the simpler macro-level matching.

One of the major problems in medical imaging is the shortage of pathology data. In most cases, the acquisition of labeled data is expensive and usually involves manual labeling by a skilled medical expert. Because of this, most medical imaging tasks suffer from a severe class imbalance with a bias towards non-pathological classes, resulting in reduced performance. The recent growth in the use of generative adversarial networks and their ability to generate synthetic data shows great promise for reducing the class imbalance problem. In this work we introduce the GC-CycleGAN model, a general method for CycleGAN factorization, utilizing Grad-CAMs as auxiliary data in the CycleGAN model to generate synthetic images. Our novel approach utilizes Grad-CAMs ability to describe class activation and uses it for improved network classification, rather than as a visualization tool. The spread of the COVID-19 pandemic is affecting the lives of millions worldwide. If proven effective, automated COVID-19 detection from chest X-ray images can be a supportive step in the fight against COVID-19. However, the task of COVID-19 classification suffers greatly from the class imbalance problem. Using the GC-CycleGAN method, we demonstrate in this work the ability to balance a heavily imbalanced dataset for the task of COVID-19 vs. non-COVID-19 pneumonia X-ray classification. We show improved results over two baselines and the COVID-Net model.

Machine learning technologies using deep neural networks (DNNs), especially convolutional neural networks (CNNs), have made automated, accurate, and fast medical image analysis a reality for many applications, and some DNN-based medical image analysis systems have even been FDA-cleared. Despite the progress, challenges remain to build DNNs as reliable as human expert doctors. It is known that DNN classifiers may not be robust to noises: by adding a small amount of noise to an input image, a DNN classifier may make a wrong classification of the noisy image (i.e., in-distribution adversarial sample), whereas it makes the right classification of the clean image. Another issue is caused by out-of-distribution samples that are not similar to any sample in the training set. Given such a sample as input, the output of a DNN will become meaningless. In this study, we investigated the in-distribution (IND) and out-of-distribution (OOD) adversarial robustness of a representative CNN for lumbar disk shape reconstruction from spine MR images. To study the relationship between dataset size and robustness to IND adversarial attacks, we used a data augmentation method to create training sets with different levels of shape variations. We utilized the PGD-based algorithm for IND adversarial attacks and extended it for OOD adversarial attacks to generate OOD adversarial samples for model testing. The results show that IND adversarial training can improve the CNN robustness to IND adversarial attacks, and larger training datasets may lead to higher IND robustness. However, it is still a challenge to defend against OOD adversarial attacks.

Gastroendoscopy is the golden standard procedure that enables medical doctors to investigate the inside of a patient's stomach. Monocular depth estimation from an endoscopic image enables the simultaneous acquisition of RGB and depth data, which can boost the capability of the endoscopy for various potential diagnostic applications, such as the RGB-D data acquisition toward whole stomach 3D reconstruction for lesion localization and local view expansion for lesion inspection. Therefore, deep-learning-based approaches are gaining traction to provide depth information in monocular endoscopy. Since it is very difficult to obtain ground-truth RGB and depth image pairs in clinical settings, computer-generated (CG) data is usually used for training the depth estimation network. However, CG data has a limitation to generate realistic RGB and depth data. In this paper, we propose a novel data generation strategy for self-supervised training to predict the depth in gastroendoscopy. To obtain dense reference depth data for training, we first reconstruct a whole stomach 3D model by exploiting chromoendoscopic images sprayed with indigo carmine (IC) blue dye. We then generate virtual no-IC images from chromoendoscopic images using CycleGAN to make our depth estimation network applicable to general endoscopic images without IC dye. We experimentally demonstrate that our proposed approach achieves plausible depth prediction on both chromoendoscopic and general white-light endoscopic images.

Generative adversarial networks (GANs) have proven useful for several medical imaging tasks, including image reconstruction and stochastic object model generation. Thus far, most of the work with GANs has been constrained to twodimensional images. Considering that medical imaging data are often inherently three-dimensional (3D), a 3D GAN would be a more principled way to synthesize realistic volumes. Training a 3D GAN is both computationally and memory intensive. However, prior work has not considered the anisotropic nature of many medical imaging systems. In this paper, the SlabGAN is proposed to reduce the inefficiencies associated with training a 3D GAN. The SlabGAN uses the progressive GAN architecture extended to 3D, but removes the requirement of the three dimensions being equal sizes. This permits the generation of anisotropic 3D volumes with large x and y dimensions. The SlabGAN is trained on MRI brain images from the fastMRI dataset to generate images of dimension 256×256×16. The x and y dimensions of these images are comparable to previously published results while requiring significantly fewer computational resources to generate. The trained SlabGAN is applicable to tasks such as 3D medical image reconstruction and thin-slice MR super resolution.

Many algorithms and methods have been proposed for inverse problems particularly with the recent surge of interest in machine learning and deep learning methods. Among all proposed methods, the most popular and effective method is the convolutional neural network (CNN) with mean square error (MSE). This method has been proven effective in super-resolution, image de-noising, and image reconstruction. However, this method is known to over-smooth images due to the nature of MSE. MSE based methods minimize Euclidean distance for all pixels between a baseline image and a generated image by CNN and ignore the spatial information of the pixels such as image texture. In this paper, we proposed a new method based on Wasserstein GAN (WGAN) for inverse problems. We showed that the WGAN-based method was effective to preserve image texture. It also used a maximum likelihood estimation (MLE) regularizer to preserve pixel fidelity. Maintaining image texture and pixel fidelity is the most important requirement for medical imaging. We used Peak Signal to Noise Ratio (PSNR) and Structure Similarity (SSIM) to evaluate the proposed method quantitatively. We also conducted the first-order and the second-order statistical image texture analysis to assess image texture.

Abdominal kidney segmentation plays an essential role in diagnosis and treatment of kidney diseases, particularly in surgical planning and clinical outcome analysis before and after kidney surgery. It still remains challenging to precisely segment the kidneys from CT images. Current segmentation approaches still suffer from CT image noises and variations caused by different CT scans, kidney location discrepancy, pathological morphological diversity among patients, and partial volume artifacts. This paper proposes a fully automatic kidney segmentation method that employs a volumetric convolution driven cascaded V-Net architecture and false positive reduction to precisely extract the kidney regions. We evaluate our method on publicly available kidney CT data. The experimental results demonstrate that our proposed method is a promising method for accurate kidney segmentation, providing a dice coefficient of 0.95 better than other approaches as well less computational time.

Percutaneous cryoablation is becoming more popular for the treatment of renal cell carcinoma. Interventional computed tomography (iCT) is commonly used for guidance but reducing radiation dose and increasing slice thickness makes super-resolution (SR) essential for improving image quality. The proposed method takes low quality (LQ), thick slice images and converts them to high quality (HQ), thin slice images while performing denoising and partial volume correction in the z-direction. As LQ and HQ iCT images are challenging to pair up, we train a 3D U-Net equipped with an up-sampling module on simulated LQ (sLQ) data and then test on the real LQ (rLQ) images with cubic interpolation and random forest as comparison. During validation on sLQ data, the U-Net outperformed interpolation and random forest (SSIM 0.9991 vs 0.9959 and 0.9985 respectively), but performance suffered when testing on the out-of-distribution rLQ images. The Dice score showed a substantial improvement when used to compare needle segmentations performed on U-Net generated images versus those from interpolation and random forest (0.4073 vs. 0.2919 and 0.3777 respectively), indicating that the U-Net is reducing the z-direction partial volume effect to a greater degree than these techniques. We have shown that a neural network trained to perform SR on simulated data outperforms interpolation and random forest on real data in terms of localisation of clinically relevant objects such as needles, despite the differing data distribution.

Automatic pancreas segmentation in abdominal CT images plays an important role in clinical applications. It can provide doctors with quantitative and qualitative information. Due to the small size, unclear edges, and the high anatomical differences between patients, it is a challenging task to accurately segment the pancreas with diseases. In this paper, we propose a new method to automatically segment the pancreas in abdominal CT images. First, we propose a contrast enhancement block. The block generates edge information and uses gating mechanism to enhance edge details of the pancreas. Second, we leverage a reverse attention block. This block utilizes the decoder feature map to guide the network to mine complementary discriminative regions. The proposed method is trained on 63 3D CT images, validated on 15 3D CT images, and tested on 28 3D CT images. Compared with manual segmentation, the mean Dice similarity coefficient can reach 86.11±8.02%. Experimental results show that our method can obtain more accurate segmentation results compared with existing segmentation methods.

Purpose Investigate and evaluate the accuracy of deep learning (DL)-based segmentation and deformable image registration (DIR) for the automatization of recurrence risk map atlas definition. Materials and methods Twelve patients with visible recurrence on 18F-DCFPyL PET/CT after prostatectomy were retrospectively analyzed. The bladder, rectum, iliac arteries and veins, and recurrence sites were manually delineated. A previously trained DL model for female pelvic anatomy was re-optimized for male to automatically segment the anatomical regions of interest (ROI). Inter-patient registration was investigated using 4 registration methods: rigid, B-Spline Plastimatch, intensity DIR, and a hybrid intensity-based DIR with varying number of controlling ROI. Performance of the methods were reported using contour-based metrics, determinant of the Jacobian, contour variability in term of volume and position, and probability of overlap with the template organs. Results Transfer learning of the DL model provided greater accuracy for the bladder and rectum than for new structures such as iliac arteries and veins with average Dice similarity coefficient ranges of 0.82-0.96 and 0.63-0.77, respectively. Compared to intensity only DIR, hybrid intensity-based DIR with controlling ROI provided better contour-based metrics, determinant of Jacobian, and less incidence of overlap between recurrence sites and template organs. Centroid position variability between the registration approaches were reported with average range of 1.6-11.3 mm and up to 5.7-30 mm. Conclusion DL and hybrid DIR models can be used to automatize inter-patient registration in the definition of population-based recurrence risk map. DIR uncertainties in the propagation of the recurrence between patients need to be carefully verified before being used in population-based model.

Renal segmentation on contrast-enhanced computed tomography (CT) provides distinct spatial context and morphology. Current studies for renal segmentations are highly dependent on manual efforts, which are time-consuming and tedious. Hence, developing an automatic framework for the segmentation of renal cortex, medulla and pelvicalyceal system is an important quantitative assessment of renal morphometry. Recent innovations in deep methods have driven performance toward levels for which clinical translation is appealing. However, the segmentation of renal structures can be challenging due to the limited field-of-view (FOV) and variability among patients. In this paper, we propose a method to automatically label the renal cortex, the medulla and pelvicalyceal system. First, we retrieved 45 clinically-acquired deidentified arterial phase CT scans (45 patients, 90 kidneys) without diagnosis codes (ICD-9) involving kidney abnormalities. Second, an interpreter performed manual segmentation to pelvis, medulla and cortex slice-by-slice on all retrieved subjects under expert supervision. Finally, we proposed a patch-based deep neural networks to automatically segment renal structures. Compared to the automatic baseline algorithm (3D U-Net) and conventional hierarchical method (3D U-Net Hierarchy), our proposed method achieves improvement of 0.7968 to 0.6749 (3D U-Net), 0.7482 (3D U-Net Hierarchy) in terms of mean Dice scores across three classes (p-value < 0.001, paired t-tests between our method and 3D U-Net Hierarchy). In summary, the proposed algorithm provides a precise and efficient method for labeling renal structures.

Clinical data elements (CDEs) (e.g., age, smoking history), blood markers and chest computed tomography (CT) structural features have been regarded as effective means for assessing lung cancer risk. These independent variables can provide complementary information and we hypothesize that combining them will improve the prediction accuracy. In practice, not all patients have all these variables available. In this paper, we propose a new network design, termed as multi-path multi-modal missing network (M3Net), to integrate the multi-modal data (i.e., CDEs, biomarker and CT image) considering missing modality with multiple paths neural network. Each path learns discriminative features of one modality, and different modalities are fused in a second stage for an integrated prediction. The network can be trained end-to-end with both medical image features and CDEs/biomarkers, or make a prediction with single modality. We evaluate M3Net with datasets including three sites from the Consortium for Molecular and Cellular Characterization of Screen-Detected Lesions (MCL) project. Our method is cross validated within a cohort of 1291 subjects (383 subjects with complete CDEs/biomarkers and CT images), and externally validated with a cohort of 99 subjects (99 with complete CDEs/biomarkers and CT images). Both cross-validation and external-validation results show that combining multiple modality significantly improves the predicting performance of single modality. The results suggest that integrating subjects with missing either CDEs/biomarker or CT imaging features can contribute to the discriminatory power of our model (p < 0.05, bootstrap two-tailed test). In summary, the proposed M3Net framework provides an effective way to integrate image and non-image data in the context of missing information.

Treatment of patients with obstructive coronary artery disease is guided by the functional significance of a coronary artery stenosis. Fractional flow reserve (FFR), measured during invasive coronary angiography (ICA), is considered the references standard to define the functional significance of a coronary stenosis. Here, we present an automatic method for non-invasive detection of patients with functionally significant coronary artery stenosis based on 126 retrospectively collected cardiac CT angiography (CCTA) scans with corresponding FFR measurement. We combine our previous works for the analysis of the complete coronary artery tree and the LV myocardium by applying convolutional autoencoders (CAEs) to characterize both, coronary arteries and the LV myocardium. To handle the varying number of coronary arteries in a patient, an attention-based neural network is trained to obtain a combined representation per patient, and to classify each patient according to the presence of functionally significant stenosis. Cross-validation experiments resulted in an average area under the receiver operating characteristic curve of 0.74, and showed that the proposed combined analysis outperformed the analysis of the coronary arteries or the LV myocardium alone. This may lead to a reduction in the number of unnecessary ICA procedures in patients with suspected obstructive CAD.

A major goal of lung cancer screening is to identify individuals with particular phenotypes that are associated with high risk of cancer. Identifying relevant phenotypes is complicated by the variation in body position and body composition. In the brain, standardized coordinate systems (e.g., atlases) have enabled separate consideration of local features from gross / global structure. To date, no analogous standard atlas has been presented to enable spatial mapping and harmonization in chest computational tomography (CT). In this paper, we propose a thoracic atlas built upon a large low dose CT database with no screening detected malignancy (age 46-79 years, mean 64.9 years). The application validity of the developed atlas is evaluated in terms of discriminative capability for different anatomic phenotypes, including body mass index (BMI), chronic obstructive pulmonary disease (COPD), and coronary artery calcification (CAC).

Vertebral bone segmentation from magnetic resonance (MR) images is a challenging task. Due to the inherent nature of the modality to emphasize soft tissues of the body, common thresholding algorithms are ineffective in detecting bones in MR images. On the other hand, it is relatively easier to segment bones from computed tomography (CT) images because of the high contrast between bones and the surrounding regions. For this reason, we perform a cross-modality synthesis between MR and CT domains for simple thresholding-based segmentation of the vertebral bones. However, this implicitly assumes the availability of paired MR-CT data, which is rare, especially in the case of scoliotic patients. In this paper, we present a completely unsupervised, fully three-dimensional (3D) cross-modality synthesis method for segmenting scoliotic spines. A 3D CycleGAN model is trained for an unpaired volume-to-volume translation across MR and CT domains. Then, the Otsu thresholding algorithm is applied to the synthesized CT volumes for easy segmentation of the vertebral bones. The resulting segmentation is used to reconstruct a 3D model of the spine. We validate our method on 28 scoliotic vertebrae in 3 patients by computing the point-to-surface mean distance between the landmark points for each vertebra obtained from pre-operative X-rays and the surface of the segmented vertebra. Our study results in a mean error of 3.41 ± 1.06 mm. Based on qualitative and quantitative results, we conclude that our method is able to obtain a good segmentation and 3D reconstruction of scoliotic spines, all after training from unpaired data in an unsupervised manner.

Accurately labeled segments of the coronary artery trees are important for diagnostic reporting of coronary artery disease. As current automatic reporting tools do not consider anatomical segment labels, accurate automatic solutions for deriving these labels would be of great value. We propose an automatic method for labeling segments in coronary artery trees represented by centerlines automatically extracted from CCTA images. Using the connectivity between the centerlines, we construct a tree graph. Coronary artery segments are defined as edges of this graph and characterized by location and geometry features. The constructed coronary artery tree is transformed into a linegraph and used as input to a graph attention network, which is trained to classify labels of coronary artery segments. The method was evaluated on 71 CCTA images, achieving an F1-score of 92.4% averaged over all patients and segments. The results indicate that graph attention networks are suitable for coronary artery tree labeling.

A decrease in volume of the olfactory bulbs is an early marker for neurodegenerative diseases, such as Parkinson’s and Alzheimer’s disease. Recently, asymmetric volumes of olfactory bulbs present in postmortem MRIs of COVID-19 patients indicate that the olfactory bulbs might play an important role in the entrance of the disease in the central nervous system. Hence, volumetric assessment of the olfactory bulbs can be valuable for various conditions. Given that manual annotation of the olfactory bulbs in MRI to determine their volume is tedious, we propose a method for their automatic segmentation. To mitigate the class imbalance caused by the small volume of the olfactory bulbs, we first localize the center of each olfactory bulb in a scan using convolutional neural networks (CNNs). We use these center locations to extract a bounding box containing both olfactory bulbs. Subsequently, the slices present in the bounding box are analyzed by a segmentation CNN that classifies each voxel as left olfactory bulb, right olfactory bulb, or background. The method achieved median (IQR) Dice coefficients of 0.84 (0.08) and 0.83 (0.08), and Average Symmetrical Surface Distances of 0.12 (0.08) and 0.13 (0.08) mm for the left and the right olfactory bulb, respectively. Wilcoxon Signed Rank tests showed no significant difference between the volumes computed from the reference annotation and the automatic segmentations. Analysis took only 0.20 second per scan and the results indicate that the proposed method could be a first step towards large-scale studies analyzing pathology and morphology of the olfactory bulbs.

The human tongue muscles plays an important role in multiple vital human functions. Most tongue regions are extensively interdigitated with two orthogonal muscle fibers. Reconstruction of the tongue muscle fiber orientations can help understand the deformation of each muscle group and its function. High angular resolution diffusion imaging (HARDI), one of the diffusion weighted imaging techniques, has been used to resolve the crossing muscle fibers in the tongue. Most existing fiber reconstruction methods use HARDI data to estimate the fiber orientation distribution function (fODF), from which the distinct fiber orientations can be identified by a peak finding algorithm. The assignment of the primary and second fiber orientations can be inconsistent with neighboring voxels. In this paper, we propose a fiber matching algorithm to refine the display of the fiber orientations, which can be used as a post-processing step for fiber reconstruction. The fiber matching algorithm takes the fiber orientations that are reconstructed by a deep convolutional neural network as input, and computes the similarity between neighboring fibers under different assignments. The optimal assignments are achieved by solving a quadratic unconstrained binary optimization model. The proposed method was shown to greatly improve the fiber assignments on synthetic tongue fiber orientations. Application to post-mortem human tongue indicated that the proposed method can reconstruct the complex muscle fibers of the human tongue and improve the visualization of the fiber orientations.

The Digital Breast Tomosynthesis (DBT) projections are obtained with low quality, being essential to use denoising methods to increase the quality of the projections. Currently, deep learning methods have become the state-of-art approach in denoising. Some papers have proposed to apply deep learning methods for denoising DBT projections, however, there is a lack of clarity in the results comparing with traditional methods. In this paper, we proposed to use a CNN to denoise DBT projections, and compare it with traditional denoising methods. The results shown that the CNN is superior quantitatively and qualitatively in comparison with the traditional methods.

Cochlear implants (CIs) have been shown to be highly effective restorative devices for patients suffering from severe-to-profound hearing loss. Hearing outcomes with CIs are dependent on the positions of the electrodes with respect to intracochlear anatomy. However, intra-cochlear anatomy can only be directly visualized using high resolution modalities such as μCT, which cannot be used in vivo. Despite this limitation, we have developed an active shape model approach for segmenting the intra-cochlear anatomy by leveraging the visible structures as landmarks. Still, due to the limited availability of μCT specimens, the segmentation method was validated on only a small dataset of 5 samples. In this study, we expand the dataset to 16 samples and provide a more comprehensive validation of the method’s performance with respect to model parameters and training set size. We found parameters that optimize mean surface segmentation performance to 0.11mm. Parameters that corresponded to tighter constraints generally led to smaller errors, and returns on segmentation performance begin diminishing after 11 samples, thus suggesting that the main performance bottleneck is due to the searching scheme rather than a limited training set size. These results are critical to understand the limitations of the method for clinical use and for future development.

Accurate assessment of fetal gestational age (GA) is critical to the clinical management of pregnancy. Industrialized countries rely upon obstetric ultrasound (US) to make this estimate. In low- and middle- income countries, automatic measurement of fetal structures using a low-cost obstetric US may assist in establishing GA without the need for skilled sonographers. We present a technique for generating raw images suitable for model training by removing caliper and text annotation and describe a fully automated pipeline for image classification, segmentation, and structure measurement to estimate the GA. he resulting framework achieves an average accuracy of 93% in classification tasks, a mean Intersection over Union accuracy of 0.91 during segmentation tasks, and a mean measurement error of 1.89 centimeters, finally leading to a 1.4 day mean average error in the predicted GA compared to expert sonographer GA estimate using the Hadlock equation.

Existing deep-learning methods achieve state-of-art segmentation of multiple heart substructures from 2D echocardiography videos, an important step in the diagnosis and management of cardiovascular disease. However, these methods generally perform frame-level segmentation, ignoring the temporal coherence in heart motion between frames, which is a useful signal in clinical protocols. In this work, we implement temporally consistent video segmentation, which has recently been shown to improve performance on the multi-structure annotated CAMUS dataset. We show that data augmentation further improves results, which are consistent with prior state-of-art works. Our 10-fold cross-validation shows that video segmentation improves the automatic comparison to clinical indices including smaller mean absolute errors for left ventricular end-diastolic volume (8.7 mL vs 9.9 mL), end-systolic volume (6.3 mL vs 6.6 mL), and ejection fraction (EF) (4.6% vs 5.3%). In segmenting key cardiac structures, video segmentation achieves mean Dice overlap of 0.93 on left ventricular endocardium, 0.95 on left ventricular epicardium, and 0.88 on left atrium. To assess clinical generalizability, we further apply the CAMUS-trained video segmentation models, without tuning, to a larger, recently published EchoNet-Dynamic clinical dataset. On 1274 patients in the test set, we obtain absolute errors of 6:3% ± 5:4 in EF, confirming the reliability of this scheme. In that the EchoNet-Dynamic videos contain limited annotation only for left ventricle endocardium, this effort extends at little cost generalizable, multi-structure video segmentation to a large clinical dataset.

To advance our understanding of speech motor control, it is essential to image and assess dynamic functional patterns of internal structures caused by the complex muscle anatomy inside the human tongue. Speech pathologists are investigating into new tools that help assessment of internal tongue muscle’s cooperative mechanics on top of their anatomical differences. Previous studies using dynamic magnetic resonance imaging (MRI) of the tongue revealed that tongue muscles tend to function in different groups during speech, especially the floor-of-the-mouth (FOM) muscles. In this work, we developed a method that analyzed the unique functional pattern of the FOM muscles in speech. First, four-dimensional motion fields of the whole tongue were computed using tagged MRI. Meanwhile, a statistical atlas of the tongue was constructed to form a common space for subject comparison, while a manually delineated mask of internal tongue muscles was used to separate individual muscle’s motion. Then we computed four-dimensional motion correlation between each muscle and the FOM muscle group. Finally, dynamic correlation of different muscle groups was compared and evaluated. We used data from a study group of nineteen subjects including both healthy controls and oral cancer patients. Results revealed that most internal tongue muscles coordinated in a similar pattern in speech while the FOM muscles followed a unique pattern that helped supporting the tongue body and pivoting its rotation. The proposed method can help provide further interpretation of clinical observations and speech motor control from an imaging point of view.

Our understanding of synaptic connectivity in the brain relies on the ability to accurately trace sparsely labeled neurons from 3D optical microscopy stacks of images. A variety of automated algorithms and software tools have been developed for this task. These algorithms can capture the general layout of neurites with high fidelity, but the resulting traces often contain topological errors such as broken and incorrectly merged branches. Even a small number of isolated topological errors can drastically alter the connectivity, and therefore, their detection and correction are paramount for connectomics studies. Here, we describe an automated trace proofreading approach that utilizes machine learning to correct trace topology. Multiple stacks of neuron images were traced by two users to create a labeled dataset and assess the baseline of inter-user variability. All traces were then disconnected at branch points and a deep neural network was trained to detect the correct way of reconnecting the branches. Custom morphological features were generated for each cluster of branch points, in a way that is dependent on a merging scenario but invariant to translations, rotations, and reflections of the cluster in the imaging plane. The features and image volume centered at the branch point were used for training a neural network that concatenates these input streams and outputs the confidence measure for different branch merging scenarios. The designed method significantly reduces the number of topological errors in automated traces and comes close to the accuracy achieved by expert users which is the gold standard in the field.

Microfractures (cracks) are the third most common cause of tooth loss in industrialized countries. If they are not detected early, they continue to progress until the tooth is lost. Cone beam computed tomography (CBCT) has been used to detect microfractures, but has had very limited success. We propose an algorithm to detect cracked teeth that pairs high resolution (hr) CBCT scans with advanced image analysis and machine learning. First, microfractures were simulated in extracted human teeth (n=22). hr-CBCT and microCT scans of the fractured and control teeth (n=14) were obtained. Wavelet pyramid construction was used to generate a phase image of the Fourier transformed scan which were fed to a U-Net deep learning architecture that localizes the orientation and extent of the crack which yields slice-wise probability maps that indicate the presence of microfractures. We then examine the ratio of high-probability voxels to total tooth volume to determine the likelihood of cracks per tooth. In microCT and hr-CBCT scans, fractured teeth have higher numbers of such voxels compared to control teeth. The proposed analytical framework provides a novel way to quantify the structural breakdown of teeth, that was not possible before. Future work will expand our machine learning framework to 3D volumes, improve our feature extraction in hr-CBCT and clinically validate this model. Early detection of microfractures will lead to more appropriate treatment and longer tooth retention.

Retinopathy of prematurity (ROP) is the main cause of blindness in children worldwide. The severity of ROP can be reflected by staging, zoning and plus disease. Specially, some studies have shown that zone recognition is more important than staging. However, due to the subjective factors, ophthalmologists are often inconsistent in their recognition of zones according to fundus images. Therefore, automated zones recognition of ROP is particularly important. In this paper, we propose a new ROP zones recognition network, in which pre-trained DenseNet121 is taken as backbone and a proposed attention block named Spatial and Channel Attention Block (SACAB) and deep supervision strategy are introduced. Our main contributions are: (1) Demonstrating the 2D convolutional neural network model pre-trained on natural images can be fine-tuned for automated zones recognition of ROP. (2) Based on pre-trained DenseNet121, we propose two improved schemes which effectively integrate attention mechanism and deep supervision learning for ROP zoning. The proposed method was evaluated on 662 retinal fundus images (82 zone I, 299 zone II, 281 zone III) from 148 examinations with 5-fold cross validation strategy. The results show that the performance of the proposed ROP zone recognition network achieves 0.8852 for accuracy (ACC), 0.8850 for weighted F1 score (W_F1) and 0.8699 for kappa. The preliminary experimental results show the effectiveness of the proposed method.

The Human BioMolecular Atlas Program (HuBMAP) seeks to create a molecular atlas at the cellular level of the human body to spur interdisciplinary innovations across spatial and temporal scales. While the preponderance of effort is allocated towards cellular and molecular scale mapping, differentiating and contextualizing findings within tissues, organs and systems are essential for the HuBMAP efforts. The kidney is an initial organ target of HuBMAP, and constructing a framework (or atlas) for integrating information across scales is needed for visualizing and integrating information. However, there is no abdominal atlas currently available in the public domain. Substantial variation in healthy kidneys exists with sex, body size, and imaging protocols. With the integration of clinical archives for secondary research use, we are able to build atlases based on a diverse population and clinically relevant protocols. In this study, we created a computed tomography (CT) phase-specific atlas for the abdomen, which is optimized for the kidney organ. A two-stage registration pipeline was used by registering extracted abdominal volume of interest from body part regression, to a high-resolution CT. Affine and non-rigid registration were performed to all scans hierarchically. To generate and evaluate the atlas, multiphase CT scans of 500 control subjects (age: 15 - 50, 250 males, 250 females) are registered to the atlas target through the complete pipeline. The abdominal body and kidney registration are shown to be stable with the variance map computed from the result average template. Both left and right kidneys are substantially localized in the high-resolution target space, which successfully demonstrated the sharp details of its anatomical characteristics across each phase. We illustrated the applicability of the atlas template for integrating across normal kidney variation from 64 cm³ to 302 cm³ .

In this study, we describe a systematic approach to optimize deep-learning-based image processing algorithms using random search. The optimization technique is demonstrated on a phantom-based noise reduction training framework; however, the techniques described can be applied generally for other deep learning image processing applications. The parameter space explored included number of convolutional layers, number of filters, kernel size, loss function, and network architecture (either U-Net or ResNet). A total of 100 network models were examined (50 random search, 50 ablation experiments). Following the random search, ablation experiments resulted in a very minor performance improvement indicating near optimal settings were found during the random search. The top performing network architecture was a U-Net with 4 pooling layers, 64 filters, 3x3 kernel size, ELU activation, and a weighted feature reconstruction loss (0.2×VGG + 0.8×MSE). Relative to the low-dose input image, the CNN reduced noise by 90%, reduced RMSE by 34%, and increased SSIM by 76% on six patient exams reserved for testing. The visualization of hepatic and bone lesions was greatly improved following noise reduction.

Due to the inter-fraction and intra-fraction variation of respiratory motion, it is highly desired to provide real-time volumetric images during the treatment delivery of lung stereotactic body radiation therapy (SBRT) for accurate and active motion management. Motivated by this need, in this study we propose a novel generative adversarial network integrated with perceptual supervision to derive instantaneous 3D image from a single 2D kV projection. Our proposed network, named TransNet, consists of three modules, i.e., encoding, transformation, and decoding modules. Rather than only using image distance loss between the generated 3D image and the ground truth 3D CT image to supervise the network, perceptual loss in feature space is integrated into loss function to force the TransNet to yield accurate lung boundary. Adversarial loss is also used to improve the realism of the generated 3D image. We conducted a simulation study on 20 patient cases, who had undergone 4D-CT scan and received lung SBRT treatments in our institution, and evaluated the efficacy and consistency of our method at four different projection angles, i.e., 0°, 30°, 60° and 90°. For each 3D CT image of a breathing phase in the 4D CT image set, we simulated its 2D projections at these two angles. Then for each projection angle, a patient’s 3D CT images of 9 phases and the corresponding 2D projection data were used for training, with the remaining phase used for testing. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) achieved by our method are 99.5±13.7HU, 23.4±2.3dB and 0.949±0.010, respectively. These results demonstrate the feasibility and efficacy of our method on generating a 3D image from single 2D projection, which provides a potential solution for in-treatment real-time on-board volumetric imaging to guide treatment delivery and ensure the effectiveness of lung SBRT treatment.

Glaucoma is a leading cause of irreversible blindness. Accurate optic disc (OD) and optic cup (OC) segmentation in fundus images is beneficial to glaucoma screening and diagnosis. Recently, convolutional neural networks have demonstrated promising progress in OD and OC joint segmentation in fundus images. However, the segmentation of OC is a challenge due to the low contrast and blurred boundary. In this paper, we propose an improved U-shape based network to jointly segment OD and OC. There are three main contributions: (1) The efficient channel attention (ECA) blocks are embedded into our proposed network to avoid dimensionality reduction and capture cross-channel interaction in an efficient way. (2) A multiplexed dilation convolution (MDC) module is proposed to extract more target features with various sizes and preserve more spatial information. (3) Three global context extraction (GCE) modules are used in our network. By introducing multiple GCE modules between encoder and decoder, the global semantic information flow from high-level stages can be gradually guided to different stages. The method proposed in this paper was tested on 240 fundus images. Compared with U-Net, Attention U-Net, Seg-Net and FCNs, the OD and OC’s mean Dice similarity coefficient of the proposed method can reach 96.20% and 90.00% respectively, which are better than the above networks.

Resting-state functional MRI (rsfMRI) provides important information for studying and mapping the activities and functions of the brain. Conventionally, rsfMRIs are often registered to structural images in the Euclidean space without considering cortical surface geometry. Meanwhile, a surface-based representation offers a relaxed coordinate chart, but this still requires surface registration for group-wise data analysis. In this work, we investigate the performance of two existing surface registration methods in a surface-based rsfMRI analysis framework: FreeSurfer and Hierarchical Spherical Deformation (HSD). To minimize registration bias, we establish shape correspondence using both methods in a groupwise manner that estimates the unbiased average of a given cohort. To evaluate their performance, we focus on neuroanatomical alignment as well as the amount of distortion that can potentially bias surface tessellation for secondary level rsfMRI data analyses. In the pilot analysis, we examine a single timepoint of imaging data from 100 subjects out of an aging cohort. Overall, HSD establishes improved shape correspondence with reduced mean curvature deviation (10.94% less on average per subject, paired t-test: p <10-10) and reduced registration distortion (FreeSurfer: average 41.91% distortion per subject, HSD: 18.63%, paired t-test: p <10-10). Furthermore, HSD introduces less distortion than FreeSurfer in the areas identified in the individual components that were extracted by surface-based independent component analysis (ICA) after spatial smoothing and time series normalization. Consequently, we show that FreeSurfer capture individual components with globally similar but locally different patterns in ICA in visual inspection.

This work aims to develop an automatic multi-organ segmentation approach based on deep learning for head - and- neck region on dual energy CT. The proposed method proposed a Mask scoring R-CNN where comprehensive features are first learnt from two independent pyramid networks and then are combined via deep attention strategy to highlight the informative ones extracted from both two channels of low and high energy CT. To perform multi-organ segmentation and solve the problem of misclassification, a mask scoring subnetwork was integrated into the Mask R-CNN framework to build the correlation between the class of potential detected organ’s ROI and the shape of that organ’s segmentation within that ROI. We trained and tested our model on DECT images from 66 head-and-neck cancer patients with manual contours of 19 organs as training target and ground truth. For large- and mid-sized organs such as brain and parotid, the proposed method successfully achieved average Dice similarity coefficient (DSC) larger than 0.8. For small-sized organs with very low contrast such as chiasm, cochlea, lens and optic nerves, the DSCs ranged between 0.5 and 0.8. With the propose method, using DECT images outperforms using SECT in all 19 organs with statistical significance in DSC (p<0.05). Quantitative results demonstrated the feasibility of the proposed method, the superiority of using DECT to conventional SECT, and the advantage of the proposed R-CNN over FCN. The proposed method has the potential to facilitate the current radiation therapy work flow in treatment planning.

Image-guided radiation therapy (IGRT) is an important technological advancement that has significantly contributed to the accuracy of radiation oncology treatment plan delivery in the last decade. However, the current standard IGRT technique of linac-mounted kilovoltage (kV) cone-beam Computed Tomography (CBCT) has limited soft tissue contrast and is prone to image artifacts, which detract from its clinical utility. It is even worse in chest CBCT compared to other anatomic sites due to respiratory motion, which could lead to mistreatment. Therefore, it is highly desirable to improve CBCT image quality to the level of a planning CT scan. In this study, we propose a novel deep learning-based method, which integrates histogram matching (HM) into a cycle-consistent adversarial network (CycleGAN) framework called HM-CycleGAN, to learn a mapping between chest CBCT images and paired planning CT images obtained at simulation. Histogram matching is performed via an informative maximizing (MaxInfo) loss calculated between planning CT and the synthetic CT (sCT) derived by feeding CBCT into the HM-CycleGAN. The proposed algorithm was evaluated using 15 sets of patient chest CBCT data, each of which has 3-5 daily CBCTs. The planning/simulation CT was used as ground truth for sCTs derived from CBCTs. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and normalized cross-correlation (NCC) indices were used to quantify the correction accuracy of the proposed algorithm. The mean MAE, PSNR and NCC were 63.2 HU, 30.2 dB, 0.96 over all CBCT fractions. The proposed method showed superior image quality, reduced noise, and artifact severity compared to the scatter correction method. Upon further improvement and clinical assessment, this method could further enhance accuracy of the current IGRT technique. The CBCT-based synthetic CT could be the critical component to achieve online adaptive radiation therapy.

Adaptive radiotherapy which relies on patient Cone-Beam CT (CBCT) imaged right before dose delivery to update the contouring and treatment plans has the potential to increase dose delivery accuracy and improve treatment outcomes. In this study, we proposed a synthetic MRI-aided multi-organ segmentation from cone-beam CT for prostate adaptive radiotherapy. A cycle-consistent generative adversarial network (CycleGAN) was first trained with pre-aligned CBCT and MRI image pairs to generate sMRI based on given CBCT image. Feature maps were then extracted from CBCT and sMRI separately using two different U-Nets. The feature maps were combined using attention gates and input to CNN to predict the final segmentation of these critical structures. The segmentation results were evaluated using 100 patients’ datasets. The Dice similarity coefficient (DSC) was 0.96±0.03, 0.91±0.08, 0.93±0.04, 0.95±0.05, and 0.95±0.05 for bladder, prostate, rectum, left femoral head (LFH) and right femoral head (RFH), respectively.

Retinal detachment (RD) refers to the separation of the retinal neuroepithelium layer (RNE) and retinal pigment epithelium (RPE), and retinoschisis (RS) is characterized by the RNE splitting into multiple layers. Retinal detachment and retinoschisis are the main complications leading to vision loss in high myopia. Optical coherence tomography (OCT) is the main imaging method for observing retinal detachment and retinoschisis. This paper proposes a U-shaped convolutional neural network with a cross-fusion global feature module (CFCNN) to achieve automatic segmentation of retinal detachment and retinoschisis. Main contributions include: (1) A new cross-fusion global feature module (CFGF) is proposed. (2) The residual block is integrated into the encoder of the U-Net network to enhance the extraction of semantic information. The method was tested on a dataset consisting of 540 OCT B-scans. With the proposed CFCNN method, the mean Dice similarity coefficient of retinal detachment and retinoschisis segmentation reached 94.33% and 90.29% and were better than some existing advanced segmentation networks.

Optical coherence tomography (OCT) is an imaging modality that is extensively used for ophthalmic diagnosis and treatment. OCT can help reveal disease-related alterations below the surface of the retina, such as retinal fluid which can cause vision impairment. In this paper, we propose a novel context attention-and-fusion network (named as CAF-Net) for multiclass retinal fluid segmentation, including intraretinal fluid (IRF), subretinal fluid (SRF) and pigment epithelial detachment (PED). To deal with the seriously uneven sizes and irregular distributions of different types of fluid, our CAF-Net proposes the context shrinkage encode (CSE) module and context pyramid guide (CPG) module to extract and fuse global context information. The CSE module embedded in the encoder path can ignore redundant information and focus on useful information by a shrinkage function. Besides, the CPG module is inserted between the encoder and decoder, which can dynamically fuse multi-scale information in high-level features. The proposed CAF-Net was evaluated on a public dataset from RETOUCH Challenge in MICCAI2017, which consists of 70 OCT volumes with three types of retinal fluid from three different types of devices. The average of Dice similarity coefficient (DSC) and Intersection over Union (IoU) are 74.64% and 62.08%, respectively.

The inter-fraction motion management of pancreatic radiotherapy remains a challenge in current clinical practice. CBCT-based adaptive radiotherapy is an emerging technique for either offline or online plan adaptations. Accurately delineating tumor targets and organs-at-risk (OARs) is an important step in adaptive re-planning process; however, manual delineation can be labor-intensive and time-consuming. Especially for online adaptation, rapid re-planning is generally required. In this study, we present a fully automated delineation method to expedite the contouring process of adaptive radiotherapy re-planning and dose-volume based plan evaluation and monitoring. In particular, to avoid scatter artifact from CBCT and improve the image quality, a cycle-consistent adversarial network was firstly used to generate synthetic CT given CBCT. Then, a mask scoring regional neural network (RCNN) has been developed to extract the features from synthetic CT for obtaining final segmentation. Metrics including Dice similarity coefficient (DSC), Hausdorff distance 95% (HD95), mean surface distance (MSD), and residual mean square distance (RMS) were used to evaluate our proposed method. Overall, DSC values ranging from 0.82 to 0.94 were achieved among 8 organs.

In the pharmaceutical industry, micro-CT images of Dutch-Belted rabbit fetuses have been used for the assessment of compound-induced skeletal abnormalities in developmental and reproductive toxicology (DART) studies. In the automated approach proposed to assess the morphology of each bone, localization and segmentation of each vertebral bone is a critical task. In this work, we are extending our previous work for the localization of cervical vertebrae to the entire spine following a multivariate regression framework based on a 3D convolutional neural network (CNN). We also introduce a multitasking 3D CNN for the segmentation of each vertebral bone, in which features at the most compact level are processed with two additional convolution layers with max pooling to generate features leading to a classification of whether the patch contains a complete vertebra or not. This multi-tasking mechanism allows us to ensure only complete pieces of vertebrae are segmented. Experimenting on 345 rabbit fetuses with 80/10/10 ratio for training/validation/testing, we were able to achieve successful localization on 94.3% of the cases (i.e. median bone-by-bone localization error under 5 voxels over the entire spine) and an average Dice similarity coefficient (DSC) of 0.80 between automated and ground truth segmentations on the testing set.

Narrow Band Imaging (NBI) is a relatively new endoscopic imaging modality that provides enhanced views of the blood vessels situated near the surface of the airway walls. Certain vessel patterns have been shown to be potential indications of cancerous lesions developing in the airways. Some recent e orts have strived to use NBI bronchoscopy to locate such vessel patterns. To find these patterns, the physician is forced to navigate the bronchoscope through the airways and manually observe potential mucosal (airway wall) vascular patterns. Unfortunately, the bronchoscopic video is often degraded by motion blur and other artifacts, thereby making an already tedious search more impractical. In addition, the degraded video can decrease the potential performance of automated vascular analysis methods. We propose to exploit the richness of the highly informative video sequence by exploring methods for super-resolution video reconstruction and image deblurring|methods which have seen spectacular success in other domains for obtaining high-resolution (HR) images from a sequence of low-resolution (LR) images. Through data derived from NBI video sequences of lung-cancer patients, we demonstrate that such methods can both substantially improve the image quality of images depicting NBI-based vascular patterns and also quantitatively enable more effective automatic segmentation of the vascular patterns.

Detecting suspicious lesions in medical imaging is the important first step in computer-aided detection (CAD) systems. However, detecting abnormalities in breast tissue is difficult due to the lesion's varying size, shape, margin, and contrast with the background tissue. We focused on mass segmentation, a method that provides notable morphological features by outlining contours of masses. Accurate segmentation is crucial for correct diagnosis. Recent advancements in deep learning have improved object detection and segmentation, and these techniques are also being applied to medical imaging studies. We focused on U-net, which is a recently developed mass segmentation algorithm based on a fully convolutional network. The U-net architecture consists of (1) a contracting path to increase the resolution of the output and (2) a symmetric expanding path to better locate the region of interest. The performance of a U-net model was tested with 63 digital mammograms from INbreast, a publicly available database. We trained the model with images resized to 40x40 pixels and conducted 10-fold cross-validation to prevent overfitting. The model's performance with respect to breast density and the lesion's BI-RADS rating was also investigated. Dice coefficients (DC) were used as a performance measure to compare the predicted segmentation of the model with the ground truth. Logistic regression and an analysis of variance were performed to determine the significance of the DCs with regards to breast density and lesion behavior and to calculate the 95% confidence interval. The average DC was 0.80. The difference between DCs for BI-RADS 2 and 4c and for BI-RADS 2 and 5 were significant, suggesting that the model has more difficulty in segmenting benign lesions.

To improve the inter-slice resolution of the orbital bones including cortical and thin bones, we propose an Orbital Bone- Super Resolution (OB-SR) with combined sagittal and axial MAE loss and sagittal Thin-Bone Structure Aware (TSA) loss. Our method consists of three stages: data preprocessing, intermediate slices generation in the sagittal plane and the orbital bone quality improvement in the axial plane. In the intermediate slices generation stage, a 2D CNN consisting of 6 convolutional layers for feature extraction, an up-sampling layer, and 4 convolutional layers for high-frequency detail refinement is performed. The loss is calculated as the sum of the two mean absolute error (MAE) losses to improve the quality of the output image from the up-sampling layer and the last convolutional layer. In the orbital bone quality improvement stage, the generated intermediate slices are compared to the corresponding original axial images using the axial MAE loss. Experimental results showed that our method can generate the intermediate slices with clear orbital bones in the sagittal and axial images and enhances the boundaries of the thin bone.

Robust and accurate segmentation results from high resolution (HR) 3D Magnetic Resonance (MR) images are desirable in many clinical applications. State-of-the-art deep learning methods for image segmentation require external HR atlas image and label pairs for training. However, the availability of such HR labels is limited due to the annotation accuracy and the time required to manually label. In this paper, we propose a 3D label super resolution (LSR) method which does not use an external image or label from a HR atlas data and can reconstruct HR annotation labels only reliant on a LR image and corresponding label pairs. In our method, we present a Deformable U-net, which uses synthetic data with multiple deformation for training and an iterative topology check during testing, to learn a label slice evolving process. This network requires no external HR data because a deformed version of the input label slice acquired from the LR data itself is used for training. The trained Deformable U-net is then applied to through-plane slices to estimate HR label slices. The estimated HR label slices are further combined by label a fusion method to obtain the 3D HR label. Our results show significant improvement compared to competing methods, in both 2D and 3D scenarios with real data.

CT systems with large detector size suffer from lower z-resolution leading to pixelated images and inability to detect small structures thus adversely impacting the diagnosis and screening. Overlap reconstruction can partially reduce the stair-step artifacts but does not improve the effect of wider slice sensitivity profile (SSP) and thus continues to have reduced visibility of smaller structures. In this work, we propose a supervised deep learning method for z-resolution enhancement such that (a) the effective SSP of resulting image is reduced, (b) quantitative values of tissue (CT numbers) and tissue-contrast are preserved; (c) very limited noise enhancement and (d) improved tissue interface in bone/soft tissue. The proposed method devises a super resolution (SURE) network which is trained to map the low resolution (LR) slices to the corresponding high resolution (HR) slices. A 2D network is trained with sagittal and coronal slices with the LR-HR pair sets. The training is performed using ground truth HR slices obtained from high end systems, and the corresponding LR slices are synthesized by either using retro reconstruction with higher slice thickness and spacing or through averaging of slices in z-direction from HR images. The network is trained with both these types of images with helical acquisition volumes from a range of scanners. Qualitative and quantitative analysis is done on the predicted HR images and compared with the original HR images. FWHM for SSP of the predicted HR images reduced from ~0.98 to ~0.73, when the target was 0.64, thus improving the real z-resolution. HU distribution of different tissue types also showed stability in terms of mean value. Noise measured through standard deviation was slightly higher than the LR image but lower than that of original HR images. PSNR also showed consistent improvement on all the cases across 3 different systems.

Refraction-contrast X-ray computed tomography (RCT), which provides a high-contrast image of biological soft tissues, has recently been used in pathology and micro anatomy research. One of the characteristics of RCT is that the projection obtained by RCT system represents the first differentiation of the object's radon transformation. Therefore, the reconstruction of RCT involves an integration process of the projection before conventional back-projection. However, this process causes noise generated by the detector or refraction angle estimation errors to propagate throughout the projection image, which leads to increase ring artifacts (RA) on the reconstructed image. Although various methods for RA removal have been developed in the absorption-based CT (ACT) field, it is difficult to eliminate surface widespread RA on RCT image. In this research, we propose a RA removal method based on conditional generative adversarial network (cGAN) for RCT. This method incorporates a sparse property of Laplacian RCT image which is easily obtained by reconstructing the differential projection. To demonstrate the effectiveness of this method, we applied the proposed and conventional ACT-based methods to simulation data of numerical phantoms and compared them by the root mean square error and the structural similarity. In the result, we demonstrated the proposed method can remove RA better than conventional methods.

In this paper, we present FlyBy CNN, a novel deep learning based approach for 3D shape segmentation. FlyBy-CNN consists of sampling the surface of the 3D object from different view points and extracting surface featuressuch as the normal vectors. The generated 2D images are then analyzed via 2D convolutional neural networkssuch as RUNETs. We test our framework in a dental application for segmentation of intra-oral surfaces. TheRUNET is trained for the segmentation task using image pairs of surface features and image labels as groundtruth. The resulting labels from each segmented image are put back into the surface thanks to our samplingapproach that generates 1-1 correspondence of image pixels and triangles in the surface model. The segmentationtask achieved an accuracy of 0.9

In this study, we propose a new deep learning-based method to predict radiation dose for prostate cancer patients undergoing high-dose-rate (HDR) brachytherapy. The proposed framework consists of three major steps, which are deformable registration via registration network (Reg-Net), consolidation and needle regression. To model the global spatial relationship among multiple organs, binary masks of the target and organs at risk were transformed into distance maps which describe the distance of each local voxel to the organ surfaces. Then, Reg-Net is utilized to deformably register the distance maps and contours of multi-atlas to match those of an arrival patient. By spatial transformation and consolidation, the corresponding dose plans of top-ranked multiple atlases are registered and fused to generate a synthetic HDR dose distribution of an arrival patient. A retrospective study on 40 patients was used to evaluate the proposed method’s efficiency. Comparison of dose volume histogram metrics of predicted dose and clinical delivered dose shows that no statistically significant difference is found. These results demonstrate the feasibility and efficacy of our deep learning-based method for HDR prostate dose prediction.

Retinal capillary non-perfusion (CNP) is one of diabetic retinal vascular diseases. As the capillaries are occluded, blood stops flowing to certain regions of the retina, resulting in the formation of non-perfused regions. Accurate determination of the area and change of CNP is of great significance in clinical judgment of the extent of vascular obstruction and selection of treatment methods. This paper proposes a novel generative adversarial framework, and realize the segmentation of non-perfusion regions in fundus fluorescein angiography images. The generator G of GANs is trained to produce “real” images; while an adversarially trained discriminator D is trained to do as well as possible at detecting “fakes” images from the generator. In this paper, a U-shape network is used as the discriminator. Our method is validated using on 138 clinical fundus fluorescein angiography images. Experimental results show that our method achieves more accurate segmentation results than that of state-of-the-art approaches.

About 5-8% of individuals over the age of 60 have dementia. With our ever-aging population this number is likely to increase, making dementia one of the most important threats to public health in the 21st century. Given the phenotypic overlap of individual dementias the diagnosis of dementia is a major clinical challenge, even with current gold standard diagnostic approaches. However, it has been shown that certain dementias show specific structural characteristics in the brain. Progressive supranuclear palsy (PSP) and multiple system atrophy (MSA) are prototypical examples of this phenomenon, as they often present with characteristic brainstem atrophy. More detailed characterization of brain atrophy due to individual diseases is urgently required to select biomarkers and therapeutic targets that are meaningful to each disease. Here we present a joint multi-atlas-segmentation and deep-learning-based segmentation method for fast and robust parcellation of the brainstem into its four substructures, i.e., the midbrain, pons, medulla, and superior cerebellar peduncles (SCP), that in turn can provide detailed volumetric information on the brainstem sub-structures affected in PSP and MSA. The method may also benefit other neurodegenerative diseases, such as Parkinson’s disease; a condition which is often considered in the differential diagnosis of PSP and MSA. Comparison with state-of-the-art labeling techniques evaluated on ground truth manual segmentations demonstrate that our method is significantly faster than prior methods as well as showing improvement in labeling the brainstem indicating that this strategy may be a viable option to provide a better characterization of the brainstem atrophy seen in PSP and MSA.

Deep learning approaches may help radiologists in the early diagnosis and timely treatment of cerebrovascular diseases. Accurate cerebral vessel segmentation of Time-of-Flight Magnetic Resonance Angiographs (TOFMRAs) is an essential step in this process. This study investigates deep learning approaches for automatic, fast and accurate cerebrovascular segmentation for TOF-MRAs. The performance of several data augmentation and selection methods for training a 2D and 3D U-Net for vessel segmentation was investigated in five experiments: a) without augmentation, b) Gaussian blur, c) rotation and flipping, d) Gaussian blur, rotation and flipping and e) different input patch sizes. All experiments were performed by patch-training both a 2D and 3D U-Net and predicted on a test set of MRAs. Ground truth was manually defined using an interactive threshold and region growing method. The performance was evaluated using the Dice Similarity Coefficient (DSC), Modified Hausdor Distance and Volumetric Similarity, between the predicted images and the interactively defined ground truth. The segmentation performance of all trained networks on the test set was found to be good, with DSC scores ranging from 0.72 to 0.83. Both the 2D and 3D U-Net had the best segmentation performance with Gaussian blur, rotation and flipping compared to other experiments without augmentation or only one of those augmentation techniques. Additionally, training on larger patches or slices gave optimal segmentation results. In conclusion, vessel segmentation can be optimally performed on TOF-MRAs using a trained 3D U-Net on larger patches, where data augmentation including Gaussian blur, rotation and flipping was performed on the training data.

A Dual-Energy CT (DECT) with a spectral detector greatly extends the capabilities of CT by incorporating energy-dependent information of the X-ray attenuation. In order to fully exploit DECT capabilities, it is required to perform a process known as spectral decomposition. However, this process is sensitive to noise, suffers from reduced photon count per layer in DECT scans and generates anti-correlated noise in the estimated material specific images. In order to overcome these problems, the Anti-Correlated Rudin, Osher and Fatemi (AC-ROF) model is applied for noise reduction, exploiting the relationship between the material-specific images. However, this model deteriorates the structural information with intense noise. In this paper we propose to extend this method by integrating it into an iterative reconstruction procedure to improve the noise reduction performance. The resulting algorithm is called Iterative Reconstruction AC-ROF, or IR-AC-ROF. We have tested AC-ROF and IR-AC-ROF algorithms with realistic brain simulation phantoms and show encouraging results indicating that the resulting material-specific images of IR-AC-ROF can generate better mono-energetic images with improved brain structure visibility. This demonstrates the benefit of including the noise reduction constraints within the reconstruction procedure, rather than using them in a post-processing step.

Bias Field correction is a crucial step in MRI preprocessing. The bias field affects the intensity uniformity in MRI images. This effect is mostly due to the in-homogeneity in the magnetic fields or variation in magnetic susceptibility during acquisition. The presence of bias field affects the tissue classification stage, as most of the common methods assume uniform intensities across same tissue. We present a deep learning approach that uses an autoencoding architecture to predict the bias field. The performance of the method is evaluated based on tissue classification accuracy compared to the ground truth result. The proposed method outperforms a traditional histogram based method and results in a more accurate tissue classification.

Cone-beam computed tomography (CBCT) has been widely used in image-guided radiation therapy. Deformable image registration (DIR) among inter-fraction CBCTs is essential in order to evaluate the geometric and anatomic changes in adaptive radiation therapy. However, the bulky data size and, more importantly, the poor quality of CBCT images, such as scattering/beam hardening artifacts, motion artifacts and CT number inaccuracy, impedes the fast and accurate image registration. In this study, we propose a novel unsupervised spatial transformation network (STN)-based DIR method for inter-fraction CBCT image registration. To evaluate the proposed method, datasets of abdominal CBCT patient images are retrospectively investigated. The preliminary results show that the proposed method, with average target registration error of 2.91±1.16 mm, has great potential in quantifying the daily anatomy changes for adaptive radiation therapy.

The study aims to assess the performance in differentiating benign from malignant kidney masses using a radiomics approach. For this retrospective study we worked with the scans of 210 patients from the publicly available KiTS19 dataset. Each scan had segmentations of the healthy kidney tissue, benign lesions and malignant tumors. In Phase 1 of our study, we reduced the number of radiomic features (105) extracted from the scans by using four feature selection and ranking algorithms: recursive feature elimination (RFE), fisher score, partial least square discriminant analysis (PLS-DA) and linear support vector machine (l-SVM). The features selected by each method were then used to train a series of random forest (RF) classifiers. In Phase 2, we trained a convolutional neural network (CNN) to automatically perform the segmentation of benign and malignant kidney masses. We then placed the best performing RF classifier from Phase 1 in series with the CNN to see if it corrected its prediction. The best classification performance was obtained when training a RF classifier with the 8 features selected by the RFE method (accuracy: 0.974). This RF model applied to the segmentations derived from the neural network improved the CNN’s overall results: the dice score for malignant mass went from 0.74 to 0.79 and dice score for benign mass from 0.55 to 0.80. The studied radiomics approach proved to be an accurate solution to classify benign and malignant kidney masses. A deep learning algorithm has shown to also benefit from its predictive power.

The thalamus is an internal structure of the brain whose changes are related to diseases such as multiple sclerosis and Parkinson’s disease. Thus, the thalamus segmentation is an important step in studies and applications related to these disorders, for example, for shape measuring and surgical planning. The most used software and tools for brain structures segmentation employ atlas-based algorithms that usually require long processing times and sometimes lead to inaccurate results on sub-cortical structures. New methods, that minimize those problems, using deep learning for segmenting brain structures have been recently proposed. However, for some structures such as the thalamus, these methods still tend to have unsatisfactory results since they rely only on T1w images, where the contrast can be low or absent. Aiming to overcome these issues, we proposed a Convolutional Neural Network (CNN) trained with multi-modal data (structural and diffusion MRI) and the use of silver standard masks created from multiple automatic segmentations. Results on a subset of 190 subjects from the Human Connectome Project (HCP) showed an improvement in segmentation quality, confirming the effectiveness of diffusion data in differentiating tissues due to measured micro-structural properties.

Pathology is an important subject in the treatment of pancreatic cancer. The tumor presented in the pathological images includes not only the tumor cells, but also the surrounding background structures. Automatic and accurate gland segmentation in histopathology images plays a significant role for cancer diagnosis and clinical application, which assist pathologists to diagnose the malignancy degree of pancreas caner. Due to the large variability of size and shape in glandular appearance and the heterogeneity between different cells, it is a challenging task to accurately segment glands in histopathology images. In this paper, a selective multi-scale attention (SMA) block is proposed for gland segmentation. First, a selection unit is used between the encoder and decoder to select features by amplifying effective information and suppressing redundant information according to a factor obtained during training. Second, we propose a multi-scale attention module to fuse feature maps at different scales. Our method is validated on a dataset of 200 images of size 512×512 from 24 H&E stained pancreas histological images. Experimental results show that our method achieves more accurate segmentation results than that of state-of-the-art approaches.

Low-dose denoising is an effective method that utilizes the power of CT for screening while avoiding high radiation exposure. Several research work has reported the feasibility of deep learning based denoising, but none of them have explored the influence of different network designs on the denoising performance. In this work, we explored the impact of three commonly adapted network design concepts in denoising: (1) Network structure, (2) Residual learning, and (3) Training loss. We evaluated the network performance using the dataset containing 76 real patient scans from Mayo Clinic Low-dose CT dataset. Experimental results demonstrated that residual blocks and residual learning are recommended to be utilized in design, while pooling is not recommended. In addition, among the classical training losses, the mean absolute error (L1) loss outperforms the mean squared error (MSE) loss.

The contrast enhancement of tumor regions in medical images can improve the performance of tumor detection, segmentation, and diagnosis. However, the main existing enhancement methods aim to enhance the contrast and the resolution on the whole image, instead of highlighting the lesion regions. The blurry edges lead to the difficulty of distinguishing the tumor from the healthy tissues accurately. This issue can be hardly solved by those global enhancement methods. In this paper, we focus on the local enhancement and propose a novel deep learning-based approach called U-SDRC to enhance the contrast between tumor regions and surrounding background tissues to make the tumor regions distinguishable. We introduce a U-net deep network to tackle this problem and present a novel SDRC loss function to achieve the goal of enhancing tumor lesions and simultaneously preserving the original appearance of other regions in the image. We evaluate our approach on a clinical dataset that comprises 1394 liver CT slices. The encouraging experimental results show that the proposed method can lead to a good visual enhancement effect and bring improvements to medical tasks such as tumor segmentation and diagnosis.

Thoracic aortic aneurysm (TAA) growth is currently assessed by changes in maximal aortic diameter (i.e., radial growth of the vessel). However, there is growing awareness that longitudinal aortic growth (i.e., elongation) is an important metric of disease status, albeit one that is difficult to measure using current clinical techniques. Previously we have proposed a method to assess 3D changes in aortic wall growth/deformation using deformable image registration with interpolation of the spatial Jacobian determinant to the aortic surface. Here we propose a method to re-orient the Jacobian into directional components relative to the aortic surface rather than the image space, allowing clinicians and researchers to isolate and study the pathologic effects of each directional component of growth separately. To this end, we first perform a deformable image registration between two aortic geometries. Second, we segment the aortic surface and centerline in the fixed image, and use the resulting geometry to construct anatomically-based local coordinate systems at each voxel of the aortic surface. Using the Jacobian matrix field resulting from the deformable registration, we can obtain the anatomically oriented Jacobian components by rotating the Jacobian matrix at each voxel so that it is aligned with the anatomically based local coordinate system. Through experiments on toy cylinders and real clinical cases, we show clear differences between the Jacobian determinant and its directional components, with the directional Jacobian component being able to remove one directional change (e.g., longitudinal) while maintaining the other (e.g., cross-sectional).

Deep learning-based segmentation algorithms for medical image require massive training datasets with accurate annotations, which is costly since it takes much human effort to manually labeling from scratch. Therefore, interactive image segmentation is important and may greatly improve the efficiency and accuracy of medical image labeling. Some interactive segmentation methods (e.g. Deep Extreme Cut and Deepgrow) may improve the labeling through minimal interactive input. However these methods only utilize the initial manually input information, while existing segmentation results (such as annotations produced by nonprofessionals or conventional segmentation algorithms) cannot be utilized. In this paper, an interactive segmentation method is proposed to make use of both existing segmentation results and human interactive information to optimize the segmentation results progressively. In this framework, the user only needs to click on the foreground or background of the target individual on the medical image, the algorithm could adaptively learn the correlation between them, and automatically completes the segmentation of the target. The main contributions of this paper are: (1) We adjusted and applied a convolutional neural network which takes medical image data and user's clicks information as input to achieve more accurate segmentation of medical images. (2) We designed an iterative training strategy to realize the applicability of the model to deal with different number of clicks data input. (3) We designed an algorithm based on false positive and false negative regions to simulate the user's clicks, so as to provide enough training data. By applying the proposed method, users can easily extract the region of interest or modify the segmentation results by multiple clicks. The experimental results of 6 medical image segmentation tasks show that the proposed method achieves more accurate segmentation results by at most five clicks.

The use of magnetic resonance-ultrasound fusion targeted biopsy improves diagnosis of aggressive prostate cancer. Fusion of ultrasound & magnetic resonance images (MRI) requires accurate prostate segmentations. In this paper, we developed a 2.5 dimensional deep learning model, ProGNet, to segment the prostate on T2-weighted magnetic resonance imaging (MRI). ProGNet is an optimized U-Net model that weighs three adjacent slices in each MRI sequence to segment the prostate in a 2.5D context. We trained ProGNet on 529 cases where experts annotated the whole gland (WG) on axial T2-weighted MRI prior to targeted prostate biopsy. In 132 cases, experts also annotated the central gland (CG) on MRI. After five-fold cross-validation, we found that for WG segmentation, ProGNet had a mean Dice similarity coefficient (DSC) of 0.91±0.02, sensitivity of 0.89±0.03, specificity of 0.97±0.00, and an accuracy of 0.95±0.01. For CG segmentation, ProGNet achieved a mean DSC 0.86±0.01, sensitivity of 0.84±0.03, specificity of 0.99±0.01, and an accuracy of 0.96±0.01. We then tested the generalizability of the model on the 60-case NCI-ISBI 2013 challenge dataset and on a local, independent 61-case test set. We achieved DSCs of 0.81±0.02 and 0.72±0.02 for WG and CG segmentation on the NCI-ISBI 2013 challenge dataset, and 0.83±0.01 and 0.75±0.01 for WG and CG segmentation on the local dataset. Model performance was excellent and outperformed state-of-art U-Net and holistically-nested edge detector (HED) networks in all three datasets.

Pharmacokinetic modeling is a mathematical modeling technique that examines the dynamics of concentration curves to reveal information about tissue microvasculature. Typically, image registration is performed as a pre- processing step to remove motion in dynamic contrast-enhanced (DCE) image sequences and ensure accurate pharmacokinetic analysis. In this work, we introduce a registration method for correcting motion in a sequence of DCE images. The proposed method involves the use of Tofts pharmacokinetic model to generate a sequence of reference images. These images simplify the challenging task of registering DCE images by pairing each frame in the motion- corrupted sequence with a reference image that resembles the overall contrast enhancement of the template. Abdominal DCE-MR images were used for validation. Reduction of motion in the registered sequence was observed both visually and quantitatively. Both global and local measures of registration accuracy obtained from the registered sequence and its associated signal intensity curves were smaller than their pre-registration counterparts.

Recently, deep learning has become much more popular in computer vision applications. The Convolutional Neural Network (CNN) has brought a breakthrough in image segmentation, especially for medical images. In this regard, the UNet is the predominant approach to the medical image segmentation task. The U-Net not only performs well in segmenting multimodal medical images generally, but also in some difficult cases. We found, however, that the classical U-Net architecture has limitations in several respects. Therefore, we applied modifications: 1) designed efficient CNN architecture to replace encoder and decoder, 2) applied residual module to replace skip connection between encoder and decoder to improve, based on the-state-of-the-art U-Net model. Following these modifications, we designed a novel architecture -- DC-UNet, as a potential successor to the U-Net architecture. We created a new effective CNN architecture and built the DC-UNet based on this CNN. We have evaluated our model on three datasets with difficult cases and have obtained a relative improvement in performance of 2.90%, 1.49%, and 11.42% respectively compared with classical UNet. In addition, we used the Tanimoto similarity measure to replace the Jaccard measure for gray-to-gray image comparisons.

Identifying target volumes is a key component in image-guided adaptive radiotherapy. Furthermore, propagating contours from treatment planning images to images acquired during treatment is a difficult image registration problem due to organ motion. Among many deformable image registration techniques, block matching has been studied extensively in prostate and bladder cancer. Here, we propose a two-pass, three-dimensional disparity regularisation that accounts for anatomical constraints by distance and neighbouring motion vectors' orientation. This approach improves the Dice similarity score of the delineation/contour propagation and results in a non- iterative and non-pyramidal method with reduced computational time.

Pathologic myopia (PM) is a major cause of legal blindness in the world. Linear lesions are closely related to PM, which include two types of lesions in the posterior fundus of pathologic eyes in optical coherence tomography (OCT) images: retinal pigment epithelium-Bruch's membrane-choriocapillaris complex (RBCC) disruption and myopic stretch line (MSL). In this paper, a fully automated method based on U-shape network is proposed to segment RBCC disruption and MSL in retinal OCT images. Compared with the original U-Net, there are two main improvements in the proposed network: (1) We creatively propose a new downsampling module named as feature aggregation pooling module (FAPM), which aggregates context information and local information. (2) Deep supervision module (DSM) is adopted to help the network converge faster and improve the segmentation performance. The proposed method was evaluated via 3-fold crossvalidation strategy on a dataset composed of 667 2D OCT B-scan images. The mean Dice similarity coefficient, Sensitivity and Jaccard of RBCC disruption and MSL are 0.626, 0.665, 0.491 and 0.739, 0.814, 0.626, respectively. The primary experimental results show the effectiveness of our proposed method.

Microcalcifications are small deposits of calcium that appear in mammograms as bright white specks on the soft tissue background of the breast. Microcalcifications may be a unique indication for Ductal Carcinoma in Situ breast cancer, and therefore their accurate detection is crucial for diagnosis and screening. Manual detection of these tiny calcium residues in mammograms is both time-consuming and error-prone, even for expert radiologists, since these microcalcifications are small and can be easily missed. Existing computerized algorithms for detecting and segmenting microcalcifications tend to suffer from a high false-positive rate, hindering their widespread use. In this paper, we propose an accurate calcification segmentation method using deep learning. We specifically address the challenge of keeping the false positive rate low by suggesting a strategy for focusing the hard pixels in the training phase. Furthermore, our accurate segmentation enables extracting meaningful statistics on clusters of microcalcifications.

Skin cancer affects more than 3 million people only in the US. Comprehensive microscopic databases include around 30 thousand samples, limiting the richness of patterns that can be presented to machine learning. To this end, generative models such as GANs have been proposed for creating realistic synthetic images but, despite their popularity, they are often difficult to train and control. Recently an autoregressive approach based on a quantized autoencoder showed state of the art performances while being simple to train and provide synthetic data generation opportunities. In the first part of this paper we evaluate the training of VQ-VAE-2 with different latent space configuration. In the second part, we show how to use a learned prior over the latent space with PixelSNAIL to generate and modify skin lesions. We show how this process can be used for powerful data augmentation and visualization for skin health, evaluating it on a downstream application that classifies malignant lesions

Diabetic retinopathy (DR) is the most common chronic complication of diabetes and the first blinding eye disease in the working population. Hard exudates (HE) is an obvious symptom of diabetic retinopathy, which has high reflectivity to light and appears as hyperreflective foci (HRF) in optical coherence tomography (OCT) images. Based on the research and improvement of U-Net, this paper proposes a selfadaptive network (SANet) for HRF segmentation. There are two main improvements in the proposed SANet: (1) In order to simplify the learning process and enhance the gradient propagation, the ordinary convolution block in the encoder structure is replaced by a dual residual module (DRM). (2) The novel self-adaptive module (SAM) is embedded in the deep layer of the model, which enables the network to integrate local features and global dependencies adaptively, and makes it adapt to the irregular shape of HRF. The dataset consists of 112 2D OCT B-scan images, which were verified by four-fold cross validation. The mean and standard deviation of Dice similarity coefficient, Jaccard index, Sensitivity and Precision are 73.69±0.72%, 59.17±1.00%, 74.57±1.16% and 75.54±1.35%, respectively. The experimental results show that the proposed method can segment HRF successfully and the performance is better than the original U-Net.

Age-Related Macular Degeneration (AMD) is a common eye disease characterized by the build-up of drusen, small deposits of extracellular materials in the macula. Early detection of drusen is key to understanding the progression of AMD. Therefore, accurate and robust segmentation of drusen during AMD progression is important for automated detection, classification, diagnosis, and prognosis tasks. Spectral-domain optical coherence tomography (SD-OCT) is a popular macular imaging modality used for these tasks. However, because of the trade-off between resolution and speed, often clinical OCT scans will contain far fewer images per volume than the 100-200 images the drusen segmentation literature generally utilizes. To address this disparity, we develop a novel drusen segmentation algorithm for SD-OCT volumes with low volumetric resolution. We achieve comparable results to similar work, while using on average 16% of the volumetric information. We evaluate our segmentation approach on manually segmented images by two graders, and achieve median Dice coefficient scores of 0.75 and 0.66, respectively, which are close to our median inter-reader variability score of 0.75.

Prior neuroimaging studies have demonstrated isolated structural and connectivity changes in the brain due to Alzheimer’s Disease (AD). However, how these changes relate to each other is not well understood. We present a preliminary study to begin to fill this gap by leveraging joint independent component analysis (jICA). We explore how jICA performs in an analysis of T1 and diffusion weighted MRI by characterizing the joint changes of complex cortical surface and structural connectivity metrics in AD in subjects from the Baltimore Longitudinal Study of Aging. We calculate 588 region-based cortical metrics and 4,753 fractional anisotropy-based connectivity metrics and project them into a low-dimensional manifold with principal component analysis. We perform jICA on the manifold and subsequently backproject the independent components to the original data space. We demonstrate component stability with 3-fold cross validation and find differential component loadings between 776 cognitively unimpaired control subjects and 23 with AD that generalizes across folds. In addition, we perform the same analysis on the surface and connectivity metrics separately and find that the joint approach identifies both novel and similar components to the separate approaches. To illustrate the joint approach’s primary utility, we provide an example hypothesis for how surface and connectivity components may vary together with AD. These preliminary results suggest jointly varying independent cortical surface and structural connectivity components can be consistently extracted from MRI data and provide a data-driven way for generating novel hypotheses about AD that may not be captured by separate analyses.

A fundamental problem in employing deep learning algorithms in the medical field is the lack of labeled data and severe class imbalance. In this work, we present novel ways to enlarge small scale datasets. We introduce an autoencoder framework comprised of an encoder and a StyleGAN generator to embed images into the latent space of StyleGAN. The autoencoder learns the disentangled latent representation of the data allowing for encoding real images to the latent space and manipulating the latent vector in a meaningful manner. We suggest ways to use the encoder along with the unique architecture of the StyleGAN generator to control the synthesized images and thus, create class-specific images that can be used to train and improve existing deep learning algorithms.

We propose a learning-based method to automatically segment lung tumor from positron emission tomography (PET)/computed tomography (CT) images. A dual pyramid mask R-CNN is introduced to enable end-to-end segmentation. To avoid the effect of useless region, mask R-CNN is used to get rid of non-tumor regions via fist locate the tumor region-of-interest (ROI) and then segment tumor within that ROI. Dual pyramid networks are used as backbone in mask R-CNN to extract comprehensive features from both CT and PET images. The binary mask of tumor of an arrival patient’s CT and PET image is generated by the welltrained network. To evaluate the proposed method, we retrospectively investigate 42 lung PET/CT datasets. On each dataset, lung tumor was delineated by physicians and was served as ground truth and training target. The proposed method was trained and evaluated by a six-fold cross validation strategy. The average centroid distance, volume difference and DSC value among all 42 datasets are 0.496±0.933mm, 0.303±0.458cc and 0.894±0.080, which indicates that the proposed method is able to generate target contour within 0.5mm error in displacement, 0.3cc error in volume size and around 90% overlapping compared with ground truth. The proposed method has great potential in improving the efficiency and mitigating the observer-dependence in tumor detection and delineation for diagnosis and therapy.

Multi-atlas segmentation methods will benefit from atlases covering the complete spectrum of population patterns, while the difficulties in generating such large enough datasets and the computation burden required in the segmentation procedure reduce its practicality in clinical application. In this work, we start from a viewpoint that different parts of the target object can be recognized by different atlases and propose a precision atlas selection strategy. By comparing regional similarity between target image and atlases, precision atlases are ranked and selected by the frequency of regional best match, which have no need to be globally similar to the target subject at either image-level or object-level, largely increasing the implicit patterns contained in the atlas set. In the proposed anatomy recognition method, atlas building is first achieved by all-totemplate registration, where the minimum spanning tree (MST) strategy is used to select a registration template from a subset of radiologically near-normal images. Then, a two-stage recognition process is conducted: in rough recognition, sub-image level similarity is calculated between the test image and each image of the whole atlas set, and only the atlas with the highest similarity contributes to the recognition map regionally; in refined recognition, the atlases with the highest frequencies of best match are selected as the precision atlases and are utilized to further increase the accuracy of boundary matching. The proposed method is demonstrated on 298 computed tomography (CT) images and 9 organs in the Head and Neck (HN) body region. Experimental results illustrate that our method is effective for organs with different segmentation challenge and samples with different image quality, where remarkable improvement in boundary interpretation is made by refined recognition and most objects achieve a localization error within 2 voxels.

Thalamus segmentation plays an important role in studies that are related to neural system diseases. Existing thalamus segmentation algorithms use traditional image processing techniques on magnetic resonance images (MRI), which suffer from accuracy and efficiency. In recent years, deep convolutional neural networks (CNN) have been able to outperform many conventional algorithms in medical imaging tasks. We propose segmenting the thalamus using a 3D CNN that takes an MPRAGE image and a set of feature images derived from a diffusion tensor image (DTI). Experimental results demonstrate that using CNNs to segment the thalamus can improve accuracy and efficiency on various datasets.

Barrett's Esophagus is a precursor of esophageal adenocarcinoma, one of the most lethal forms of cancer. Volumetric laser endomicroscopy (VLE) is a relatively new technology used for early detection of abnormal cells in BE by imaging the inner tissue layers of the esophagus. Computer-Aided Detection (CAD) shows great promise in analyzing the VLE frames due to the advances in deep learning. However, a full VLE scan produces 1,200 scans of 4,096 x 2,048 pixels, making automated pre-processing for the tissue of interest extraction necessary. This paper explores an object detection for tissue detection in VLE scans. We show that this can be achieved in real time with very low inference time, using single-stage object detection like YOLO. Our best performing model achieves a value of 98.23% for the mean average precision of bounding boxes correctly predicting the tissue of interest. Additionally, we have found that the tiny YOLO with Partial Residual Networks architecture further reduces the inference speed with a factor of 10, while only sacrificing less than 1% of accuracy. This proposed method does not only segment the tissue of interest in real time without any latency, but it can also achieve this efficiently using limited GPU resources, rendering it attractive for embedded applications. Our paper is the first to introduce object detection as a new approach for VLE-data tissue segmentation and paves the way for real-time VLE-based detection of early cancer in BE.

Cardiac Magnetic Resonance (CMR) is the most effective tool for the assessment and diagnosis of a heart condition, which malfunction is the world's leading cause of death. Software tools leveraging Artficial Intelligence already enhance radiologists and cardiologists in heart condition assessment but their lack of transparency is a problem. This project investigates if it is possible to discover concepts representative for different cardiac conditions from the deep network trained to segment cardiac structures: Left Ventricle (LV), Right Ventricle (RV) and Myocardium (MYO), using explainability methods that enhances classification system by providing the score-based values of qualitative concepts, along with the key performance metrics. With introduction of a need of explanations in GDPR explainability of AI systems is necessary. This study applies Discovering and Testing with Concept Activation Vectors (D-TCAV), an interpretaibilty method to extract underlying features important for cardiac disease diagnosis from MRI data. The method provides a quantitative notion of concept importance for disease classified. In previous studies, the base method is applied to the classification of cardiac disease and provides clinically meaningful explanations for the predictions of a black-box deep learning classifier. This study applies a method extending TCAV with a Discovering phase (D-TCAV) to cardiac MRI analysis. The advantage of the D-TCAV method over the base method is that it is user-independent. The contribution of this study is a novel application of the explainability method D-TCAV for cardiac MRI analysis. D-TCAV provides a shorter pre-processing time for clinicians than the base method.

Liver stiffness is an essential clinical biomarker for diagnosing liver fibrosis and cirrhosis. In current clinical practice, elastography techniques are standard non-invasive diagnosis tools to assess stiffness of liver, using either Ultrasound (US) or magnetic resonance imaging (MRI). However, the US elastography yields ≈ 10 % failure rate and degraded performance on obese patients, while the MR elastography is costlier and less available. Compared with US and MRI, the computerized tomography (CT) imaging has not been widely used in measuring liver stiffness. In this paper, we performed a pilot study to assess if volumetric variations of liver can be captured from paired inspiratory-expiratory chest (PIEC) CT scans. To enable the assessment, we propose a Hierarchical Intra- Patient Organ-specific (HIPO) registration pipeline to quantify the partial liver volumetric variations with lung pressure from a respiratory cycle. The PIEC protocol is employed since it naturally provides two paired CT scans with liver deformation from regulated respiratory motions. For the subjects whose registration results passed both an automatic quantitative quality assurance (QA) and another visual qualitative QA, 6:0% average volumetric variations of liver were measured, from inspiratory phase to expiratory phase. Future clinical validations will be required to validate the findings in this pilot study.

The state-of-the-art methods of Magnetic Resonance Imaging (MRI) denoising technologies have improved significantly in the past decade, particularly those based in deep learning. However, the major issues in deep learning based denoising algorithms is both that the model architectures are not built for the complex noise distributions inherent in MRI, and that the data given to these algorithms is typically synthetic, and thus, they fail to generalize to spatially variant noise distributions. The noise varies greatly dependent upon such factors as pulse sequence of the MRI sequence, reconstruction method, coil configuration, physiological activities, etc. To overcome these issues, we have created HydraNet, a multi-branch deep neural network architecture that learns to denoise MR images at a multitude of noise levels, and which has critically been trained using only real image pairs of high and low signal-to-noise ratio (SNR) images. We prove the superiority of HydraNet at denoising complex noise distributions in comparison to the leading deep learning method in our experimentation, in addition to non-local collaborative filtering-based methods, quantitatively in both Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM), and qualitatively upon inspection of denoised MRI samples.

Accurate vertebral segmentation is important for image-related assessment of spine pathologies. Recent spine segmentation techniques focus on deep learning-based methods. However, they typically require a large training dataset with accurate annotations, which may not be feasible in practice. To mitigate the challenge, here we propose a data augmentation scheme based on a single spine to incorporate biomechanically constrained intervertebral motion (rotation and translation) to generate a large training dataset (N=3000). Vertebrae are then segmented using a point cloud-based deep learning architecture, PointNet++, which appears not to have been applied to spine segmentation before. Spine testing samples (N=90) are first generated by applying our data augmentation technique to three separate spines. Segmentation performances are compared with two baseline data augmentation methods that do not include intervertebral motion. Then, we further evaluate our technique based on 8 spine samples from actual image acquisitions. In both cases, we show that incorporation of vertebral level-wise data augmentation improves segmentation accuracy in terms of mean Dice coefficient (e.g., 0.932 vs. 0.902 for augmented testing samples and 0.940 vs. 0.924 for acquired testing samples). These results suggest that our vertebral level-wise data augmentation is useful to facilitate deep learning-based spine segmentation, which is important especially when it is not feasible to generate accurate annotations for a large number of training samples.

Time-of-Flight Magnetic Resonance Angiographs (TOF-MRAs) enable visualization and analysis of cerebral arteries. This analysis may indicate normal variation of the configuration of the cerebrovascular system or vessel abnormalities, such as aneurysms. A model would be useful to represent normal cerebrovascular structure and variabilities in a healthy population and to differentiate from abnormalities. Current anomaly detection using autoencoding convolutional neural networks usually use a voxelwise mean-error for optimization. We propose optimizing a variational-autoencoder (VAE) with structural similarity loss (SSIM) for TOF-MRA reconstruction. A patch-trained 2D fully-convolutional VAE was optimized for TOF-MRA reconstruction by comparing vessel segmentations of original and reconstructed MRAs. The method was trained and tested on two datasets: the IXI dataset, and a subset from the ADAM challenge. Both trained networks were tested on a dataset including subjects with aneurysms. We compared VAE optimization with L2-loss and SSIM-loss. Performance was evaluated between original and reconstructed MRAs using mean square error, mean-SSIM, peak-signal-to-noise-ratio and dice similarity index (DSI) of segmented vessels. The L2-optimized VAE outperforms SSIM, with improved reconstruction metrics and DSIs for both datasets. Optimization using SSIM performed best for visual image quality, but with discrepancy in quantitative reconstruction and vascular segmentation. The IXI dataset had overall better performance, potentially due to the larger, more diverse training data. Reconstruction metrics, including SSIM, were lower for MRAs including aneurysms. A SSIM-optimized VAE improved the visual perceptive image quality of TOF-MRA reconstructions. A L2-optimized VAE performed best for TOF-MRA reconstruction, where the vascular segmentation is important. SSIM is a potential metric for anomaly detection of MRAs.

At present, high myopia has become a hot spot for eye diseases worldwide because of its increasing prevalence. Linear lesion is an important clinical signal in the pathological changes of high myopia. ICGA is considered to be the “Ground Truth” for the diagnosis of linear lesions, but it is invasive and may cause adverse reactions such as allergy, dizziness, and even shock in some patients. Therefore, it is urgent to find a non-invasive imaging modality to replace ICGA for the diagnosis of linear lesions. Multi-color scanning laser (MCSL) imaging is a non-invasive imaging technique that can reveal linear lesion more richly than other non-invasive imaging technique such as color fundus imaging and red-free fundus imaging and some other invasive one such as fundus fluorescein angiography (FFA). To our best knowledge, there are no studies focusing on the linear lesion segmentation based on MCSL images. In this paper, we propose a new U-shape based segmentation network with multi-scale and global context fusion (SGCF) block named as SGCNet to segment the linear lesion in MCSL images. The features with multi-scales and global context information extracted by SGCF block are fused by learnable parameters to obtain richer high-level features. Four-fold cross validation was adopted to evaluate the performance of the proposed method on 86 MCSL images from 57 high myopia patients. The IoU coefficient, Dice coefficient, Sensitivity coefficient and Specialty are 0.494±0.109, 0.654±0.104, 0.676±0.131 and 0.998±0.002, respectively. Experiment results indicate the effectiveness of the proposed network.

Object-based co-localization of uorescent signals allows the assessment of interactions between two (or more) biological entities using spatial information. It relies on object identification with high accuracy to separate uorescent signals from the background. Object detectors using convolutional neural networks (CNN) with annotated training samples could facilitate the process by detecting and counting fluorescent-labeled cells from uorescence photomicrographs. However, datasets containing segmented annotations of colocalized cells are generally not available, and creating a new dataset with delineated masks is label-intensive. Also, the colocalization coefficient is often not used as a component during training with the CNN model. Yet, it may aid with localizing and detecting objects during training and testing. In this work, we propose to address these issues by using a quantification coefficient for co-localization called Manders overlapping coefficient (MOC)¹ as a single-layer branch in a CNN. Fully convolutional one-state (FCOS)² with a Resnet101 backbone served as the network to evaluate the effectiveness of the novel branch to assist with bounding box prediction. Training data were sourced from lab curated uorescence images of neurons from the rat hippocampus, piriform cortex, somatosensory cortex, and amygdala. Results suggest that using modified FCOS with MOC outperformed the original FCOS model for accuracy in detecting uorescence signals by 1.1% in mean average precision (mAP). The model could be downloaded from https://github.com/Alphafrey946/Colocalization-MOC.

Registration of retinal images is an important technique for facilitating the diagnosis and treatment of many eye diseases. Recent studies have shown that deep learning methods can be used for image registration, which is usually faster than conventional registration methods. However, it is not trivial to obtain ground truth for supervised methods and popular unsupervised methods perform not well for retinal images. Therefore, we present a weakly-supervised learning method for affine registration of fundus image. The framework consists of multiple steps, rigid registration, overlap calculation and affine registration. In addition, we introduce a keypoint matching loss to replace common similarity metrics loss used in unsupervised methods. On a fundus image dataset related to multiple eye diseases, our framework can achieve more accurate registration results than that of state-of-the-art deep learning approaches.

In Computed Tomography (CT), problems like low to high dose image conversion, denoising and super-resolution using deep regression network have gained a lot of importance. This led to the study of multiple denoising approaches with these networks, as non-linear characteristics of the network often changes the noise pattern. Noise2Noise (N2N) and similar studies have shown the impact of these networks, where one can use noisy image pairs and exploit the uncorrelated nature of noise to generate denoised images. In this paper, we study the behaviour of regression network for a domain translation between one energy image to another energy and its impact on noise. Inter-kVp translation leads to change in attenuation values in CT image. A design of experiments is set up using CATSim phantom with different tissue types at varying levels of density and proportions of materials, for heart, soft tissue, fat, calcification and bone. The intent is to understand the impact of inter and intra-tissue dynamic ranges, as this network learns intensity translation of the image between 2 kVp levels and noise characteristics. The results demonstrate ability of regression network to change intensty values from low kVp image to high kVp image. It also shows impact on noise level in a tissue is proportional to (a) intra-tissue variability and (b) the desired mean shift of the tissue between two energy levels. Simultaneously, there is a marked change in the artifacts and resultant image quality, as expected, through the learning method.

This paper newly proposes a segmentation method of infected area for COVID-19 (Coronavirus Disease 2019) infected lung clinical CT volumes. COVID-19 spread globally from 2019 to 2020, causing the world to face a globally health crisis. It is desired to estimate severity of COVID-19, based on observing the infected area segmented from clinical computed tomography (CT) volume of COVID-19 patients. Given the lung field from a COVID-19 lung clinical CT volume as input, we desire an automated approach that could perform segmentation of infected area. Since labeling infected area for supervised segmentation needs a lot of labor, we propose a segmentation method without labeling of infected area. Our method refers to a baseline method utilizing representation learning and clustering. However, the baseline method is likely to segment anatomical structures with high H.U. (Houns field) intensity such as blood vessel into infected area. Aiming to solve this problem, we propose a novel pre-processing method that could transform high intensity anatomical structures into low intensity structures. This pre-processing method avoids high intensity anatomical structures to be mis-segmented into infected area. Given the lung field extracted from a CT volume, our method segment the lung field into normal tissue, ground GGO (ground glass opacity), and consolidation. Our method consists of three steps: 1) pulmonary blood vessel segmentation, 2) image inpainting of pulmonary blood vessel based on blood vessel segmentation result, and 3) segmentation of infected area. Compared to the baseline method, experimental results showed that our method contributes to the segmentation accuracy, especially on tubular structures such as blood vessels. Our method improved normalized mutual information score from 0.280 (the baseline method) to 0.394.

This work proposes a novel method for automatic detection of COVID-19 infection in CT images. COVID-19 has been a devastating infectious disease that ravaged the global community for last several months. Even though there are ongoing efforts to rapidly make all COVID-19 information and resources freely available to global community, availability of large amount of labeled data for disease diagnosis, prognosis and prediction is still a challenging task. Many of the recently published studies on detection of COVID-19 patients using deep learning methods do not have adequate dataset for robust training, validation, and testing. Therefore, this work develops a computational image analysis method that uses small amount of COVID-19 CT images for infection detection in CT images. The proposed method generates lung mask by segmenting the area of lung, after removing isolated high intensity areas from the surrounding ribs in the input CT image by employing an optimal segmentation algorithm and morphological operations. Next, the proposed method employs the segmented lung mask to detect areas of infection in the lung. To distinguish between disease and non-disease areas in the lung, the variance of the gray-level of each region is computed and used as feature to detect region of disease which are distinguished by low variance compare to Non-COVID-19 regions. The performance of proposed scheme is evaluated on a dataset of 72 CT-images (36 Non-Covid-19 images and 36 Covid-19 images) with ground-truth generated by radiologists. The accuracy of detection for Covid-19 images is 91.7% and that for Non-Covid-19 images is 91.7%, respectively. The proposed algorithm cannot distinguish between COVID-19 disease and other disorder in lung of patient.

Adaptive radiation therapy uses deformable image registration to warp the dose from the planning computed tomography (CT) image to the daily cone-beam CT (CBCT) image acquired using the onboard volumetric imaging. Image quality of this CBCT image is usually inferior due to poor soft-tissue contrast of organs such as the prostate, causing registration algorithms to underperform in terms of accuracy. To alleviate this problem, we develop a hybrid image-similarity cost function that incorporates a point-to-distance map (PD) metric as one of its components. Given a pair of segmented images, structures on the fixed image are represented as sets of points while structures on the moving image are described as distance maps. The total distance of all fixed points to their associated boundaries on the moving image constitutes the PD metric, which is combined with the more traditional intensity similarity metric between the fixed and moving images. In this work, we use cubic B-splines as the registration transform. Our approach is validated using the pelvic reference dataset wherein the prostate, bladder, and rectum are manually contoured from the CT and CBCT images by a medical expert to obtain the segmented fixed and moving images. Accuracy of the deformable registration is quantified using the Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (95% HD), with and without the PD metric. Results demonstrate much improved overlap between the fixed and warped contours once the PD metric is applied. Moreover, the computational overhead associated with adding the PD metric is minimal.

Chronic obstructive pulmonary disease (COPD) is a common inflammatory disease characterized by restricted lung airflow affecting over 300 million people worldwide. Quantitative computed tomography (CT) has become a benchmark for large multi-center pulmonary research studies for assessment of airway and parenchymal physiology and function towards understanding the occurrence and progression of the disease. Airway tree segmentation is a precursor for such approaches; but current industry-standard methods require manual post-segmentation correction to remove leakages and add missing airway branches. Recently, deep learning (DL) methods have gained popularity in medical image segmentation and outperformed traditional image processing methods due to their data-driven optimization schemes of multi-layered and multi-scale features. Generalizability of DL methods is a lingering concern and essential in multi-site CT-based pulmonary studies due to varying CT imaging settings at different sites. In this paper, we examine the generalizability of a recently developed fully automated DL-based airway segmentation method using low-dose chest CT images from the NELSON lung cancer screening study. The DL method was trained using high-dose chest CT scans from the Iowa cohort of COPDGene study at baseline visits and applied on blinded low-dose images. Results show the recent DL-based method is generalizable to blinded low-dose chest CT imaging, and it achieves branch-level accuracies of 100, 99.6, and 96.0% at segmental, sub-segmental, and sub-sub-segmental branches along the five clinically significant bronchial paths (RB1, RB4, RB10, LB1, and LB10).

Deep Learning outperforms prior art in medical imaging tasks. It has been applied to segmentation of Magnetic Resonance Imaging (MRI) scans, where consecutive slices capture relevant body structures for visualization and diagnosis of medical condition. In this work we investigate experimentally the factors that improve segmentation performance of MRI sequences of abdominal organs, including network architecture, pre-training, data augmentation and improvements to loss function. After comparing segmentation network architectures, we choose the best performing one and experimented improvements (data augmentation, training choices). Finally, metrics are fundamental and IoU of each organ in particular, therefore we change loss function to IoU and evaluate the resulting quality. We show that DeepLabV3 is better than competitors by 20 percentage points (pp) or more (depending on the competitor), data augmentation and further enhancements improve performance of DeepLabV3 by 12 percentage points (pp) in average, and that loss function improves performance by up to 13pp as well. Finally, we discuss challenges and further work.

Difficulty in validating accuracy remains a substantial setback in the field of surface-based cortical thickness (CT) measurement due to the lack of experimental validation against ground truth. Although methods have been developed to create synthetic datasets for this purpose, none provide a robust mechanism for measuring exact thickness changes with surface-based approaches. This work presents a registration-based technique for inducing synthetic cortical atrophy to create a longitudinal, ground truth dataset specifically designed for ac- curacy validation of surface-based CT measurements. Across the entire brain, we show our method can induce up to between 0.6 and 2.6 mm of localized cortical atrophy in a given gyrus depending on the region's original thickness. By calculating the image deformation to induce this atrophy at 400% of the original resolution in each direction, we can induce a sub-voxel resolution amount of atrophy while minimizing partial volume effects. We also show that our method can be extended beyond application to CT measurements for the accuracy validation of longitudinal cortical segmentation and surface reconstruction pipelines when measuring accuracy against cortical landmarks. Importantly, our method relies exclusively on publicly available software and datasets.