This PDF file contains the front matter associated with SPIE Proceedings Volume 11315, including the Title Page, Copyright information, Table of Contents, Author and Conference Committee lists.

C-Arm positioning for interventional spine procedures can often be associated with a steep learning curve. The current training standards involve using real X-rays on cadavers or via apprenticeship-based programs. To help limit excess radiation exposure, several radiation-free training systems have been proposed in the literature but there lacks a hands-on, cost-effective simulator that does not require access to a physical C-Arm. In order to expand the accessibility of radiation-free C-Arm training, we have developed a 10:1 scaled down C-Arm simulator using 3D-printed parts and wireless accelerometers for tracking. We generated Digitally Reconstructed Radiographs (DRRs) in real-time using a 1-dimensional transfer function operating on a ray-traced projection of a patient CT scan. To evaluate the efficacy of the system as a training tool, we conducted a user study in which anesthesiology and orthopedic residents were evaluated on the accuracy of their C-Arm placement for three standard views used in spinal injection procedures. Both the experimental group and control group were given the same evaluation task with the experimental group receiving 5 minutes of training on the system using real-time DRRs and a standardized two page curriculum on proper image acquisition. The experimental group achieved an angular error of 4.76±1.66° which was lower than the control group at 6.88±3.67° and the overall feedback of the system was positive based on a Likert scale questionnaire filled out by each participant. The results indicate that our system has high potential for improving C-Arm placement in interventional spine procedures and we plan to conduct a follow-up study to evaluate the long-term training capabilities of the simulator.

Medical augmented reality has been actively studied for decades and many methods have been proposed to revolutionize clinical procedures. One example is the camera augmented mobile C-arm (CAMC), which provides a real-time video augmentation onto medical images by rigidly mounting and calibrating a camera to the imaging device. Since then, several CAMC variations have been suggested by calibrating 2D/3D cameras, trackers, and more recently a Microsoft HoloLens to the C-arm. Different calibration methods have been applied to establish the correspondence between the rigidly attached sensor and the imaging device. A crucial step for these methods is the acquisition of X-Ray images or 3D reconstruction volumes; therefore, requiring the emission of ionizing radiation. In this work, we analyze the mechanical motion of the device and propose an alternative method to calibrate sensors to the C-arm without emitting any radiation. Given a sensor is rigidly attached to the device, we introduce an extended pivot calibration concept to compute the fixed translation from the sensor to the C-arm rotation center. The fixed relationship between the sensor and rotation center can be formulated as a pivot calibration problem with the pivot point moving on a locus. Our method exploits the rigid C-arm motion describing a Torus surface to solve this calibration problem. We explain the geometry of the C-arm motion and its relation to the attached sensor, propose a calibration algorithm and show its robustness against noise, as well as trajectory and observed pose density by computer simulations. We discuss this geometric-based formulation and its potential extensions to different C-arm applications.

Currently, uoroscopy and conventional digital subtraction angiography are used for imaging guidance in endovascular aortic repair (EVAR) procedures. Drawbacks of these image modalities are X-ray exposure, the usage of contrast agents and the lack of depth information. To overcome these disadvantages, a catheter prototype containing a multicore fiber with fiber Bragg gratings for shape sensing and three electromagnetic (EM) sensors for locating the shape was built in this study. Furthermore, a model for processing the input data from the tracking systems to obtain the located 3D shape of the first 38 cm of the catheter was introduced: A spatial calibration between the optical fiber and each EM sensor was made in a calibration step and used to obtain the located shape of the catheter in subsequent experiments. The evaluation of our shape localization method with the catheter prototype in different shapes resulted in average errors from 0.99 to 2.29 mm and maximum errors from 1.73 to 2.99 mm. The experiments showed that an accurate shape localization with a multicore fiber and three EM sensors is possible, and that this catheter guidance is promising for EVAR procedures. Future work will be focused on the development of catheter guidance based on shape sensing with a multicore fiber, and the orientation and position of less than three EM sensors.

X-ray fluoroscopy is commonly used to guide needles during transjugular intrahepatic portosystemic shunt (TIPS) procedures. Respiratory motion and the 2D nature of the x-ray projections, however, make it difficult to accurately guide the needle from a hepatic vein, through the liver parenchyma, and into the portal vein, which is not visible on x-ray images in the absence of continuous contrast-enhancement. Due to these challenges, multiple needle passes are often required, which increase the risk for perforation of the liver capsule and hemorrhage. To overcome these challenges, we propose a motion-compensated 3D needle guidance system, which generates a respiratory motion model of the portal venous system using a 3D DSA and a contrast enhanced 2D x-ray sequence acquired under free breathing conditions. The respiratory motion is tracked during the needle pass based on brightness variations above and below the diaphragm in the 2D images, which allows for creation of a motion-compensated surface model of the target vasculature. Additionally, a 3D needle reconstruction algorithm from two 2D x-ray images is presented, which allows for motion-compensated 3D device imaging. A preliminary pig study was performed to evaluate the feasibility of the proposed techniques. The biplane needle reconstruction was compared to conventional cone beam CT acquisitions, where a root mean squared distance of 0.98 mm and a tip localization error of 1.22 mm were measured. The maximum error of the estimated vascular motion per frame in the two pig studies was 0.63 mm and 1.63 mm respectively. If successfully translated to clinical TIPS procedures, the proposed needle guidance could result in fewer unnecessary needle passes and therefore shorter procedure times and lower risk to the patient.

Electromagnetic image guidance systems have emerged as more secure methods to improve the performance of several catheter-based minimally invasive surgical procedures. Small sensors are incorporated within catheters and guidewires in order to track and guide in real-time their position and orientation with a reduced intra- procedural radiation exposure and contrast agent injections. One of the major limits of these systems is related to the unsuitable sensorization strategy for the J-tip guidewires, due to the structural constraints of the sensor coils available on the market. In this work we present preliminary results on a sensors bending test in static conditions to assess whether and when the precision of the sensor remains unchanged and/or deteriorates. In the worst case, the highest standard deviation is less than 0:10 mm.

Cochlear implants (CIs) are surgically implanted neural prosthetic devices used to treat severe-to-profound hearing loss. Our group has developed Image-Guided Cochlear Implant Programming (IGCIP) techniques to assist audiologists with the configuration of the implanted CI electrodes. CI programming is sensitive to the spatial relationship between the electrodes and intra cochlear anatomy (ICA) structures. We have developed algorithms that permit determining the position of the electrodes relative to the ICA structure using pre- and post-implantation CT image pairs. However, these do not extend to CI recipients for whom pre-implantation CT (Pre-CT) images are not available because post-implantation CT (Post-CT) images are affected by strong artifacts introduced by the metallic implant. Recently, we proposed an approach that uses conditional generative adversarial nets (cGANs) to synthesize Pre-CT images from Post-CT images. This permits to use algorithms designed to segment Pre-CT images even when these are not available. We have shown that it substantially and significantly improves the results obtained with our previous published methods that segment post- CT images directly. Here we evaluate the effect of this new approach on the final output of our IGCIP techniques, which is the configuration of the CI electrodes, by comparing configurations of the CI electrodes obtained using the real and the synthetic Pre-CT images. In 22/87 cases synthetic image lead to the same results as the real images. Because more than one configuration may lead to equivalent neural stimulation patterns, visual assessment of solutions is required to compare those that differ. This study is ongoing.

PURPOSE: It is difficult to simulate realistic ultrasound images due to the complexity of acoustic artifacts that contribute to a real ultrasound image. We propose to evaluate the realism of ultrasound images simulated using a generative adversarial network. METHODS: To achieve our goal, kidney ultrasounds were collected, and relevant anatomy was segmented to create anatomical label-maps using 3D Slicer. Adversarial networks were trained to generate ultrasound images from these labelmaps. Finally, a two-part survey of 4 participants with sonography experience was conducted to assess the realism of the generated images. The first part of the survey consisted of 50 kidney ultrasound images; half of which were real while the other half were simulated. Participants were asked to label each of the 50 ultrasound images as either real or simulated. In the second part of the survey, the participants were presented with ten simulated images not included in the first part of the survey and asked to evaluate the realism of the images. RESULTS: The average number of correctly identified images was 28 of 50 (56%). On a scale of 1-5, where 5 is indistinguishable from real US, the generated images received an average score of 3.75 for realistic anatomy and 4.0 for realistic ultrasound effects. CONCLUSIONS: We evaluated the realism of kidney ultrasound images generated using adversarial networks. Generative adversarial networks appear to be a promising method of simulating realistic ultrasound images from crosssectional anatomical label-maps.

We present the application of deep multi-class classifiers for registration of the pre-radiation image (CBCT) to the treatment planning image (planCT) in Radiation Therapy (RT). We train a multi-class classifier on different classes of displacement between 3D patches of images and use it for registration. As the initial displacement between images might be large, we train multiple classifiers for different resolutions of the data to capture larger displacements in coarser resolutions. We show that having only a few patients, the deep multi-class classifiers enable an accurate and fast rigid registration for CBCT to planCT even with significantly different fields of view. Our work lays the foundation for deformable image registration and prediction of registration uncertainty which can be utilized for adaptive RT.

4D thoracic images constructed from free-breathing 2D slice acquisitions based on dynamic magnetic resonance imaging (dMRI) provide clinicians the capability of examining the dynamic function of the left and right lungs, left and right hemidiaphragms, and left and right chest wall separately for thoracic insufficiency syndrome (TIS) treatment [1]. There are two shortcomings of the existing 4D construction methods [2]: a) the respiratory phase corresponding to end expiration (EE) and end inspiration (EI) need to be manually identified in the dMRI sequence; b) abnormal breathing signals due to nontidal breathing cannot be detected automatically which affects the construction process. Since the typical 2D dynamic MRI acquisition contains ~3000 slices per patient, handling these tasks manually is very labor intensive. In this study, we propose a deep-learning-based framework for addressing both problems via convolutional neural networks (CNNs) [3] and Long Short-Term Memory (LSTM) [4] models. A CNN is used to extract the motion characteristics from the respiratory dMRI sequences to automatically identify contiguous sequences of slices representing exhalation and inhalation processes. EE and EI annotations are subsequently completed by comparing the changes in the direction of motion of the diaphragm. A LSTM network is used for detecting abnormal respiratory signals by exploiting the nonuniform motion feature sequence of abnormal breathing motions. Experimental results show the mean error of labeling EE and EI is ~0.3 dMRI time point unit (much less than one time point). The accuracy of abnormal cycle detection reaches 80.0%. The proposed approach achieves results highly comparable to manual labeling in accuracy but with close to full automation of the whole process. The framework proposed here can be readily adapted to other modalities and dynamic imaging applications.

Purpose: Complex, involuntary, non-periodic, deformable motion presents a confounding factor to cone-beam CT (CBCT) image quality due to long (>10 s) scan times. We report and demonstrate an image-based deformable motion compensation method for CBCT, including phantom, cadaver, and animal studies as precursors to clinical studies. Methods: The method corrects deformable motion in CBCT scan data by solving for a motion vector field (MVF) that optimizes a sharpness criterion in the 3D image (viz., gradient entropy). MVFs are estimated by interpolating M locally rigid motion trajectories across N temporal nodes and are incorporated in a modified 3D filtered backprojection approach. The method was evaluated in a cervical spine phantom under flexion, and a cadaver undergoing variable magnitude of complex motion while imaged on a mobile C-arm (Cios Spin 3D, Siemens Healthineers, Forchheim, Germany). Further assessment was performed on a preclinical animal study using a clinical fixed-room C-arm (Artis Zee, Siemens Healthineers, Forchheim, Germany). Results: In phantom studies, the algorithm resolved visibility of cervical vertebrae under situations of strong flexion, reducing the root-mean-square error by 60% when compared to a motion-free reference. Reduced motion artifacts (blurring, streaks, and loss of soft-tissue edges) were evident in abdominal CBCT of a cadaver imaged during small, medium, and large motion-induced deformation. The animal study demonstrated reduction of streaks from complex motion of bowel gas during the scan. Conclusion: Overall, the studies demonstrate the robustness of the algorithm to a broad range of motion amplitudes, frequencies, data sources (i.e., mobile or fixed-room C-arms) and other confounding factors in real (not simulated) experimental data (e.g., truncation and scatter). These preclinical studies successfully demonstrate reduction of motion artifacts in CBCT and support translation of the method to clinical studies in interventional body radiology.

We present a co-robotic ultrasound imaging system that tracks lesions undergoing physiological motions, e.g., breathing, using 2D B-mode images. The approach embeds the in-plane and out-of-plane transformation estimation in a proportional joint velocity controller to minimize the 6-degree-of-freedom (DoF) transformation error. Specifically, we propose a new method to estimate the out-of-plane translation using a convolutional neural network based on speckle decorrelation. The network is trained on anatomically featureless gray-scale B-mode images and is generalized to different tissue phantoms. The tracking algorithm is validated in simulation with mimicked respiratory motions, which demonstrates the feasibility of stabilizing biopsy through ultrasound guidance.

During total hip arthroplasty (THA) procedures, orthopedic surgeons rely on visual and tactile cues to minimize the risk of a limb length discrepancy (LLD) arising from improper implant placement and selection. This paper outlines a novel application of a commercial infrared (IR) motion controller, Leap Motion, to function as an optical tracker of LLD. Additionally, a custom benchtop testing model was used to evaluate accuracy and precision of this commercial IR motion controller against a custom-built IR stereo vision camera. This paper includes a discussion of the algorithms and infrared technology used to image and localize points. The novel design discussed in this paper has clinical applications for a custom-built IR stereo vision setup that will be positioned above the incision site during a THA procedure. Accompanying this solution concept is a fiducial with IR-visible landmarks for quick and accurate point selection and a software algorithm that measures changes between pre- and post-operative leg length.

The most common chronic inflammatory and chronic joint diseases in Canada are rheumatoid arthritis (RA) and osteoarthritis (OA) respectively. The current methods for monitoring the development of these diseases and their response to treatment involve acquiring x-ray and magnetic resonance imaging (MRI) images and comparing the results to the patient’s symptoms during examination. However, x-ray imaging is associated with difficulties when trying to interpret 3D anatomy, and MRI is associated with high operating costs, long waitlists, long scan times, and is inaccessible to mobility-impaired patients. Our solution to these limitations is the use of three-dimensional ultrasound (3D US) imaging for providing bedside monitoring of RA and OA progression and their response to treatment. This project validates two 3D US acquisition devices: tilt acquisition and linear acquisition scanning methods. The linear and volumetric measurement capabilities were validated through scanning phantoms and segmenting resulting images at repeated time points. A proof-of-concept volunteer scan was conducted to compare the capabilities of 3D US against MRI for measuring articular cartilage volumes. Results indicated that the linear measurement errors for both the tilt and linear scanners were <5% of the known phantom dimensions. The volumetric measurement errors were <5% for the linear scanner and >10% (29.84%) for the tilt scanner. The percent difference between the volumes of the articular cartilage measured using 3D US and MRI of a healthy volunteer’s knee was 6.46%. The linear scanner is therefore better suited for clinical scanning than the tilt scanner due to its smaller linear and volumetric errors.

Purpose. Fracture reduction is a challenging part of orthopaedic pelvic trauma procedures, resulting in poor long-term prognosis if reduction does not accurately restore natural morphology. Manual preoperative planning is performed to obtain target transformations of target bones – a process that is challenging and time-consuming even to experts within the rapid workflow of emergent care and fluoroscopically guided surgery. We report a method for fracture reduction planning using a novel image-based registration framework. Method. An objective function is designed to simultaneously register multi-body bone fragments that are preoperatively segmented via a graph-cut method to a pelvic statistical shape model (SSM) with inter-body collision constraints. An alternating optimization strategy switches between fragments alignment and SSM adaptation to solve for the fragment transformations for fracture reduction planning. The method was examined in a leave-one-out study performed over a pelvic atlas with 40 members with two-body and three-body fractures simulated in the left innominate bone with displacements ranging 0–20 mm and 0°–15°. Result. Experiments showed the feasibility of the registration method in both two-body and three-body fracture cases. The segmentations achieved Dice coefficient of median 0.94 (0.01 interquartile range [IQR]) and root mean square error (RMSE) of 2.93 mm (0.56 mm IQR). In two-body fracture cases, fracture reduction planning yielded 3.8 mm (1.6 mm IQR) translational and 2.9° (1.8° IQR) rotational error. Conclusion. The method demonstrated accurate fracture reduction planning within 5 mm and shows promise for future generalization to more complicated fracture cases. The algorithm provides a novel means of planning from preoperative CT images that are already acquired in standard workflow.

Pelvic trauma surgical procedures rely heavily on guidance with 2D fluoroscopy views for navigation in complex bone corridors. This “fluoro-hunting” paradigm results in extended radiation exposure and possible suboptimal guidewire placement from limited visualization of the fractures site with overlapped anatomy in 2D fluoroscopy. A novel computer visionbased navigation system for freehand guidewire insertion is proposed. The navigation framework is compatible with the rapid workflow in trauma surgery and bridges the gap between intraoperative fluoroscopy and preoperative CT images. The system uses a drill-mounted camera to detect and track poses of simple multimodality (optical/radiographic) markers for registration of the drill axis to fluoroscopy and, in turn, to CT. Surgical navigation is achieved with real-time display of the drill axis position on fluoroscopy views and, optionally, in 3D on the preoperative CT. The camera was corrected for lens distortion effects and calibrated for 3D pose estimation. Custom marker jigs were constructed to calibrate the drill axis and tooltip with respect to the camera frame. A testing platform for evaluation of the navigation system was developed, including a robotic arm for precise, repeatable, placement of the drill. Experiments were conducted for hand-eye calibration between the drill-mounted camera and the robot using the Park and Martin solver. Experiments using checkerboard calibration demonstrated subpixel accuracy [−0.01 ± 0.23 px] for camera distortion correction. The drill axis was calibrated using a cylindrical model and demonstrated sub-mm accuracy [0.14 ± 0.70 mm] and sub-degree angular deviation.

In this paper we introduce a novel and enabling MRI-compatible needle guidance toolkit intended to streamline arthrography procedures, eliminating the need for ionizing radiation exposure during this diagnostic procedure. We developed a flexible 2D grid template with unique patterns in which each point on the grid can be uniquely represented and denoted with respect to surrounding patterns. This MRI-visible non-repeating grid template sits on top of the patient’s skin in the region of interest and allows the radiologist to visualize the skin surface in the MR images and correlates each point in MR image with the corresponding point on the grid. In this manner, the radiologist can intuitively find the entry point on the skin according to their plan. An MRI-compatible handheld positioning device consisting of a needle guide, two baseline bubble inclinometers, and an MRI-compatible stabilizer arm allows the radiologist to adjust the orientation of the needle in two directions. The radiologist selects an entry point on MR images, identifies a grid location through which the needle would be projected to pass on the image, and then reproduces this needle position and angulation using the MRI-compatible handheld device and the physical grid. To evaluate the accuracy of needle targeting with the MRI-compatible needle guidance toolkit, we used the kit to target 10 locations in a phantom in a Philips Achieva 1.5T MRI. The average targeting error was 2.2±0.7 mm. Average targeting procedure time was around 20 minutes for each target.

3D ultrasound imaging can be used in the context of robotic needle steering to reach a physical target with a flexible, steerable needle. During the insertion, the tissue may be deformed by the inserted needle, patient breathing or external force application. It may therefore be necessary to track intra-operatively the displacement of the target. Most ultrasound based needle steering works concentrate on 2D ultrasound probes, which do not allow to simultaneously track both the target and the needle during 3D needle steering. Physical target tracking in 3D ultrasound-guided needle steering is seldom carried out, and may require computational power that is precious for intra-operative needle steering. This paper proposes a new approach for computationally inexpensive and precise tracking of a moving target in 3D Bmode ultrasound volumes. It is based on the interconnection of intensity-based tracking and motion estimation algorithms. The intensity-based tracking consists in a 3D extension of the Diamond Shape block matching algorithm, used here for the first time in 3D ultrasound volumes for tissue tracking. The motion estimation is done by linear Kalman filtering. It predicts the next target position and ensures faster and more robust convergence of the Diamond Shape block matching algorithm. An experimental validation on ex-vivo tissue is proposed with promising tracking precision (estimated average error of 0.3mm) while significantly lowering the computational cost when compared to classical block matching based tracking.

Image guidance aids neurosurgeons in making critical clinical decisions of safe maximal resection of diseased tissue. The brain however undergoes significant non-linear structural deformation on account of dura opening and tumor resection. Deformable registration of pre-operative ultrasound to intra-operative ultrasound may be used in mapping of pre-operative planning MRI to intraoperative ultrasound. Such mapping may aid in determining tumor resection margins during surgery. In this work, brain structures visible in pre- and intra-operative 3D ultrasound were used for automatic deformable registration. A Gaussian mixture model was used to automatically segment structures of interest in pre- and intra-operative ultrasound and patch-based normalized cross-correlation was used to establish correspondences between segmented structures. An affine registration based on correspondences was followed by B-spline based deformable registration to register pre- and intra-operative ultrasound. Manually labelled landmarks in pre- and intra-operative ultrasound were used to quantify the mean target registration error. We achieve a mean target registration error of 1.43±0.8 mm when validated with 17 pre- and intra-operative ultrasound image volumes of a public dataset.

High-dose-rate interstitial gynecologic brachytherapy requires multiple needles to be inserted into the tumor and surrounding area, avoiding nearby healthy organs-at-risk (OARs), including the bladder and rectum. We propose the use of a 360° three-dimensional (3D) transvaginal ultrasound (TVUS) guidance system for visualization of needles and report on the implementation of two automatic needle segmentation algorithms to aid the localization of needles intraoperatively. Two-dimensional (2D) needle segmentation, allowing for immediate adjustments to needle trajectories to mitigate needle deflection and avoid OARs, was implemented in near real-time using a method based on a convolutional neural network with a U-Net architecture trained on a dataset of 2D ultrasound images from multiple applications with needle-like structures. In 18 unseen TVUS images, the median position difference [95% confidence interval] was 0.27 [0.20, 0.68] mm and mean angular difference was 0.50 [0.27, 1.16]° between manually and algorithmically segmented needles. Automatic needle segmentation was performed in 3D TVUS images using an algorithm leveraging the randomized 3D Hough transform. All needles were accurately localized in a proof-of-concept image with a median position difference of 0.79 [0.62, 0.93] mm and median angular difference of 0.46 [0.31, 0.62]°, when compared to manual segmentations. Further investigation into the robustness of the algorithm to complex cases containing large shadowing, air, or reverberation artefacts is ongoing. Intraoperative automatic needle segmentation in interstitial gynecologic brachytherapy has the potential to improve implant quality and provides the potential for 3D ultrasound to be used for treatment planning, eliminating the requirement for post-insertion CT scans.

Purpose: Repetitive Transcranial Magnetic Stimulation (rTMS) is an important treatment option for medication resistant depression. It uses an electromagnetic coil that needs to be positioned accurately at a specific location and angle next to the head such that specific brain areas are stimulated. Existing image-guided neuronavigation systems allow accurate targeting but add cost, training and setup times, preventing their wide-spread use in the clinic. Mixed-reality neuronavigation can help mitigate these issues and thereby enable more widespread use of image-based neuronavigation by providing a much more intuitive and streamlined visualization of the target. A mixed-reality neuronavigation system requires two core functionalities: 1) tracking of the patient's head and 2) visualization of targeting-related information. Here we focus on the head tracking functionality and compare three different head tracking methods for a mixed-reality neuronavigation system. Methods: We integrated three head tracking methods into the mixed reality neuronavigation framework and measured their accuracy. Specifically, we experimented with (a) marker-based tracking with a mixed reality headset (optical see-through head-mounted display (OST-HMD)) camera, (b) marker-based tracking with a world-anchored camera and (c) markerless RGB-depth (RGB-D) tracking with a world-anchored camera. To measure the accuracy of each approach, we measured the distance between real-world and virtual target points on a mannequin head. Results: The mean tracking error for the initial head pose and the head rotated by 10° and 30° for the three methods respectively was: (a) 3.54±1.10 mm, 3.79±1.78 mm and 4.08±1.88 mm, (b) 3.97±1.41 mm, 6.01±2.51 mm and 6.84±3.48 mm, (c) 3.16±2.26 mm, 4.46±2.30 mm and 5.83±3.70 mm. Conclusion: For the initial head pose, all three methods achieved the required accuracy of < 5 mm for TMS treatment. For smaller head rotations of 10°, only the marker-based (a) and markerless method (c) delivered sufficient accuracy for TMS treatment. For larger head rotations of 30°, only the marker-based method (a) achieved sufficient accuracy. While the markerless method (c) did not provide sufficient accuracy for TMS at the larger head rotations, it offers significant advantages such as occlusion-handling and stability and could potentially meet the accuracy requirements with further methodological refinements.

Deep brain stimulation (DBS) is an interventional treatment for Parkinson’s disease in which electrodes are placed into specific locations in the basal ganglia to alleviate symptoms such as tremor and dyskinesia. Due to the small size and low contrast of specific targets of interest, such as the subthalamic nucleus (STN), localisation of these structures from pre-operative MRI is of great value. These localisation approaches are often atlas-based, using registration algorithms to align patient images with a prior annotated atlas. However, these methods require a large amount of time, especially for correctly estimating deformation fields, or are prone to error for smaller structures such as the STN. This paper investigates two deep learning frameworks for the localisation of the stn from T1- and T2-weighted MRI using convolutional neural networks which estimate its centroid. These methods are compared against an atlas-based segmentation using the ParkMedAtlis v3 atlas, showing an improvement in localisation error in the range of ≈0.5-1.3 mm with a reduction of orders of magnitude of computation time. This method of STN localisation will allow us in future to automatically identify the STN for DBS surgical planning as well as define a greatly reduced region-of-interest for more accurate segmentation of the STN.

The success of brain surgery depends on the exact and effective treatment of pathological alterations while preserving functional tissue and essential vessels. Varying intraoperative imaging methods have been developed to achieve this goal. For cortical perfusion imaging, application of time-resolved thermography in combination with an intravenously applied cold bolus became a promising approach. This work provides a regularized semiparametric regression framework that detects the cold signal response function while compensating arbitrary background signals using penalized B-Splines. This enables weak thermal signal detection to give information about blood flow even in small vessels, applicable for intraoperative cortical blood flow mapping.

In medically intractable epilepsy cases, subdural electrodes are often implanted for localization of epileptogenic foci. Electrode array placement is typically planned prior to surgery to ensure coverage of potential pathological tissues. Image guidance ensures accurate and effective placement of electrodes but the efficacy can be degraded by intraoperative brain shift. We have developed a guidance system to plan electrode placement interactively and execute the plan through image guidance, and compensate for brain shift due to dural opening by updating electrode positions to maintain their relative positions with respect to the cortical surface. One patient case was evaluated retrospectively to assess the performance of the guidance system. Preoperative magnetic resonance images were segmented to create a 3D model of the brain. Users selected a pre-defined electrode grid template and interactively placed it onto the segmented brain. The grid was deformed to wrap over the cortical surface based on local curvature. Intraoperative stereovision data was registered with preoperative MR to measure cortical shift, and the average shift across the exposed cortical surface was used to update electrode positions. A DICOM image stack can be created and transferred to the navigation system to create a model to represent the electrode array, and the model can be visualized on the operating microscope for intraoperative guidance. Distance change from each electrode to three landmarks on the cortical surface was quantified to assess the accuracy of electrode updating. The overall workflow time was <5 min with an overall accuracy of 0.36 mm across three landmarks.

Accurate surgical placement of electrodes is essential to successful deep brain stimulation (DBS) for patients with neurodegenerative diseases such as Parkinson’s disease. However, the accuracy of pre-operative images used for surgical planning and guidance is often degraded by brain shift during surgery. To predict such intra-operative target deviation due to brain shift, we have developed a finite-element biomechanical model with the assimilation of intraoperative sparse data to compute a whole brain displacement field that updates preoperative images. Previously, modeling with the incorporation of surface sparse data achieved promising results at deep brain structures. However, access to surface data may be limited during a burr hole-based procedure where the size of exposed cortex is too small to acquire adequate intraoperative imaging data. In this paper, our biomechanical brain model was driven by deep brain sparse data that was extracted from lateral ventricles using a Demon’s algorithm and the simulation result was compared against the one resulted from modeling with surface data. Two patient cases were explored in this study where preoperative CT (preCT) and postoperative CT (postCT) were used for the simulation. In patient case one of large symmetrical brain shift, results show that model driven by deep brain sparse data reduced the target registration error(TRE) of preCT from 3.53 to 1.36 and from 1.79 to 1.17 mm at AC and PC, respectively, whereas results from modeling with surface data produced even lower TREs at 0.58 and 0.69mm correspondingly; However, in patient case two of large asymmetrical brain shift, modeling with deep brain sparse data yielded the lowest TRE of 0.68 from 1.73 mm. Results in this study suggest that both surface and deep brain sparse data are capable of reducing the TRE of preoperative images at deep brain landmarks. The success of modeling with the assimilation of deep brain sparse data alone shows the potential of implementing such method in the OR because sparse data at lateral ventricle can be acquired using ultrasound imaging.

PURPOSE: The iKnife is a new surgical tool designed to aid in tumor resection procedures by providing enriched chemical feedback about the tumor resection cavity from electrosurgical vapors. We build and compare machine learning classifiers that are capable of distinguishing primary cancer from surrounding tissue at different stages of tumor progression. In developing our classification framework, we implement feature reduction and recognition tools that will assist in the translation of xenograft studies to clinical application and compare these tools to standard linear methods that have been previously demonstrated. METHODS: Two cohorts (n=6 each) of 12 week old female immunocompromised (Rag2−/−;Il2rg−/−) mice were injected with the same human breast adenocarcinoma (MDA-MB-231) cell line. At 4 and 6 weeks after cell injection, mice in each cohort were respectively euthanized, followed by iKnife burns performed on tumors and tissues prior to sample collection for future studies. A feature reduction technique that uses a neural network is compared to traditional linear analysis. For each method, we fit a classifier to distinguish primary cancer from surrounding tissue. RESULTS: Both classifiers can distinguish primary cancer from metastasis and surrounding tissue. The classifier that uses a neural network achieves an accuracy of 96.8% and the classifier without the neural network achieves an accuracy of 96%. CONCLUSIONS: The performance of these classifiers indicate that this device has the potential to offer real-time, intraoperative classification of tissue. This technology may be used to assist in intraoperative margin detection and inform surgical decisions to offer a better standard of care for cancer patients.

In the Western world, nearly 1 in 14 men will be diagnosed with prostate cancer in their lifetimes. The gold standard for detection of PCa is histopathology analysis of biopsy cores taken using trans-rectal ultrasound (TRUS) guidance, a procedure that has a false negative rate of 30 − 45% and serious side effects. Multiparametric MRI (MP-MRI) is quickly becoming part of the standard of care to detect PCa lesions. According to recent multi-centre studies, it has the potential to decrease false positives for PCa detection and reduce the need for biopsy. At the same time, deep learning approaches for aiding radiologists in PCa diagnosis have been on the rise. Most of the solutions in the literature benefit from abundant high-quality data, which limits their translation to clinical settings. For PCa in particular, close to 83% of clinical MRI systems in Canada are 1.5 T, and many centres may not have the high throughput volume of patients required for building locally accurate machine learning models. In this paper, we present preliminary results from a deep learning framework built using publicly available 3.0 T MP-MRI data and re-purposed for 1.5 T clinical data. We achieve areas under the receiver operating curve of up to 0.76 and provide visualization of the most informative areas of the images for the deep models. Our proposed approach has the potential to allow local hospitals to use pre-built AI models fine-tuned for their own cases, taking advantage of externally available large data sets.

Because of the deformation of the brain during neurosurgery, intraoperative imaging can be used to visualize the actual location of the brain structures. These images are used for image-guided navigation as well as determining whether the resection is complete and localizing the remaining tumor tissue. Intraoperative ultrasound (iUS) is a convenient modality with short acquisition times. However, iUS images are difficult to interpret because of the noise and artifacts. In particular, tumor tissue is difficult to distinguish from healthy tissue and it is very difficult to delimit tumors in iUS images. In this paper, we propose an automatic method to segment low grade brain tumors in iUS images using a 2-D and 3-D U-Net. We trained the networks on three folds with twelve training cases and five test cases each. The obtained results are promising, with a median Dice score of 0.72. The volume differences between the estimated and ground truth segmentations were similar to the intra-rater volume differences. While these results are preliminary, they suggest that deep learning methods can be successfully applied to tumor segmentation in intraoperative images.

There are significant challenges in global healthcare delivery at the moment. Some countries have abundant services, but are stuck with a rather nimble and expensive system that focuses on incremental innovations. Other geographies are still in need of basic tools, infrastructure and require completely different, inexpensive, and with that more disruptive solutions to satisfy their healthcare needs. Next Generation Healthcare systems with a focus on prevention / early detection and pro-active therapy will employ exponential technologies (AI, Big Data, Blockchain, Sensor Technology, Synthetic Biology, Tissue Engineering, Robotics, 3D Printing, ...) that will surely lead to significant changes in the way we experience, think about, and deliver healthcare and in which a digitally empowered patient will play a more important role. In the coming years/decades we will experience a shift from the current SICK-CARE provision to real HEALTHCARE to a focus on personal HEALTH, supported by an integration of patient generated health data with other external diagnostic and therapeutic data components creating a digital health twin as a base for prevention and patient centered precision medicine. Education and training of Biomedical Engineers needs to be adjusted to these developments focussing on solving unmet clinical needs, which requires a solid understanding of current health problems (regional, global), future technologies, economic realities and global health markets. The paper will present and discuss some of these future global innovation needs and the subsequent need for a change in the biomedical engineering curriculum that needs to include learning- (creativity, critical thinking, collaboration) and life skills (flexibility, leadership, social), as well as basic economic and entrepreneurial training — all of which are currently not taught or emphasized.

Wearable augmented reality (AR) is an emerging technology with enormous potential for use in the medical field, from training and procedure simulations to image-guided surgery. Medical AR seeks to enable surgeons to see tissue segmentations in real time. With the objective of achieving real-time guidance, the emphasis on speed produces the need for a fast method for imaging and classification. Hyperspectral imaging (HSI) is a non-contact, optical imaging modality that rapidly acquires hundreds of images of tissue at different wavelengths, which can be used to generate spectral data of the tissue. Combining HSI information and machine-learning algorithms allows for effective tissue classification. In this paper, we constructed a brain tissue phantom with porcine blood, yellow-dyed gelatin, and colorless gelatin to represent blood vessels, tumor, and normal brain tissue, respectively. Using a segmentation algorithm, hundreds of hyperspectral images were compiled to classify each of the pixels. Three segmentation labels were generated from the data, each with a different type of tissue. Our system virtually superimposes the HSI channels and segmentation labels of a brain tumor phantom onto the real scene using the HoloLens AR headset. The user can manipulate and interact with the segmentation labels and HSI channels by repositioning, rotating, changing visibility, and switching between them. All actions can be performed through either hand or voice controls. This creates a convenient and multifaceted visualization of brain tissue in real time with minimal user restrictions. We demonstrate the feasibility of a fast and practical HIS-AR technique for potential use of image-guided brain surgery.

Purpose: This study aims to investigate the accuracy of a cross platform augmented reality (AR) system for percutaneous needle interventions irrespective of operator error. In particular, we study the effect of the relative position and orientation of the AR device and the marker, the location of the target, and the angle of needle on the overlay accuracy. Method: A needle guidance AR platform developed using Unity and Vuforia SDK platforms was used to display a planned needle trajectory for targets via mobile and wearable devices. To evaluate the system accuracy, a custom phantom embedded with metal fiducial markers and an adjustable needle guide was designed to mimic different relative position and orientation scenarios of the smart device and the marker. After segmenting images of CT-visible fiducial markers as well as different needle trajectories, error was defined by comparing them to the corresponding augmented target/needle trajectory projected by smartphone and smartglasses devices. Results: The augmentation error for targets and needle trajectories were reported as a function of marker position and orientation, as well as the location of the targets. Overall, the image overlay error for needle trajectory was 0.28±0.32° (Max = 0.856°) and 0.41±0.23° (Max = 0.805°) using the iPhone and HoloLens glasses, respectively. The overall image overlay error for targets was 1.75±0.59 mm for iPhone, and 1.74±0.86 mm for HoloLens. Conclusions: The image overlay error caused by different sources can be quantified for different AR devices.

Guided biopsy of soft tissue lesions can be challenging in the presence of sensitive organs or when the lesion itself is small. Computed tomography (CT) is the most frequently used modality to target soft tissue lesions. In order to aid physicians, small field of view (FOV) low dose non-contrast CT volumes are acquired prior to intervention while the patient is on the procedure table to localize the lesion and plan the best approach. However, patient motion between the end of the scan and the start of the biopsy procedure can make it difficult for a physician to translate the lesion location from the CT onto the patient body, especially for a deep-seated lesion. In addition, the needle should be managed well in three-dimensional trajectories in order to reach the lesion and avoid vital structures. This is especially challenging for less experienced interventionists. These usually result in multiple additional image acquisitions during the course of procedure to ensure accurate needle placement, especially when multiple core biopsies are required. In this work, we present an augmented reality (AR)-guided biopsy system and procedure for soft tissue and lung lesions and quantify the results using a phantom study. We found an average error of 0.75 cm from the center of the lesion when AR guidance was used, compared to an error of 1.52 cm from the center of the lesion during unguided biopsy for soft tissue lesions while upon testing the system on lung lesions, an average error of 0.62 cm from the center of the tumor while using AR guidance versus a 1.12 cm error while relying on unguided biopsies. The AR-guided system is able to improve the accuracy and could be useful in the clinical application.

Minimally Invasive Surgery (MIS) techniques have gained rapid popularity among surgeons since they offer significant clinical benefits including reduced recovery time and diminished post-operative adverse effects. However, conventional endoscopic systems output monocular video which compromises depth perception, spatial orientation and field of view. Suturing is one of the most complex tasks performed under these circumstances. Key components of this tasks are the interplay between needle holder and the surgical needle. Reliable 3D localization of needle and instruments in real time could be used to augment the scene with additional parameters that describe their quantitative geometric relation, e.g. the relation between the estimated needle plane and its rotation center and the instrument. This could contribute towards standardization and training of basic skills and operative techniques, enhance overall surgical performance, and reduce the risk of complications. The paper proposes an Augmented Reality environment with quantitative and qualitative visual representations to enhance laparoscopic training outcomes performed on a silicone pad. This is enabled by a multi-task supervised deep neural network which performs multi-class segmentation and depth map prediction. Scarcity of labels has been conquered by creating a virtual environment which resembles the surgical training scenario to generate dense depth maps and segmentation maps. The proposed convolutional neural network was tested on real surgical training scenarios and showed to be robust to occlusion of the needle. The network achieves a dice score of 0.67 for surgical needle segmentation, 0.81 for needle holder instrument segmentation and a mean absolute error of 6.5 mm for depth estimation.

Image fusion-guided interventions often require planning MR/CT and interventional U/S images to be registered in realtime. Organ motion, patient breathing and inconsistent ultrasound probe positioning during intervention, all contribute to the challenges of real-time 3D deformable registration, where alignment accuracy and computation time are often mutual trade-offs. In this work, we propose a novel framework to align planning and interventional 3D U/S by training patientspecific deep-deformation models (PsDDM) at the planning stage. During intervention, planning 3D U/S volumes are efficiently warped onto the interventional 3D U/S volumes using the trained deep-deformation model, thus enabling the transfer of other modality (planning MR/CT) information in real-time on interventional images. The alignment of planning MR/CT to planning U/S is not time-critical as these can be aligned before the intervention with desired accuracy using any known multimodal deformable registration method. The feasibility of training PsDDM is shown on liver U/S data acquired with a custom-built MR-compatible, hands-free 3D ultrasound probe that allows simultaneous acquisition of planning MR and U/S. Liver U/S volumes exhibit large motion in time due to respiration and therefore serve as a good anatomy to quantify the accuracy of the PsDDM. For quantitative evaluation of the PsDDM, a large vessel bifurcation was manually annotated on 9 U/S volumes that were not used for training the PsDDM but from the same subject. Mean target registration error (TRE) between the centroids was 0.84mm ± 0.39mm, mean Hausdorff distance (HD) was 1.80mm ± 0.29mm and mean surface distance (MSD) was 0.44mm ± 0.06mm for all volumes. In another experiment, the PsDDM was trained using liver volumes from one scanning session, while the model was tested on data from a separate scanning session of the same patient, for which qualitative alignment results were presented.

Mitral valve repair or replacement is important in the treatment of mitral regurgitation. For valve replacement, a transcatheter approach had the possibility of decrease the invasiveness of the procedure while retaining the benefit of replacement over repair. However, fluoroscopy images acquired during the procedure provide no anatomical information regarding the placement of the probe tip once the catheter has entered a cardiac chamber. By using 3D ultrasound and registering the 3D ultrasound images to the fluoroscopy images, a physician can gain a greater understanding of the mitral valve region during transcatheter mitral valve replacement surgery. In this work, we present a graphical user interface which allows the registration of two co-planar X-ray images with 3D ultrasound during mitral valve replacement surgery.

Echocardiography is widely used for obtaining images of the heart for both preoperative diagnostic and intraoperative purposes. For procedures targeting the mitral valve, transesophageal echocardiography (TEE) is the primary imaging modality used as it provides clear 3D images of the valve and surrounding tissues. However, TEE suffers from image artifacts and signal dropout, particularly for structures lying below the valve including chordae tendineae. In order to see these structures, alternative echo views are required. However due to the limited field of view obtainable, the entire ventricle cannot be directly visualized in sufficient detail from a single image acquisition in 3D. This results in a large learning curve for interpreting these images as the multiple views must be reconciled mentally by a clinician. We propose applying an image compounding technique to TEE images acquired from a mid-esophageal position and a number of transgastric positions in order to reconstruct a high-detail image of the mitral valve and sub-valvular structures. This compounding technique utilizes a semi-simultaneous group-wise registration to align the multiple 3D volumes, followed by a weighted intensity compounding step. This compounding technique is validated using images acquired of a custom silicone phantom, excised porcine mitral valve units, and two patient data sets. We demonstrate that this compounding technique accurately captures the physical structures present, including the mitral valve, chordae tendineae and papillary muscles.

External beam therapy has recently been proposed for treatment of ventricular tachycardia (VT). One of the goals of VT ablation is to eliminate myocardial channels by homogenizing infarct scar, thereby inhibiting reentry and eliminating VT. It has been previously shown that proton beam therapy can create lesions in healthy myocardial tissue, thereby suggesting a potential for treatment of VT. To date; however, the effects of proton beam energy in infarcted myocardial tissue is unknown. In the current work, we track proton beam dose maps in a series of delayed contrast-enhanced magnetic resonance imaging swine datasets across 16 weeks to assess the effects of proton beam ablation on infarcted myocardial tissue.

We propose a consistent ultrasound volume stitching framework, with the intention to produce a volume with higher image quality and extended field-of-view in this work. Directly using pair-wise registrations for stitching may lead to geometric errors. Therefore, we propose an approach to improve the image alignment by optimizing a consistency metric over multiple pairwise registrations. In the optimization, we utilize transformed points to effectively compute a distance between rigid transformations. The method has been evaluated on synthetic, phantom and clinical data. The results indicate that our transformation optimization method is effective and our stitching framework has a good geometric precision. Also, the compound images have been demonstrated to have improved CNR values.

We developed a fully automated method for classifying A-line coronary plaques in intravascular optical coherence tomography images using combined deep learning and textural features. The proposed method was trained on 4,292 images from 48 pullbacks giving 80 manually labeled, volumes of interest. Preprocessing steps including guidewire/shadow removal, lumen boundary detection, pixel shifting, and noise reduction were employed. We built a convolutional neural network to extract the deep learning features from the preprocessed image. Traditional textural features were also extracted and combined with deep learning features. Feature selection was performed using the minimum redundancy maximum relevance method. Combined features were utilized as inputs for a random forest classifier. After classification, conditional random field (CRF) method was used for classification noise cleaning. We determined a sub-feature set with the most predictive power. With CRF noise cleaning, sensitivities/specificities were 82.2%/ 90.8% and 82.4%/89.2% for fibrolipidic and fibrocalcific classes, respectively, with good Dice coefficients. The classification noise cleaning step improved performance metrics by nearly 10-15%. The predicted en face classification maps of entire pullbacks agreed favorably to the manually labeled counterparts. Both assessments suggested that our automated measurements gave clinically relevant results. The proposed method is very promising with regards to both clinical treatment planning and research applications.

Ureteroscopy is a conventional procedure used for localization and removal of kidney stones. Laser is commonly used to fragment the stones until they are small enough to be removed. Often, the surgical team faces tremendous challenge to successfully perform this task, mainly due to poor image quality, presence of floating debris and occlusions in the endoscopy video. Automated localization and segmentation can help to perform stone fragmentation efficiently. However, the automatic segmentation of kidney stones is a complex and challenging procedure due to stone heterogeneity in terms of shape, size, texture, color and position. In addition, dynamic background, motion blur, local deformations, occlusions, varying illumination conditions and visual clutter from the stone debris make the segmentation task even more challenging. In this paper, we present a novel illumination invariant optical flow based segmentation technique. We introduce a multi-frame based dense optical flow estimation in a primal-dual optimization framework embedded with a robust data-term based on normalized correlation transform descriptors. The proposed technique leverages the motion fields between multiple frames reducing the effect of blur, deformations, occlusions and debris; and the proposed descriptor makes the method robust to illumination changes and dynamic background. Both qualitative and quantitative evaluations show the efficacy of the proposed method on ureteroscopy data. Our algorithm shows an improvement of 5-8% over all evaluation metrics as compared to the previous method. Our multi-frame strategy outperforms classically used two-frame model.

4D fluoroscopy is a method for real-time 3D visualization of endovascular devices using biplane fluoroscopy. Frame-byframe (15 fps) 4D reconstructions of the device are overlayed on 3D vascular anatomy derived from 3D-DSA. We describe a 4D-assisted guidance platform that provides virtual endoscopic renderings of blood vessels and report on its use for navigating guidewires in a patient-specific vascular phantom. The 4D-assisted platform provides two 4D display modes plus conventional 2D fluoroscopy. Virtual endoscopic 4D mode shows a real-time view of the guidewire tip and the downstream vessel with a viewpoint inside the vessel. Path-planning highlights the target vessel branches. External 4D display mode provides an external rotatable viewpoint of the device and vasculature. In a phantom study, operators navigated a guidewire through branches of a 3D-printed phantom. Performance was compared to navigation with 2D fluoroscopy alone. Operators rated the degree to which they used the 2D and 4D display modes on a Likert scale (1-never, 5-almost always). Quantitative imaging metrics were obtained from processed video recordings. Three users completed 15 of 15 challenges with 4D-assisted display, whereas the 2D-only guidance completion rate was 13/15. With both 2D and 4D displays available, users reported using the 2D display never-to-sometimes (median score = 2) and 4D display often or almost always (median = 5). The virtual endoscopic viewpoint was utilized more frequently than the external viewpoint. 4D fluoroscopy with virtual endoscopic display provides a new and potentially useful mode for visualization of guidewire and catheter manipulations in complex vascular anatomy.

PURPOSE: Surgical navigation has remained a challenge in emergent procedures with mobile targets. Recent advances in RGB-Depth (RGB-D) camera technology have expanded the opportunities for adopting computer vision solutions in computer-assisted surgery. The purpose of this study was to demonstrate the capacity of an RGB-D camera rigidly fixed to a video projector to perform optical marker tracking, depth-based patient registration, and projected target visualization using open-source software. METHODS: The accuracy of marker tracking and system calibration was tested by projecting points onto the corners of a geometrically patterned marker, which was imaged in several locations and orientations. The depth-based registration and practical projection accuracy of the system was evaluated by targeting specific locations on a simulated patient and measuring the distance of the projected points from their targets. RESULTS: The average Euclidean distance between the marker corners and projected points was 5.34 mm (SD 2.93 mm), and the target projection error in the manikin trial following depth-based registration was 4.01 mm (SD 1.51 mm). On average, the distance of the captured point cloud from the manikin model after registration was 1.83 mm (SD 0.15 mm), while the fiducial registration error associated with registering a commercial tracking system to the manikin was 2.47 mm (SD 0.63 mm). CONCLUSION: This study highlights the potential application of RGB-D cameras calibrated to portable video projectors as inexpensive, rapidly deployable navigation systems for use in a variety of procedures, and demonstrates that their accuracy in performing patient registration is suitable for many bedside interventions.

Purpose: Surgical training could be improved by automatic detection of workflow steps, and similar applications of image processing. A platform to collect and organize tracking and video data would enable rapid development of image processing solutions for surgical training. The purpose of this research is to demonstrate 3D Slicer / PLUS Toolkit as a platform for automatic labelled data collection and model deployment. Methods: We use PLUS and 3D Slicer to collect a labelled dataset of tools interacting with tissues in simulated hernia repair, comprised of optical tracking data and video data from a camera. To demonstrate the platform, we train a neural network on this data to automatically identify tissues, and the tracking data is used to identify what tool is in use. The solution is deployed with a custom Slicer module. Results: This platform allowed the collection of 128,548 labelled frames, with 98.5% correctly labelled. A CNN was trained on this data and applied to new data with an accuracy of 98%. With minimal code, this model was deployed in 3D Slicer on real-time data at 30fps. Conclusion: We found the 3D Slicer and PLUS Toolkit platform to be a viable platform for collecting labelled training data and deploying a solution that combines automatic video processing and optical tool tracking. We designed an accurate proof-of-concept system to identify tissue-tool interactions with a trained CNN and optical tracking.

We present an improved patient-specific bronchoscopic navigation scheme by using visual SLAM for bronchoscope tracking. Bronchoscopic navigation system is used to assist physicians during the bronchoscopy examination. Conventional navigation system obtain the camera pose of bronchoscope based on image similarity of real bron- choscopic (RB) and virtual bronchoscopic (VB) images or the pose information from the additional sensor. We propose to use visual SLAM for bronchoscope tracking. The tracking procedure of visual SLAM is improved for processing bronchoscopic scene by considering the inter-frame displacement to filter 2D-3D matches used for pose optimization. The tracking result is registered to CT images to find the relationship between RB and CT the coordinate system. Virtual bronchoscopic views are generated corresponding to real bronchoscopic views by using the registration result and camera pose. Experimental results showed that our proposed method track more frames with higher accuracy on average than the the previous method. The virtual bronchoscopic views have high similarity with real bronchoscopic views.

PURPOSE: Raman spectroscopy is an optical imaging technique used to characterize tissue via molecular analysis. The use of Raman spectroscopy for real-time intraoperative tissue classification requires fast analysis with minimal human intervention. In order to have accurate predictions and classifications, a large and reliable database of tissue classifications with spectra results is required. We have developed a system that can be used to generate an efficient scanning path for robotic scanning of tissues using Raman spectroscopy. METHODS: A camera mounted to a robotic controller is used to take an image of a tissue slide. The corners of the tissue slides within the sample image are identified, and the size of the slide is calculated. The image is cropped to fit the size of the slide and the image is manipulated to identify the tissue contour. A grid set to fit around the size of the tissue is calculated and a grid scanning pattern is generated. A masked image of the tissue contour is used to create a scanning pattern containing only the tissue. The tissue scanning pattern points are transformed to the robot controller coordinate system and used for robotic tissue scanning. The pattern is validated using spectroscopic scans of the tissue sample. The run time of the tissue scan pattern is compared to a region of interest scanning pattern encapsulating the tissue using the robotic controller. RESULTS: The average scanning time for the tissue scanning pattern compared to region of interest scanning reduced by 4 minutes and 58 seconds. CONCLUSION: This method reduced the number of points used for automated robotic scanning, and can be used to reduce scanning time and unusable data points to improve data collection efficiency.

Purpose. We report the initial development of an image-based solution for robotic assistance of pelvic fracture fixation. The approach uses intraoperative radiographs, preoperative CT, and an end effector of known design to align the robot with target trajectories in CT. The method extends previous work to solve the robot-to-patient registration from a single radiographic view (without C-arm rotation) and addresses the workflow challenges associated with integrating robotic assistance in orthopaedic trauma surgery in a form that could be broadly applicable to isocentric or non-isocentric C-arms. Methods. The proposed method uses 3D-2D known-component registration to localize a robot end effector with respect to the patient by: (1) exploiting the extended size and complex features of pelvic anatomy to register the patient; and (2) capturing multiple end effector poses using precise robotic manipulation. These transformations, along with an offline hand-eye calibration of the end effector, are used to calculate target robot poses that align the end effector with planned trajectories in the patient CT. Geometric accuracy of the registrations was independently evaluated for the patient and the robot in phantom studies. Results. The resulting translational difference between the ground truth and patient registrations of a pelvis phantom using a single (AP) view was 1.3 mm, compared to 0.4 mm using dual (AP+Lat) views. Registration of the robot in air (i.e., no background anatomy) with five unique end effector poses achieved mean translational difference ~1.4 mm for K-wire placement in the pelvis, comparable to tracker-based margins of error (commonly ~2 mm). Conclusions. The proposed approach is feasible based on the accuracy of the patient and robot registrations and is a preliminary step in developing an image-guided robotic guidance system that more naturally fits the workflow of fluoroscopically guided orthopaedic trauma surgery. Future work will involve end-to-end development of the proposed guidance system and assessment of the system with delivery of K-wires in cadaver studies.

Image-guided biopsy is crucial for diagnosis, staging, and treatment planning of women with breast cancer. Conventional imaging modalities to guide breast biopsy are not optimal, often resulting in inconclusive or false-negative diagnosis. High-resolution breast-specific functional imaging using positron emission mammography (PEM) has demonstrated increased sensitivity and diagnostic accuracy for detecting breast tumours compared to conventional modalities. PEM shows potential to be an optimal method for tumour detection, but anatomical reference and visualization for needle guidance are not available for guiding biopsy. This paper reports on the development of a mechatronic guidance system designed for integration with an advanced PEM system and ultrasound (US) to improve sampling accuracy during imageguided breast biopsy. The system contains a mechatronic guidance arm and biopsy device, with an integrated US transducer and core-needle biopsy gun. Custom software modules were developed for 3D needle tracking and user guidance. To validate and demonstrate the utility of its system components, the mechatronic guidance arm and biopsy device were assessed using a simulated PEM detector plate. Registration to the simulated detector plate and validation demonstrates accurate needle tracking with a mean Target Registration Error <1mm. Biopsy accuracy was evaluated under ideal detection and mechatronic guidance conditions using tissue-mimicking breast phantoms, allowing targets <1mm to be sampled within 95% confidence. Our results show feasibility for the mechatronic guidance system to be integrated with PEM and US for accurate image-guided breast biopsy. Current work focuses on evaluating the complete mechatronic system ability to accurately guide, target, and biopsy simulated breast lesions.

Femroplasty is a proposed therapeutic method for preventing osteoporotic hip fractures in elderly. Patientspecific femoroplasty based on a preoperative plan, however, requires accurate 3D pose estimation of the proximal femur. Prior work on the development of a navigation system required the attachment of an optical fiducial to the osteoporotic femur. In this paper, we propose a fiducial-free 2D/3D registration pipeline, which uses the pelvis to initialize the femur registration. Intraoperative X-ray images are taken from multiple views to perform compound object registration. The proposed method was tested through a comprehensive simulation study. Performance was evaluated on the femoral head center frame registration error and the injection entry point error on the trochanter surface. The algorithm achieved a mean accuracy of 2.75 ± 2.19 mm for the entry point ℓ₂ distance and 0.81 ± 0.58° for the planning path direction, which is sufficiently accurate for robotic drill positioning.

Medical ultrasound is extensively used to define tissue textures and to characterize lesions, and it is the modality of choice for detection and follow-up assessment of thyroid diseases. Classical medical ultrasound procedures are performed manually by an occupational operator with a hand-held ultrasound probe. These procedures require high physical and cognitive burden and yield clinical results that are highly operator-dependent, therefore frequently diminishing trust in ultrasound imaging data accuracy in repetitive assessment. A robotic ultrasound procedure, on the other hand, is an emerging paradigm integrating a robotic arm with an ultrasound probe. It achieves an automated or semi-automated ultrasound scanning by controlling the scanning trajectory, region of interest, and the contact force. Therefore, the scanning becomes more informative and comparable in subsequent examinations over a long-time span. In this work, we present a technique for allowing operators to reproduce reliably comparable ultrasound images with the combination of predefined trajectory execution and real-time force feedback control. The platform utilized features a 7-axis robotic arm capable of 6-DoF force-torque sensing and a linear-array ultrasound probe. The measured forces and torques affecting the probe are used to adaptively modify the predefined trajectory during autonomously performed examinations and probe-phantom interaction force accuracy is evaluated. In parallel, by processing and combining ultrasound B-Mode images with probe spatial information, structural features can be extracted from the scanning volume through a 3D scan. The validation was performed on a tissue-mimicking phantom containing thyroid features, and we successfully demonstrated high image registration accuracy between multiple trials.

Minimally invasive approaches to treating tumors of the pharynx and larynx like trans-oral surgery have improved patient outcome, but challenges remain in localizing tumors for margin control. Introducing necessary retractors and scopes deforms the anatomy and the tumor, rendering preoperative imaging inaccurate and making tumor localization difficult. This paper describes a pipeline that uses preoperative imaging to generate a hybrid FEM-multibody model and then dynamically simulates tongue deformation due to insertion of an electromagnetically-tracked laryngoscope. We hypothesize that the simulation output will be a sufficient estimate of the final intraoperative state and thus provide the surgeon with more accurate guidance during surgical resection. This pipeline was trialed on a cadaver head. The skull, mandible, and laryngoscope were tracked, and fiducial clips were embedded in the tongue for calculating target localization error (TLE) between the simulated and real tongue deformation. Registration accuracies were 1.1, 1.3, and 0.8 mm, respectively, for the tracked skull, mandible, and laryngoscope, and tracking and segmentation validation between the last tracked frame and the ground-truth intraoperative CT was 0.8, 0.9, and 1.2 mm, respectively. TLE of 6.4±2.5 mm was achieved for the full pipeline, in contrast to the total tongue deformation of 37.2±11.4 mm (via tongue clips) between the preoperative and intraoperative CT. Use of tracking and deformation modeling is viable to estimate deformation of the tongue during laryngoscopy. Future work involves additional intraoperative data streams to help further refine model parameters and improve localization.

Image-guided liver surgery relies on intraoperatively acquired data to create an accurate alignment between image space and the physical patient anatomy. Often, sparse data of the anterior liver surface can be collected for these registrations. However, achieving accurate registration to sparse surface data when soft tissue deformation is present remains a challenging open problem. While many approaches have been developed, a common standard for comparing algorithm performance has yet to be adopted. The image-to-physical liver registration sparse data challenge offers a publicly available dataset of realistic sparse data patterns collected on a deforming liver phantom for the purpose of evaluating and comparing potential registration approaches. Additionally, the challenge is designed to allow testing and characterization of these methods as a general utility for the registration community. Using this challenge environment, an inverse biomechanical method for deformable registration to sparse data was investigated with respect to how whole-organ target registration error (TRE) is impacted by a model parameter that controls the spatial reconstructive resolution of mechanical loads applied to the organ. For this analysis, this resolution parameter was varied across a wide range of values and TRE was calculated from the challenge dataset. An optimal parameter value for model resolution was found and average TRE across the 112 sparse data challenge cases was reduced to 3.08 ± 0.85 mm, an approximate 32% improvement over previously reported results. The value of the data offered by the sparse data challenge is evident. This work was performed entirely using information automatically generated by the challenge submission and processing site.

Because many patients diagnosed with hepatocellular carcinoma are not eligible for liver transplantation or resection, there has been a great deal of interest in developing locoregional therapies such as thermal ablation. One such thermal ablation therapy is microwave ablation. While benefits have been gained in the management of disease, local recurrence in locoregional therapies is still very common and represents a significant problem. One suggested factor is the presence of soft tissue deformation which is thought to compromise image-to-physical targeting of diseased tissue. This work focuses on presenting a hepatic phantom with an embedded mock tumor target and studying the effects of deformation on ablation when using image-to-physical rigid and non-rigid alignment approaches. While being deformable, the hepatic phantom was designed to enable optical visibility of the ablation zone with target lesion visibility in CT images post-treatment using albumin, agar, formaldehyde, and water constituents. Additionally, a physical mock tumor target phantom was embedded in the hepatic liver phantom and contained CT contrast agent for the designation of lesion prior to mock intervention. Using this phantom, CT scans and sparse-surface data were collected to perform rigid and nonrigid registrations. The registrations allowed for the navigation of the ablation probe to the center of the mock lesions using a custom-built guidance system; this was then followed by microwave ablation treatments. Approximately 96.8% of the mock lesion was ablated using nonrigid registration to guide delivery while none of the mock lesion was ablated using the rigid alignment for guidance, i.e. a completely missed target. This preliminary data demonstrates an improvement in the accuracy of target ablation using a guidance system that factors in soft tissue deformation.

Although robotic instrumentation has revolutionized manipulation in oncologic laparoscopy, there remains a significant need for image guidance during the exposure portion of certain abdominal procedures. The high degree of mobility and potential for deformation associated with abdominal organs and related structures poses a significant challenge to implementing image-based navigation for the initial phase of robot-assisted laparoscopic partial nephrectomy (RALPN). This work introduces two key elements of a RALPN exposure simulation framework: a model for laparoscopic exposure and a compact representation of anatomical geometry suitable for integration into a statistical estimation framework. Data to drive the exposure simulation were collected during a clinical RALPN case in which the robotic endoscope was tracked in six dimensions. An initial rigid registration was performed between a preoperative CT scan and the frame of the optical tracker, allowing the endoscope trajectory to be replayed over tomography to simulate anatomical observations with realistic kinematics. CT data from five study subjects were combined with four publicly available datasets to produce a mean kidney shape. This template kidney was fit back to each of the input models by optimally tuning a set of eight parameters, achieving an average RMSE of 2.18mm. These developments represent important steps toward a full, clinically-relevant framework for simulating organ exposure and testing navigation algorithms. In future work, a particle filter estimation scheme will be integrated into the simulation to incrementally optimize correspondences between parametric anatomical models and simulated or reconstructed endoscopic observations.

Accurate segmentation of organs-at-risk is important in prostate cancer radiation therapy planning. However, poor soft tissue contrast in CT makes the segmentation task very challenging. We propose a deep convolutional neural network approach to automatically segment the prostate, bladder, and rectum from pelvic CT. A hierarchical coarse-to-fine segmentation strategy is used where the first step generates a coarse segmentation from which an organ-specific region of interest (ROI) localization map is produced. The second step produces detailed and accurate segmentation of the organs. The ROI localization map is generated using a 3D U-net. The localization map helps adjusting the ROI of each organ that needs to be segmented and hence improves computational efficiency by eliminating irrelevant background information. For the fine segmentation step, we designed a fully convolutional network (FCN) by combining a generative adversarial network (GAN) with a U-net. Specifically, the generator is a 3D U-net that is trained to predict individual pelvic structures, and the discriminator is an FCN which fine-tunes the generator predicted segmentation map by comparing it with the ground truth. The network was trained using 100 CT datasets and tested on 15 datasets to segment the prostate, bladder and rectum. The average Dice similarity (mean±SD) of the prostate, bladder and rectum are 0.90±0.05, 0.96±0.06 and 0.91±0.09, respectively, and Hausdorff distances of these three structures are 5.21±1.17, 4.37±0.56 and 6.11±1.47 (mm), respectively. The proposed method produces accurate and reproducible segmentation of pelvic structures, which can be potentially valuable for prostate cancer radiotherapy treatment planning.

With the advent of Cardiac Cine Magnetic Resonance (CMR) Imaging, there has been a paradigm shift in medical technology, thanks to its capability of imaging different structures within the heart without ionizing radiation. However, it is very challenging to conduct pre-operative planning of minimally invasive cardiac procedures without accurate segmentation and identification of the left ventricle (LV), right ventricle (RV) blood-pool, and LV-myocardium. Manual segmentation of those structures, nevertheless, is time-consuming and often prone to error and biased outcomes. Hence, automatic and computationally efficient segmentation techniques are paramount. In this work, we propose a novel memory-efficient Convolutional Neural Network (CNN) architecture as a modification of both CondenseNet, as well as DenseNet for ventricular blood-pool segmentation by introducing a bottleneck block and an upsampling path. Our experiments show that the proposed architecture runs on the Automated Cardiac Diagnosis Challenge (ACDC) dataset using half (50%) the memory requirement of DenseNet and one-twelfth (∼ 8%) of the memory requirements of U-Net, while still maintaining excellent accuracy of cardiac segmentation. We validated the framework on the ACDC dataset featuring one healthy and four pathology groups whose heart images were acquired throughout the cardiac cycle and achieved the mean dice scores of 96.78% (LV blood-pool), 93.46% (RV blood-pool) and 90.1% (LVMyocardium). These results are promising and promote the proposed methods as a competitive tool for cardiac image segmentation and clinical parameter estimation that has the potential to provide fast and accurate results, as needed for pre-procedural planning and / or pre-operative applications

Planning the optimal time of intervention for pulmonary valve replacement surgery in patients with the congenital heart disease Tetralogy of Fallot (TOF) is mainly based on ventricular volume and function according to current guidelines. Both of these two biomarkers are most reliably assessed by segmentation of 3D cardiac magnetic resonance (CMR) images. In several grand challenges in the last years, U-Net architectures have shown impressive results on the provided data. However, in clinical practice, data sets are more diverse considering individual pathologies and image properties derived from different scanner properties. Additionally, specific training data for complex rare diseases like TOF is scarce. For this work, 1) we assessed the accuracy gap when using a publicly available labelled data set (the Automatic Cardiac Diagnosis Challenge (ACDC) data set) for training and subsequent applying it to CMR data of TOF patients and vice versa and 2) whether we can achieve similar results when applying the model to a more heterogeneous data base. Multiple deep learning models were trained with four-fold cross validation. Afterwards they were evaluated on the respective unseen CMR images from the other collection. Our results confirm that current deep learning models can achieve excellent results (left ventricle dice of 0.951±0.003/0.941±0.0007 train/validation) within a single data collection. But once they are applied to other pathologies, it becomes apparent how much they overfit to the training pathologies (dice score drops between 0.072±0.001 for the left and 0.165±0.001 for the right ventricle).

Breast cancer is the most common cancer in American women, and is the second most deadly. Current guidance approaches for breast cancer surgery provide distance to a seed implanted near the tumor centroid. Large deformations between preoperative imaging and surgical presentation, coupled with the lack of tumor extent information leads to difficulty in ensuring complete tumor resection. Here we propose a novel image guidance platform that utilizes character-based fiducials for easy detection and small fiducial points for precise and accurate localization. Our system is work-flow friendly, and near-real time with use of stereo cameras for surface acquisition. Using simple image processing techniques, the proposed technique can localize fiducials and character labels, providing updates without relying on video history. Character based fiducial labels can be recognized and used to determine correspondence between left and right images in a pair of stereo cameras, and frame to frame in a sequence of images during a procedure. Letters can be recognized with 89% accuracy using the MATLAB built in optical character recognition function, and an average of 81% of points can be accurately labeled and localized. The stereo camera system can determine surface points with accuracy below 2mm when compared to optically tracked stylus points. These surface points are incorporated to a four-panel guidance display that includes preoperative supine MR, tracked ultrasound, and a model view of the breast and tumor with respect to optically tracked instrumentation.

Surgical tools detection for intraoperative surgical navigation system is essential for better coordination among surgical team in operating room. Because Orthopaedic surgery (OS) differs from laparoscopic, due to a large variety of surgical instruments and techniques making its procedures complicated. Compared to usual object detection in natural images, OS video images are confounded by inhomogeneous illumination; it is hard to directly apply existing studies that are developed for others. Additionally, acquiring Orthopaedic surgery videos is difficult due to recording of surgery videos in restricted surgical environment. Therefore, we propose a deep learning (DL) approach for surgery tools detection in OS videos by integrating knowledge of diverse representative surgery and non-surgery images of tools into the model using transfer learning (TL) and data augmentation. The proposed method has been evaluated for five surgical tools using knee surgery images following 10-fold cross validation. It shows, proposed model (mAP 62.46%) outperforms over conventional model (mAP 60%).

Spinal degeneration and deformity present an enormous healthcare burden, with spine surgery among the main treatment modalities. Unfortunately, spine surgery (e.g., lumbar fusion) exhibits broad variability in the quality of outcome, with ~20-40% of patients gaining no benefit in pain or function (“failed back surgery”) and earning criticism that is difficult to reconcile versus rapid growth in frequency and cost over the last decade. Vital to advancing the quality of care in spine surgery are improved clinical decision support (CDS) tools that are accurate, explainable, and actionable: accurate in prediction of outcomes; explainable in terms of the physical / physiological factors underlying the prediction; and actionable within the shared decision process between a surgeon and patient in identifying steps that could improve outcome. This technical note presents an overview of a novel outcome prediction framework for spine surgery (dubbed SpineCloud) that leverages innovative image analytics in combination with explainable prediction models to achieve accurate outcome prediction. Key to the SpineCloud framework are image analysis methods for extraction of high-level quantitative features from multi-modality peri-operative images (CT, MR, and radiography) related to spinal morphology (including bone and soft-tissue features), the surgical construct (including deviation from an ideal reference), and longitudinal change in such features. The inclusion of such image-based features is hypothesized to boost the predictive power of models that conventionally rely on demographic / clinical data alone (e.g., age, gender, BMI, etc.). Preliminary results using gradient boosted decision trees demonstrate that such prediction models are explainable (i.e., why a particular prediction is made), actionable (identifying features that may be addressed by the surgeon and/or patient), and boost predictive accuracy compared to analysis based on demographics alone (e.g., AUC improved by ~25% in preliminary studies). Incorporation of such CDS tools in spine surgery could fundamentally alter and improve the shared decisionmaking process between surgeons and patients by highlighting actionable features to improve selection of therapeutic and rehabilitative pathways.

This paper contributes to the burgeoning field of surgical data science. Specifically, multi-modal integration of relevant patient data is used to determine who should undergo a complex pancreatic resection. Intraductal papillary mucinous neoplasms (IPMNs) represent cystic precursor lesions of pancreatic cancer with varying risk for malignancy. We combine radiomic analysis of diagnostic computed tomography (CT) with protein markers extracted from the cyst fluid to create a unified prediction model to identify high-risk IPMNs. Patients with high-risk IPMN would be sent for resection, whereas patients with low-risk cystic lesions would be spared an invasive procedure. We extracted radiomic features from CT scans and combined this with cyst-fluid markers. The cyst fluid model yielded an area under the curve (AUC) of 0.74. Adding the QI model improved performance with an AUC of 0.88. Radiomic analysis of routinely acquired CT scans combined with cyst fluid inflammatory markers provides accurate prediction of risk of pancreatic cancer progression.

Cochlear implants (CI) use an array of electrodes surgically threaded into the cochlea to restore hearing sensation. Techniques for predicting the insertion depth of the array into the cochlea could guide surgeons towards more optimal placement of the array to reduce trauma and preserve the residual hearing. After analyzing the CT images of 86 lateral wall positioned straight electrode array CI recipients, a linear regression model is proposed for insertion depth prediction based on cochlea size, array geometry, and base insertion depth (BID). The model is able to accurately predict angular insertion depths with standard deviation of 41 degrees.

Glioblastoma, the most common and aggressive adult brain tumor, is considered non-curative at diagnosis. Current literature shows promise on imaging-based overall survival prediction for patients with glioblastoma while integrating advanced (structural, perfusion, and diffusion) multiparametric magnetic resonance imaging (AdvmpMRI). However, most patients prior to initiation of therapy typically undergo only basic structural mpMRI (BasmpMRI, i.e., T1,T1-Gd,T2,T2-FLAIR) pre-operatively, rather than Adv-mpMRI. Here we assess a retrospective cohort of 101 glioblastoma patients with available Adv-mpMRI from a previous study, which has shown that an initial feature panel (IFP) extracted from Adv-mpMRI can yield accurate overall survival stratification. We further focus on demonstrating that equally accurate prediction models can be constructed using augmented feature panels (AFP) extracted solely from Bas-mpMRI, obviating the need for using Adv-mpMRI. The classification accuracy of the model utilizing Adv-mpMRI protocols and the IFP was 72.77%, and improved to 74.26% when utilizing the AFP on Bas-mpMRI. Furthermore, Kaplan-Meier analysis demonstrated superior classification of subjects into short-, intermediate-, and long-survivor classes when using AFP on Basic-mpMRI. This quantitative evaluation indicates that accurate survival prediction in glioblastoma patients is feasible by using solely Bas-mpMRI and integrative radiomic analysis can compensate for the lack of Adv-mpMRI. Our finding holds promise for predicting overall survival based on commonly-acquired Bas-mpMRI, and hence for potential generalization across multiple institutions that may not have access to Adv-mpMRI, facilitating better patient selection.

Lung cancer is the deadliest form of cancer. New lung cancer screening programs are currently being deployed worldwide. This increases the premium on accurate lung cancer staging as well as increasing the number of detected early-stage cancer patients. Accurate staging requires sampling lymph nodes in a sufficient number of nodal stations throughout the central chest. To this end, physicians use the world standard International Association for the Study of Lung Cancer’s (IASLC) TNM lung cancer staging model and lymph node station map. To determine the nodal stage, the physician performs a bronchoscopic lymph node staging procedure, a minimally-invasive procedure in which the physician samples lymph nodes from multiple diagnostic sites in the central chest using a bronchoscope. Image guided-bronchoscopy (IGB) systems, now a part of widely-accepted practice, greatly assist in this procedure by drawing upon information from the patient’s three-dimensional (3D) x-ray computed tomography (CT) scan. Unfortunately, even with modern IGB systems, most physicians still do not stage lung cancer in a comprehensive manner, sampling only a few nodes for each patient. Furthermore, current IGB systems do not integrate N stage information into their planning or guidance strategies, nor do they attempt to optimize procedure routes to efficiently visit multiple diagnostic sites. To bridge this gap, we propose new methods tailored to planning more comprehensive lymph node staging procedures. Specifically, our development features two interconnected contributions toward creating a computer-based planning and guidance system. First is a method for defining the nodal staging zones and automatically labeling the nodal stage value of each defined lymph node. Second, we develop a method for creating efficient multi-destination guidance plans visiting diagnostic sites throughout the central chest. We demonstrate our methods using CT image data collected from human lung cancer patients.

To diagnose peripheral lung tumors, bronchoscopy is recommended for tissue sampling. Because most peripheral target regions of interest reside outside the airways, during bronchoscopy radial probe endobronchial ultrasound (RP-EBUS) is often used to provide extraluminal information and assist in locating biopsy sites. RP-EBUSguided transbronchial needle aspiration (TBNA) has demonstrated its potential over conventional TBNA to improve the diagnostic yield of peripheral pulmonary nodule biopsy. Meanwhile, image-guided bronchoscopy systems have been introduced to enable better planning and navigation. Currently, the physician has to deal with two disconnected imaging domains during live bronchoscopy; i.e., an image-guided system and RP-EBUS. The state-of-the-art image-guided bronchoscopy systems provide no guidance for RP-EBUS and biopsy targeting during live bronchoscopy, which is needed for accurate diagnosis of peripheral pulmonary nodules. To fill this gap, we expand the concept of virtual bronchoscopy (VB) in image-guided bronchoscopy systems and build a virtual RP-EBUS model to simulate the RP-EBUS probe in the computed-tomography (CT)-based virtual chest space. Results with human patient data illustrate the synchronized visualization of the virtual RP-EBUS model with the 3D chest model.

Brain shift is a non-rigid deformation of brain tissue that is affected by loss of cerebrospinal fluid, tissue manipulation and gravity among other phenomena. This deformation can negatively influence the outcome of a surgical procedure since surgical planning based on pre-operative image becomes less valid. We present a novel method to compensate for brain shift that maps preoperative image data to the deformed brain during intra-operative neurosurgical procedures and thus increases the likelihood of achieving a gross total resection while decreasing the risk to healthy tissue surrounding the tumor. Through a 3D/2D non-rigid registration process, a 3D articulated model derived from pre-operative imaging is aligned onto 2D images of the vessels viewed through the surgical miscroscopic intra-operatively. The articulated 3D vessels constrain a volumetric biomechanical model of the brain to propagates cortical vessel deformation to the parenchyma and in turn to the tumor. The 3D/2D non-rigid registration is performed using an energy minimization approach that satisfies both projective and physical constraints. Our method is evaluated on real and synthetic data of human brain showing both quantitative and qualitative results and exhibiting its particular suitability for real-time surgical guidance.

Fusion biopsy reduces false negative rates in prostatic cancer detection compare to systemic biopsy. However, accuracy in biopsy sampling depends upon quality of alignment between pre-operative 3D MR and intra-operative 2D US. During live biopsy, the US-MR alignment may be disturbed due to prostate or patient rigid motion. Further, prostate gland deform due to probe pressure, which add error in biopsy sampling. In this paper, we describe a method for real-time 2D-3D multimodal registration, utilizing deep learning, to correct for rigid and deformable errors. Our method do not require an intermediate 3D US and works in real-time with an average runtime of 112 ms for both rigid and deformable corrections. On 12 patient data, our method reduces mean trans-registration error (TRE) from 8.890±5.106 mm to 2.988±1.513 mm, comparable to other state of the arts in accuracy.

In neurosurgery, technical solutions for visualizing the border between healthy brain and tumor tissue is of great value, since they enable the surgeon to achieve gross total resection while minimizing the risk of damage to eloquent areas. By using real-time non-ionizing imaging techniques, such as hyperspectral imaging (HSI), the spectral signature of the tissue is analyzed allowing tissue classification, thereby improving tumor boundary discrimination during surgery. More particularly, since infrared penetrates deeper in the tissue than visible light, the use of an imaging sensor sensitive to the near-infrared wavelength range would also allow the visualization of structures slightly beneath the tissue surface. This enables the visualization of tumors and vessel boundaries prior to surgery, thereby preventing the damaging of tissue structures. In this study, we investigate the use of Diffuse Reflectance Spectroscopy (DRS) and HSI for brain tissue classification, by extracting spectral features from the near infra-red range. The applied method for classification is the linear Support Vector Machine (SVM). The study is conducted on ex-vivo porcine brain tissue, which is analyzed and classified as either white or gray matter. The DRS combined with the proposed classification reaches a sensitivity and specificity of 96%, while HSI reaches a sensitivity of 95% and specificity of 93%. This feasibility study shows the potential of DRS and HSI for automated tissue classification, and serves as a fjrst step towards clinical use for tumor detection deeper inside the tissue.

Patient-to-image registration is a key step for guidance in computer-assisted surgery during interventions like cochlear implant or deep brain stimulation surgeries. Automatizing fiducial detection and localization in pre-operative images of the patient can lead to better registration accuracy, reduced human errors and shorter intervention time. We present an algorithm that builds on earlier approaches with morphological functions and pose estimation algorithms. A Convolutional Neural Network is proposed for the fiducial classification task. A digital experiment, with cone-beam CT imaging software, is performed to determine the accuracy of the algorithm. The inputs to this software are virtual phantoms represented as 3D surface models (meshes) of skull and (screw and spherical) fiducial markers combined with specific imaging parameters like material properties, detector resolution, etc. The software generates realistic CT images for establishing a ground-truth measure to validate the algorithm. The localized fiducial positions in the image by the presented algorithm were compared to the actual known positions in the phantom models. The difference represents the fiducial localization error (FLE). Validation data sets with different slice thicknesses contain screws and spherical markers of different dimensions. The achieved FLE mean and its standard deviation for a screw and spherical marker are 58 (14) μm and 14 (6) μm, respectively. Large marker volume and smaller voxel size yield smaller FLEs. Furthermore, we found that attenuating noise by mesh smoothing has a minor effect on localization accuracy.

Purpose We present a systematic error analysis for a navigation system which allows image fusion between two arbitrary image modalities. A prototype was built for prostate biopsies merging pre-operative magnetic resonance images (MRI) with interventional 3D free-hand transrectal US (TRUS). Methods The fusion between MRI and TRUS images was performed by replacing the inter-modal MRI-TRUS registration by an intra-modal 3D/3D-US/TRUS registration and a pre-operative image fusion between MRI and 3D(-abdominal)-US using an optical tracking system (OTS). The error of the inter-modal MRI/3D(-abdominal)- US image fusion was evaluated using a multi-cone phantom and a prostate phantom by calculating a target registration error (TRE) and the Dice score of segmented lesions (radius 5 mm) from the prostate phantom. The pre-operative 3D(-abdominal)-US was then registered with the TRUS images and the TRE error was calculated. Finally, a Dice score of segmented lesions between TRUS and MRI was determined. Results The fiducial registration error for the 3D(-abdominal)-US and TRUS calibration resulted in 0:87 mm and 1:18 mm, respectively. The TRE for the 3D abdominal transducer was 1:08 mm. The TRE of the MRI/3D- US image fusion using a multi-cone phantom was 1:81 mm. The Dice-score calculated from test lesions segmented in MRI and TRUS resulted in an error of 0.67. The TRE between the 3D-US and the TRUS was 2:10 mm. The Dice score from segmented lesions in MRI and TRUS was 0:67. Conclusions The intra-modal registration via OTS between MRI and TRUS images showed sufficient accuracy for prostate biopsy.

Colonoscopies are routine, low-risk procedures that are used to screen patients for diseases like colorectal cancer. However, often times they are performed with moderate sedation (i.e. conscious sedation) which may result in pain for patients. With moderate sedation, these procedures can be extremely painful, with one study reporting that more than 20% of patients experienced severe pain. This pain is usually the result of a phenomenon called “endoscopic looping,” which occurs when the scope loops within the patient’s bowels and stretches out their intestines. Looping is extremely common, and can occur in up to 90% of procedures. This study reports a low-cost endoscopy visualization device aimed to decrease patient pain during colonoscopies and time lost due to complications experienced during procedures. The device consists of a flexible piezoelectric sensor to detect applied forces during looping, bending, or compression. In order to use the device, the endoscopist inserts the piezoelectric cable fully within the working channel of the colonoscope before the colonoscopy begins. The piezoelectric cable is connected to an external monitor. If extreme forces or bends are detected by the piezoelectric cable, a notification can appear on screen that a loop is forming. Once the colonoscope reaches the desired location, the endoscopist removes the piezoelectric cable, leaving the working channel open for use by other tools, such as biopsy forceps.

Nowadays, 3D ultrasound (US) has been employed rapidly in medical intervention therapies, such as cardiac catheterization. To efficiently interpret 3D US images and localize the catheter during the surgery, an experienced sonographer is required. As a consequence, image-based catheter detection can be a benefit to sonographer to localize the instrument in the 3D US images timely. Conventionally, the 3D imaging methods are based on the Cartesian domain, which is limited by bandwidth and information lose when it is converted from the original acquisition space-Frustum domain. The exploration of catheter segmentation in Frustum space helps to reduce the computational cost and improve efficiency. In this paper, we present a catheter segmentation method in 3D Frustum image via a deep convolutional network (DCNN). To better describe 3D information and reduce the complexity of DCNN, cross-planes with spatial gaps are extracted for each voxel. Then, the cross-planes of the voxel are processed by the DCNN to distinguish it, whether it is a catheter voxel or not. To accelerate the prediction efficiency on whole US Frustum volume, a filter-based pre-selection is applied to reduce the computational cost of the DCNN. Based on experiments on the ex-vivo dataset, our proposed method can segment the catheter in Frustum images with 0.67 Dice score within 3 seconds, which indicates the possibility of real-time application.

This work explores a Generative Adversarial Network (GAN) based approach for hemorrhage detection on color Doppler ultrasound images of blood vessels. Given the challenges of collecting hemorrhage data and the inherent pathology variability, we investigate an unsupervised anomaly detection network which learns a manifold of normal blood flow variability and subsequently identifies anomalous flow patterns that fall outside the learned manifold. As an initial, feasibility study, we collected ultrasound color Doppler images of brachial arteries from 11 healthy volunteers. The images were pre-processed to mask out velocities in surrounding tissues and were subsequently cropped, resized, augmented and normalized. The network was trained on 1530 images from 8 healthy volunteers and tested on 70 images from 2 healthy volunteers. In addition, the network was tested on 6 synthetic images generated to simulate blood flow velocity patterns at the site of hemorrhage. Results show significant (p<0.05) differences in anomaly scores of normal arteries and simulated injured arteries. The residual images, or the reconstruction error maps, show promise in localizing anomalies at pixel level.

Prostate segmentation on magnetic resonance images (MRI) is an important step for prostate cancer diagnosis and therapy. After the birth of deep convolution neural network (DCNN), prostate segmentation has achieved great success in supervised segmentation. However, these works are mostly based on abundant fully labeled pixel-level image data. In this work, we propose a weakly supervised prostate segmentation (WS-PS) method based on image-level labels. Although the image-level label is not sufficient for an exact prostate contour, it contains potential information which is helpful to make sure a coarse contour. This information is referred to confident information in this paper. Our WS-PS method includes two steps which are mask generation and prostate segmentation. First, the mask generation (MG) exploits a class activation maps (CAM) technique to generate a coarse probability map for MRI slices based on image-level label. These elements of the coarse map which have higher probability are considered to contain more confident information. To make use of confident information from coarse probability map, a similarity model (S-Model) is introduced to refine the coarse map. Second, the prostate segmentation (PS) uses a residual U-Net with a size constraint loss to segment prostate based on the refined mask obtained from MG. The proposed method achieves a mean Dice similarity coefficient (DSC) of 83.39% as compared to the manually delineated ground-truth. The experimental results indicate that our weakly supervised method can achieve a satisfactory segmentation on prostate MRI only with image-level labels.

PURPOSE: Virtual reality (VR) simulation is an effective training system for medical residents, allowing them to gain and improve upon surgical skills in a realistic environment while also receiving feedback on their performance. Percutaneous nephrolithotomy is the most common surgical treatment for the removal of renal stones. We propose a workflow to generate 3D soft tissue and bone models from computed tomography (CT) images, to be used and validated in a VR nephrolithotomy simulator. METHODS: Venous, delay, non-contrast, and full body CT scans were registered and segmented to generate 3D models of the abdominal organs, skin, and bone. These models were decimated and re-meshed into low-polygon versions while maintaining anatomical accuracy. The models were integrated into a nephrolithotomy simulator with haptic feedback and scoring metrics. Urology surgical experts assessed the simulator and its validity through a questionnaire based on a 5-point Likert scale. RESULTS: The workflow produced soft tissue and bone models from patient CT scans, which were integrated into the simulator. Surgeon responses indicated level 3 and above for face validity and level 4 and above for all other aspects of medical simulation validity: content, construct, and criterion. CONCLUSION: We designed an effective workflow to generate 3D models from CT scans using open source and modelling software. The low resolution of these models allowed integration in a VR simulator for visualization and haptic feedback, while anatomical accuracy was maintained.

Innovative technologies for minimally invasive interventions have the potential to add value to vascular procedures in the hybrid operating theater (HOT). Restricted budgets require prioritization of the development of these technologies. We aim to provide vascular surgeons with a structured methodology to incorporate possibly conflicting criteria in prioritizing the development of new technologies. We propose a multi-criteria decision analysis framework to evaluate the value of innovative technologies for the HOT based on the MACBETH methodology. The framework is applied to a specific case: the new HOT in a large teaching hospital. Three upcoming innovations are scored for three different endovascular procedures. Two vascular surgeons scored the expected performance of these innovations for each of the procedures on six performance criteria and weighed the importance of these criteria. The overall value of the innovations was calculated as the weighted average of the performance scores. On a scale from 0-100 describing the overall value, the current HOT scored halfway the scale (49.9). A wound perfusion measurement tool scored highest (69.1) of the three innovations, mainly due to the relatively high score for crural revascularization procedures (72). The novel framework could be used to determine the relative value of innovative technologies for the HOT. When development costs are assumed to be similar, and a single budget holder decides on technology development, priority should be given to the development of a wound perfusion measurement tool.

PURPOSE: There are several interstitial (needle based) image-guided ablation planning systems available, but most of them are closed or unsupported. We propose an open source software platform for the planning of image-guided interstitial ablation procedures, providing generic functionality and support for specialized plug-ins. METHODS: The patient’s image data is loaded or streamed into the system and the relevant structures are segmented. The user places fiducial points as ablation needle entries and tips, sets the ablation times, and the thermal dose is calculated by a dose engine. The thermal dose is then visualized on the 2D image slices and 3D rendering using a combination of isodose lines and surfaces. Quantitative feedback is provided by dose volume histograms. The treatment plan can be iteratively edited until satisfactory dose distribution is achieved. We performed a usability study with eight novice users in which they were asked to create a satisfactory treatment plan. RESULTS: Interventionists can use the proposed system to create and visualize thermal ablation plans. Researchers can use the platform to create a wide range of specialized applications by adding plug-ins for various types of ablation methods, thermal models, and dose calculation engines. Existing extensions of the platform can provide real-time imaging and tracked or robotic navigation to aid the user in optimal needle placement. From our usability study, the users found the visual information well represented and the platform intuitive to use. The users averaged 4.4 recalculation attempts before finding an optimal treatment, which was evaluated as 100% necrosis of the tumor. CONCLUSION: The developed platform fulfills a demand for a universal and shared ablation planning system. While also being supported by the state-of-the-art development of specialized plug-ins, the open source system can adapt to the desired dose calculation or ablation procedure.

PURPOSE: Scoliosis screening is currently only implemented in a few countries due to the lack of a safe and accurate measurement method. Spinal ultrasound is a viable alternative to X-ray, but manual annotation of images is difficult and time consuming. We propose using deep learning through a U-net neural network that takes consecutive images per individual input, as an enhancement over using single images as input for the neural network.

METHODS: Ultrasound data was collected from nine healthy volunteers. Images were manually segmented. To accommodate for consecutive input images, the ultrasound images were exported along with previous images stacked to serve as input for a modified U-net. Resulting output segmentations were evaluated based on the percentage of true negative and true positive pixel predictions.

RESULTS: After comparing the single to five-image input arrays, the three-image input had the best performance in terms of true positive value. The single input and three-input images were then further tested. The single image input neural network had a true negative rate of 99.79%, and a true positive rate of 63.56%. The three-image input neural network had a true negative rate of 99.75%, and a true positive rate of 66.64%.

CONCLUSION: The three-image input network outperformed the single input network in terms of the true positive rate by 3.08%. These findings suggest that using two additional input images consecutively preceding the original image assist the neural network in making more accurate predictions.

Cardiac surgeons rely on diagnostic imaging for preoperative planning. Recently, developments have been made on improving 3D ultrasound (US) spatial compounding tailored for cardiac images. Compounded 3D ultrasound volumes are able to capture complex anatomical structures at a level similar to a CT scan, however these images are difficult to display and visualize due to an increased amount of surrounding tissue captured including excess noise at the volume boundaries. Traditional medical image visualization software does not easily allow for viewing 2D slices at arbitrary angles, and 3D rendering techniques do not adequately capture depth information without the use of advanced transfer functions or other depth-encoding techniques that must be tuned to each individual data set. Previous studies have shown that the effective use of virtual reality (VR) can improve image visualization, usability and reduce surgical errors in case planning. We demonstrate the novel use of a VR system for the application of measuring chordal lengths from compounded transesophageal and transgastric echocardiography (TEE, TTE) ultrasound images. Compounded images are constructed from TEE (en-face) views registered and spatially compounded with multiple TEE transgastric views in order to capture both the mitral valve leaflets and chordae tendineae with high levels of detail. Users performed the task of taking linear measurements of chordae visible in these images using both traditional software and a VR platform. Compared to traditional software, the VR platform offered a more intuitive experience with respect to orientation, however users felt there was a lack of precision when performing the measurement tasks.

In open spine surgery, the accuracy of image guidance is compromised by alignment change between supine preoperative CT images (pCT) and prone intraoperative positioning. We have developed a level-wise registration framework to compensate for the intervertebral motion by updating pCT to match with intraoperative stereovision (iSV) data of the exposed spine. In this study, we compared performance of the iSV image updating system in different lengths of exposure using retrospective data from one cadaver pig specimen. Specifically, L1 to L6 were exposed and 3 metallic mini-screws were implanted on each level as “ground truth” locations. The spine was positioned supine to acquire pCT, and then positioned prone to acquire iSV using a hand-held iSV device. One image pair of iSV was acquired from each exposed vertebra. Three exposure lengths were evaluated by selecting data from corresponding levels to compare performance: 6 levels, 4 levels, and 3 levels. Accuracy of iSV updating was assessed through point-to-point registration error (ppRE) using mini-screw locations, and the average accuracy was 1.26±0.77 mm, 1.54±0.62 mm, and 1.38±0.44 mm, for the three exposure lengths, respectively. The time cost was ~10-15 min and similar in all three exposure sizes. Results indicate that performance of iSV image updating was similar in different lengths of exposure, and the accuracy was within clinically acceptable range (2 mm).

PURPOSE: Current lumbar puncture simulators lack visual feedback of the needle path. We propose a lumbar puncture simulator that introduces a visual virtual reality feedback to enhance the learning experience. This method incorporates virtual reality and a position tracking system. We aim to assess the advantages of the stereoscopy of virtual reality (VR) on needle insertion skills learning. METHODS: We scanned and rendered spine models into three-dimensional (3D) virtual models to be used in the lumbar puncture simulator. The motion of the needle was tracked relative to the spine model in real-time using electromagnetic tracking, which allows accurate replay of the needle insertion path. Using 3D Slicer and SlicerVR, we created a virtual environment with the tracked needle and spine. In this study, 23 medical students performed a traditional lumbar puncture procedure using the augmented simulator. The participants’ insertions were tracked and recorded, allowing them to review their procedure afterwards. Twelve students were randomized into a VR group; they reviewed their procedure in VR, while the Control group reviewed their procedures on computer monitor. Students completed a standard System Usability Survey (SUS) about the system, and a self-reported confidence scale (1-5) in performing lumbar puncture. RESULTS: We integrated VR visual feedback in a traditional lumbar puncture simulator. The VR group gave an average 70.4 on the System Usability Survey (SUS) vs. 66.8 of the Control group. The only negative feedback on VR was that students felt they required technical assistance to set it up (SUS4). The results show a general affinity for VR and its easeof- use. Furthermore, the mean confidence level rose from 1.6 to 3.2 in the VR group, vs. 1.8 to 3.1 in the Control group (1.6 vs. 1.3 improvement). CONCLUSION: The VR-augmented lumbar puncture simulator workflow incorporates visual feedback capabilities and accurate tracking of the needle relative to the spine model. Moreover, VR feedback allow for a more comprehensive spatial awareness of the target anatomy for improved learning.

Purpose. Surgical treatment of spinal deformity often seeks to achieve a given change in spinal curvature for the desired surgical outcome. However, it can be difficult to reliably evaluate changes in spinal curvature in the operating room based on qualitative evaluation or radiographs covering a limited field of view (FOV). We report a prototype beam filtration hardware configuration constructed on the O-arm imaging system and an image reconstruction algorithm for extended view (EV) imaging to enable such clear, long-length visualization of the spine and long surgical constructs. Methods. EV imaging on the O-arm involves a novel multi-slot collimator and longitudinal translation of the gantry. A weighted-backprojection algorithm was developed for EV image reconstruction. Image quality and geometric accuracy was evaluated in simulation and phantom studies to quantitatively characterize the depth resolution and potential sources of geometric distortion. A cadaver study was conducted to verify the quality of visualization in EV images and the potential for measurement of global spinal alignment (GSA) in the operating room. Results. EV imaging provided images spanning up to 65 cm length. Analogous to tomosynthesis, EV image reconstruction provides a modest degree of depth resolution and out-of-plane clutter rejection. The phantom study presenting highcontrast spheres in foam-core exhibited ~11% reduction in signal magnitude at 60 cm from the specified focal plane. The geometric accuracy of EV image reconstructions was high for objects at the focal plane, and distortion outside the focal plane was accurately described by predictions based on the known system geometry and object location. In addition to extending the FOV length by more than a factor of 3, EV images demonstrated strong improvement in visual image quality compared to a plain radiograph, and provided clear visualization of structures necessary for evaluation of GSA– e.g., vertebral endplates at the cervical-thoracic, thoraco-lumbar, and lumbar-sacral junctions. Conclusions. The multi-slot EV imaging technique offers a promising means for intraoperative visualization and assessment of spinal deformity correction through improved visualization over a long FOV and accurate measurement of distance and angles for GSA analysis. Future work involves integration of EV imaging with automated vertebral labeling, GSA analysis, and registration of surgical instrumentation in long surgical constructs.

Navigated bronchoscopy for the lung biopsy using an electro-magnetic (EM) sensor is often inaccurate due to patient breathing movement during procedures. The objective of this study is to evaluate whether registration of neural network- generated depth images can localize the bronchoscope in navigated bronchoscopy negating the need for EM sensor and error caused by breathing motion. [Methods] Dual CNN-generated depth images followed chained ICP registration were validated in the study. Accuracy was measured by the error between the location after registration and the location of the standard electromagnetic sensor. Difference in accuracy between regions that the neural networks had trained on (seen regions) and regions the networks had never encountered (unseen regions) was validated. [Results] The data collected points to the success of the bronchoscopic localization. Overall mean error of accuracy was 8.75 mm and the overall standard deviation was 4.76mm. For the seen region, the mean error was 6.10mm and the standard deviation was 2.65mm. For the unseen region, the mean error was 11.6mm and the standard deviation was 4.87mm. The results of the two-sample t-test shows that there is a statistically significant difference between the unseen and the seen region. [Conclusion] The results for registration demonstrate that this technique has potential to be implemented in navigational bronchoscopy. The technique produced less error than the electromagnetic sensor in practice, especially accounting for the estimated practical error due to experimental setup.

Physical replicas of patient specific heart valve pathologies may improve clinicians’ ability to plan the optimal treatment for patients with complex valvular heart disease. Our previous work has demonstrated the ability to replicate patient pathology of the adult mitral valve (MV) in a dynamic environment [13]. Infant congenital heart defects present possibly the most challenging form of valvular disease, given the range of pathologies, the relative size of these valves compared to adult anatomy, and the rarity of congenital heart disease. Patient specific valve models could be particularly valuable for pediatric cardiologists and surgeons, as a means to both plan for and practice interventions. Our current goal is to assess our ability to apply our workflow to the more challenging case of the tricuspid valve (TV) presented in cases of hypoplastic left heart syndrome (HLHS). We explore the feasibility of adapting our previous workflow for creating dynamic silicone MV models for pre-surgical planning and simulation training, to developing 3D echocardiogram derived, patient specific TV models for use in a physical heart simulator. These models are intended for characterization of the TV, and exploration of the relationship between specific anatomical features and tricuspid regurgitation (TR) severity. The simulations may be relevant to pre-surgical planning of repair of the particularly complex and unique anatomical pathologies presented in children with HLHS.

Respiratory motion management is crucial for maintaining robustness of the delivered dose in radiotherapy. This is particularly relevant for spot-scanned proton therapy, where motion-induced dosimetric “interplay” effects can severely perturb the planned dose distribution. Our proton therapy vendor developed a stereoscopic kV x-ray image guidance platform along with a 3D/2D image matching algorithm for 6 degreeof-freedom patient positioning with a robotic couch (6DOF couch). However, this vendor-provided solution lacks the capability to adequately handle real-time kV fluoroscopy, which is crucial for aspects of motion management. To address this clinical gap, we augmented vendor’s system with a custom signal processing pathway to passively listening for flat-panel detector (FPD) data stream (Camera Link) and handle fluoroscopic frames independently in real time. Additionally, we built a novel calibration phantom and the accompanying room-geometry-specific calibration routine for projective overlay of DICOM-RT structures onto the 2D FPD frames. Because our calibration routine has been developed independently, this tool may also serve as an independent means to test and validate the vendor’s imaging geometry calibration. We developed a Windows-based application in .NET/C# to drive all data acquisition and processing. Having DICOM integration with our treatment planning infrastructure, this therapy tool automatically archives clinical x-ray data to a HIPAA-compliant cloud, and therefore serves as a data interface to retrieve previously recorded x-ray images and cine video streams. This functions as a platform for image guidance research in the future. The next goal on our roadmap is to develop deep-learning methods for real-time soft-tissue-based tumor tracking.

PURPOSE: The segmentation of Computed Tomography (CT) colonography images is important to both colorectal research and diagnosis. This process often relies on manual interaction, and therefore depends on the user. Consequently, there is unavoidable interrater variability. An accurate method which eliminates this variability would be preferable. Current barriers to automated segmentation include discontinuities of the colon, liquid pooling, and that all air will appear the same intensity on the scan. This study proposes an automated approach to segmentation which employs a 3D implementation of U-Net. METHODS: This research is conducted on 76 CT scans. The U-Net comprises an analysis and synthesis path, both with 7 convolutional layers. By nature of the U-Net, output segmentation resolution matches the input resolution of the CT volumes. K-fold cross-validation is applied to ensure no evaluative bias, and accuracy is assessed by the Sørensen-Dice coefficient. Binary cross-entropy is employed as a loss metric. RESULTS: Average network accuracy is 98.81%, with maximum and minimum accuracies of 99.48% and 97.03% respectively. Standard deviation of K accuracies is 0.5%. CONCLUSION: The network performs with considerable accuracy, and can reliably distinguish between colon, small intestine, lungs, and ambient air. A low standard deviation is indicative of high consistency. This method for automatic segmentation could prove a supplemental or alternative tool for threshold-based segmentation. Future studies will include an expanded dataset and a further optimized network.

Breast shape aesthetic is most desired outcome of cosmetic and reconstructive breast surgery. Post-operative (post-op) patient satisfaction largely depends on patient expectations, hence appropriately communicating information about surgical options to patients and moderating patient expectations is critical in surgical planning. Breast modeling and computational simulation can help mitigate this challenge and provide an effective tool for presenting potential surgical outcomes and eliciting patient preferences. Most available computational models lack the ability to provide realistic estimation of breast shape changes. We have previously developed a Fourier spherical harmonics (SPHARM) based computational approach to model breast shape¹. SPHARM modeling results in 1320 coefficients that are effective descriptors of the 3D breast shape. In this study, we develop a framework to transform the SPHARM coefficients of the pre-operative (pre-op) breast to generate an estimation of the post-op breast shape for cosmetic (e.g. implant-based augmentation) and reconstructive (e.g. implant, autologous tissue, etc.) surgery procedures. Least squares optimization was used to realize the transformation between the pre- and post-op SPHARM coefficients. We demonstrate the feasibility of our approach using data from patients who have undergone bilateral implant-based breast reconstruction as part of cancer treatment. We trained a random forest regression² function using SPHARM coefficients and their corresponding shape transformation vectors for 21 preop breasts and validated the regressor using a test dataset of 41 breasts. Our preliminary results demonstrate feasibility of the proposed data-driven approach to model transformation of the pre-op breast to its post-op form for a given surgical procedure.

Transrectal ultrasound (TRUS) fusion-guided biopsy and brachytherapy (BT) offer promising diagnostic and therapeutic improvements to conventional practice for prostate cancer. One key component of these procedures is accurate segmentation of the prostate in three-dimensional (3D) TRUS images to define margins used for accurate targeting and guidance techniques. However, manual prostate segmentation is a time-consuming and difficult process that must be completed by the physician intraoperatively, often while the patient is under sedation (biopsy) or anesthetic (BT). Providing physicians with a quick and accurate prostate segmentation immediately after acquiring a 3D TRUS image could benefit multiple minimally invasive prostate interventional procedures and greatly reduce procedure time. Our solution to this limitation is the development of a convolutional neural network to segment the prostate in 3D TRUS images using multiple commercial ultrasound systems. Training of a modified U-Net was performed on 84 end-fire and 122 side-fire 3D TRUS images acquired during clinical biopsy and BT procedures. Our approach for 3D segmentation involved prediction on 2D radial slices, which were reconstructed into a 3D geometry. Manual contours provided the annotations needed for the training, validation, and testing datasets, with the testing dataset consisting of 20 unseen 3D side-fire images. Pixel map comparisons (Dice similarity coefficient (DSC), recall, and precision) and volume percent difference (VPD) were computed to assess error in the segmentation algorithm. Our algorithm performed with a 93.5% median DSC and 5.89% median VPD with a <0.7 s computation time, offering the possibility for reduced treatment time during prostate interventional procedures.

Minimally invasive focal ablation of liver cancer is an alternative technique to conventional methods for early-stage tumors. Sufficient therapy is provided when ablation applicators are placed at their intended target locations, but current practices are occasionally unable to achieve the required degree of accuracy, as observed by local cancer recurrence rates. We have developed a mechanically assisted 3D ultrasound (US) imaging and guidance system capable of providing geometrically variable images to increase intraoperative spatial information and propose a new method for placing therapeutic applicators during focal liver tumor ablations. A three-motor mechanical mover was designed to provide linear, tilt, and combined hybrid geometries for user-defined 3D US fields-of-view. This mover can manipulate any clinically available 2D US transducer mounted in a transducer-specific 3D-printed holders and is held by a counterbalanced mechanical guidance system, which contains electromagnetic brakes and encoders to track the position of the transducer. Fabrication of a transducer-specific needle guide allowed for information from 3D US images and targets to be overlaid with live 2D US to perform an image-guided workflow. End-to-end testing from 3D US acquisition to needle insertion was performed with a mock phantom procedure to assess overall needle placement accuracy and a potential clinical workflow. Mean applicator placement error was 3.8 ± 1.9 mm for all trials and demonstrated that our 3D US image-guided system may be a feasible approach for guiding ablation applicators accurately during focal liver tumor ablation procedures.

There is an increasing need for automatic and objective surgical skill assessment methods to make the surgeon training process more effective. We studied an automatic skill assessment system based on the surgical instrument tip trajectories extracted from cadaveric trans-nasal endoscopic sinus surgery videos. We proposed a tracking algorithm by combining a segmentation-based instrument tip detector and Kalman filter. For surgical skill assessment, we explored four new motion-related metrics. The proposed method has been tested with 10 surgery videos from 4 experts and 5 trainees and shown its potential for the automatic surgical skill assessment.

Purpose. Initial prototype for an intraoperative slot-scan imaging and registration technique is reported for 3D guidance and confirmation of surgical constructs in spine surgery – specifically, involving placement of pedicle screws in multilevel spinal fusion or deformity correction. The unique projection geometry provided by a multi-slot collimator is exploited to produce parallax views of the radiographic scene for accurate 3D registration from a single scan. Methods. The approach takes advantage of the collimator apertures that form disparate fan-beam views to perform 3D- 2D registration of 3D implant models (pedicle screws) and 2D raw detector slot-scan measurements acquired via linear motion of the gantry. Experiments using a prototype O-arm (Medtronic, Littleton MA) were conducted in a cadaver specimen to evaluate the geometric accuracy of multi-slot registration to that of conventional 3D-2D registration from multiple fluoroscopic views. Results. Cadaver studies showed the multi-slot apertures to provide a sufficient degree of parallax to estimate the 3D location of implants. Spinal pedicle screws were registered from a single slot-scan, with mean target registration error of <1.5 mm, comparable to the margins of errors for optical surgical tracking and dual radiograph (AP + Lat) registration. Conclusions. Presented work demonstrates the feasibility of using a multi-slot collimator to perform 3D pose estimation of the patient and surgical implants in intraoperative images. The method is particularly suitable to evaluating the quality of long surgical constructs, as in spinal deformity correction. Ongoing work includes radiation dosimetry in comparison to conventional radiography and streamlined integration with tools for automatic analysis of global spinal alignment.

Recently, in the United States as well as other countries, a shortage of obstetrician and gynecologist (ob-gyns) has grown seriously. The obstetrics and gynecology have a high burnout rate compared to other medical specialties because of increased workloads and competing for administrative demands. Then, there is a demand for assisting the procedure of prenatal care, especially ultrasonography. Although several robotic-assisted ultrasound imaging platforms have been developed, there were few platforms focusing on prenatal care. In this paper, we proposed an ultrasonography assistance robot for prenatal care to improve the workload of obstetricians and gynecologists. In prenatal care, it is crucially important to satisfy the safety for the pregnant women and fetus compared to other regions of ultrasonography. This paper serves as the proof of concept of the ultrasonography assistance robot for prenatal care by demonstrating the scan of uterus and estimating amniotic fluid volume for assessing fetus health with the fetal US imaging phantom, and clinical feasibility to one pregnant woman. As the key technology to satisfy the safety and acquired image quality, the mechanism with constant springs that the US probe can be shifted flexibly depending on the abdominal height was proposed. The proposed robot system enabled to scan the entire uterus area keeping the contact force under the force applied in clinical procedures (about 15 N) to the fetus phantom. Additionally, as the first application for evaluating fetus health automatically, the system to estimate the amniotic fluid volume (AFV) based on the acquired US images with the robot system was developed and evaluated with the fetus phantom. The result shows estimation errors within 10%. Finally, we demonstrated the robotic US scan to one pregnant woman and successfully observed the body parts of fetus.

Purpose. Conventional model-based 3D-2D registration algorithms can be challenged by limited capture range, model validity, and stringent intraoperative runtime requirements. In this work, a deep convolutional neural network was used to provide robust initialization of a registration algorithm (known-component registration, KC-Reg) for 3D localization of spine surgery implants, combining the speed and global support of data-driven approaches with the previously demonstrated accuracy of model-based registration. Methods. The approach uses a Faster R-CNN architecture to detect and localize a broad variety and orientation of spinal pedicle screws in clinical images. Training data were generated using projections from 17 clinical cone-beam CT scans and a library of screw models to simulate implants. Network output was processed to provide screw count and 2D poses. The network was tested on two test datasets of 2,000 images, each depicting real anatomy and realistic spine surgery instrumentation – one dataset involving the same patient data as in the training set (but with different screws, poses, image noise, and affine transformations) and one dataset with five patients unseen in the test data. Assessment of device detection was quantified in terms of accuracy and specificity, and localization accuracy was evaluated in terms of intersection-overunion (IOU) and distance between true and predicted bounding box coordinates. Results. The overall accuracy of pedicle screw detection was ~86.6% (85.3% for the same-patient dataset and 87.8% for the many-patient dataset), suggesting that the screw detection network performed reasonably well irrespective of disparate, complex anatomical backgrounds. The precision of screw detection was ~92.6% (95.0% and 90.2% for the respective same-patient and many-patient datasets). The accuracy of screw localization was within 1.5 mm (median difference of bounding box coordinates), and median IOU exceeded 0.85. For purposes of initializing a 3D-2D registration algorithm, the accuracy was observed to be well within the typical capture range of KC-Reg.¹ Conclusions. Initial evaluation of network performance indicates sufficient accuracy to integrate with algorithms for implant registration, guidance, and verification in spine surgery. Such capability is of potential use in surgical navigation, robotic assistance, and data-intensive analysis of implant placement in large retrospective datasets. Future work includes correspondence of multiple views, 3D localization, screw classification, and expansion of the training dataset to a broader variety of anatomical sites, number of screws, and types of implants.

PURPOSE: Endoscopic vessel harvesting is the preferred minimally invasive approach to obtain grafts for coronary bypass surgeries, however it requires extensive practice to minimize vessel damage. We propose to create a surgical training simulation with visual and haptic feedback. In this study, we focus on analyzing the force and torque peaks on the surgical retractor during the procedure. METHODS: The original retractor handle was 3D scanned and modified to attach an ATI Mini40 force-torque transducer. The forces and torques in two radial artery and two saphenous vein procedures in human cadavers were recorded. The measurements, endoscopic video and surgical surface video were collected. The median and interquartile range of the force and torque peaks were calculated for the artery and vein harvesting procedures. RESULTS: The median and interquartile range for saphenous vein harvests was larger than radial artery harvests. The largest median force and torque generated in the vein was 11.654 N [posterior] and 0.661 Nm [- frontal], whereas in the artery was 6.163 N [anterior] and 0.381 Nm [+ frontal], respectively. CONCLUSION: The distribution of force and torque peaks in the retractor was found for endoscopic vessel harvests. This data can be used to design a haptic user interface, and to establish expert benchmarks for learning curve evaluation.

Medical image segmentation is a complex and critical step in the field of medical image processing and analysis. Manual annotation of the medical image requires a lot of effort by professionals, which is a subjective task. In recent years, researchers have proposed a number of models for automatic medical image segmentation. In this paper, we formulate the medical image segmentation problem as a Markov Decision Process (MDP) and optimize it by reinforcement learning method. The proposed medical image segmentation method mimics a professional delineating the foreground of medical images in a multi-step manner. The proposed model get notable accuracy compared to popular methods on prostate MR data sets. Meanwhile, we adopted a deep reinforcement learning (DRL) algorithm called deep deterministic policy gradient (DDPG) to learn the segmentation model, which provides an insight on medical image segmentation problem.

Lung cancer continues to be the most common type of cancer worldwide. In radiation therapy, high doses of radiation are used to destroy tumors. Adapting radiotherapy to breathing patterns has always been a major concern when dealing with tumors in thoracic or upper abdomen regions. Precise estimation of respiratory signal ensures least damage to healthy tissues surrounding the tumor as well as misrepresentation of the target location. The main objective of this work is to develop a method to extract the breathing signal directly from a given sequence of cone-beam computed tomography (CBCT) projections without depending on any external devices such as spirometer, pressure belt, or implanted infrared markers. The proposed method implements optical flow to track the movement of pixels between each pair of successive CBCT projection images through the entire set of projections. As the optical flow operation results in a high dimensional dataset, dimensionality reduction using linear and kernel based principal component analysis (PCA) are applied on the optical flow dataset to transform it into a lower-dimensional dataset ensuring that only the most distinctive components are present. The proposed method was tested on XCAT phantom datasets1 simulating cases of regular and irregular breathing patterns and cases where the diaphragm was partially visible in certain projection images. The extracted breathing signal using the proposed method was compared to the ground truth signal. Results showed that the extracted signal correlated well with ground truth signal with a mean phase shift not exceeding 1.5 projection in all cases.

Here we report on a phantom targeting study for accuracy evaluation of our body-mounted robot for Magnetic Resonance Imaging (MRI) guided arthrography. We use a standardized method developed in a multi-institute effort with the aim of providing an objective method for accuracy and signal-to-noise Ratio (SNR) evaluation of MRI-compatible robots. The medical definition of arthrography is the radiographic visualization of a joint (as the hip or shoulder) after the injection of a radiopaque substance. That procedure provides an evaluation of the joints using two medical imaging modalities, fluoroscopic x-ray imaging and MRI. Conventional arthrography is done in two stages: first the contrast dye injected into the joint (fluoroscopic procedure) and then an MRI to evaluate the joint space. Our MRI-guided compatible robot is intended to enable needle placement in the MRI environment, streamlining the procedure. The targeting study was conducted using the quality assessment mockup phantom and associated software called QARAI that was developed by the URobotics Laboratory at Johns Hopkins and colleagues. The mockup contains four embedded fiducials and an 8 by 8 grid which is used to automatically identify the targeting points with high accuracy. The study was conducted on a Philips Achieva 1.5T MRI system and 10 points were targeted. All targets were reached with an average error of 2.71mm. The targeting algorithm, as well as the control of the robot, were completed using robot control modules developed with the open source software 3D Slicer.

Cochlear implants (CI) use an array of electrodes surgically threaded into the cochlea to restore hearing sensation. Techniques for predicting the insertion depth of the array into the cochlea could guide surgeons towards more optimal placement of the array in order to reduce trauma and preserve the residual hearing of the patient. In addition to the electrode array geometry (length and diameter), both the base insertion depth (BID) and the cochlear scale impact the overall array insertion depth. In this paper, we investigated the influence of these parameters on overall insertion depth with the purpose of developing a model which can make preoperative predictions of insertion depth of lateral wall cochlear implant electrode arrays. CT images of 86 lateral wall positioned straight electrode array CI recipients were analyzed. Using previously developed automated algorithms, relative electrode position inside the cochlea as well as the cochlea scale was measured from the CT images. A linear regression model is proposed for insertion depth prediction based on cochlea size, array geometry, and BID. The model is able to accurately predict angular insertion depths with standard deviation of 41 degrees. Surgeons may use this model for patient-customized selection of the electrode array and/or to plan a base insertion depth for a given array that minimizes the likelihood of causing trauma to regions of the cochlea where residual hearing exists.

Cochlear implants (CIs) use an array of electrodes implanted in the cochlea to directly stimulate the auditory nerve. After surgery, CI recipients undergo many programming sessions with an audiologist who adjusts CI processor settings to improve performance. However, few tools exist to help audiologists know what settings will lead to better performance. In this paper, we propose a new method to assist audiologists by determining a customized firing order of the electrodes on the array using image-based models of patient specific neural stimulation patterns. Our models permit estimating the time delay needed after firing an electrode so that the nerve fibers they stimulate can recover from the refractory period. These predictions allow us to design an optimization algorithm that determines a customized electrode firing order that minimizes negative effects of overlapping stimulation between electrodes. The customized order reduces how often nerves that are in a refractory state from previous stimulation by one electrode are targeted for activation by a subsequent electrode in the sequence. Our experiments show that this method is able to reduce the theoretical stimulation overlap artifacts and could lead to improved hearing outcomes for CI recipients.

Kidney biopsies are currently performed using preoperative imaging to identify the lesion of interest and intraoperative imaging used to guide the biopsy needle to the tissue of interest. Often, these are not the same modalities forcing the physician to perform a mental cross-modality fusion of the preoperative and intraoperative scans. This limits the accuracy and reproducibility of the biopsy procedure. In this study, we developed an augmented reality system to display holographic representations of lesions superimposed on a phantom. This system allows the integration of preoperative CT scans with intraoperative ultrasound scans to better determine the lesion’s real-time location. An automated deformable registration algorithm was used to increase the accuracy of the holographic lesion locations, and a magnetic tracking system was developed to provide guidance for the biopsy procedure. Our method achieved a targeting accuracy of 2.9 ± 1.5 mm in a renal phantom study.

Tumor phantoms (TP) have been described for the purposes of training surgical residents and further understanding tissue characteristics in malignancy. To date, there has not been a tumor phantom described for the purposes of research and training in oncologic surgery of the head and neck focusing on the larynx and pharynx. With the goal of providing radiographic, visual, and physical mimicry of head and neck squamous cell carcinoma (HNSCC), a phantom was developed as a proposed training and research tool for trans-oral surgical procedures such as transoral laser microsurgery (TLM) and transoral robotic surgery (TORS). TP’s were constructed with an agar-gelatin chicken stock base to approximate reported physical properties, then glutaraldehyde and Omnipaque-350 were used as a fixative and to enhance CT-visualization respectively. Further, to ensure heterogeneity in radiographic imaging, other materials like olive oil and condensed milk were explored. These ingredients were combined with the use of a novel, 3D printed, syringe adaptor designed to allow for the direct injection of the liquid tumor into model tissue. TP’s fixed quickly in vivo upon implantation and were imaged using CT and segmented. This injection-based model was piloted in bovine tissue and verified in porcine tissue with excess Omnipaque-350 for volumetric reliability then optimized utilizing 6 well plates. Following radiographic optimization, the viscoelastic properties of TP’s were measured through uniaxial compression. We observed a Young’s modulus similar to published literature values and consistent reproducibility. Most notably, our proposed TP can be used by multiple specialties by altering the color and concentration of agar in the base solution to approximate physical properties.

Cine cardiac magnetic resonance imaging (CMRI), the current gold standard for cardiac function analysis, provides images with high spatio-temporal resolution. Computing clinical cardiac parameters like ventricular blood-pool volumes, ejection fraction and myocardial mass from these high resolution images is an important step in cardiac disease diagnosis, therapy planning and monitoring cardiac health. An accurate segmentation of left ventricle blood-pool, myocardium and right ventricle blood-pool is crucial for computing these clinical cardiac parameters. U-Net inspired models are the current state-of-the-art for medical image segmentation. SegAN, a novel adversarial network architecture with multi-scale loss function, has shown superior segmentation performance over U-Net models with single-scale loss function. In this paper, we compare the performance of stand-alone U-Net models and U-Net models in SegAN framework for segmentation of left ventricle blood-pool, myocardium and right ventricle blood-pool from the 2017 ACDC segmentation challenge dataset. The mean Dice scores achieved by training U-Net models was on the order of 89.03%, 89.32% and 88.71% for left ventricle blood-pool, myocardium and right ventricle blood-pool, respectively. The mean Dice scores achieved by training the U-Net models in SegAN framework are 91.31%, 88.68% and 90.93% for left ventricle blood-pool, myocardium and right ventricle blood-pool, respectively.

Tooth decay is one of the most common chronic infectious diseases worldwide. Bacteria from the oral biofilm create a local acidic environment that demineralizes the enamel in the caries disease process. By optically imaging plaque pH in pits and fissures and contacting surfaces of teeth, then medicinal therapies can be accurately applied to prevent or monitor the reversal of caries. To achieve this goal, the fluorescence emission from an aqueous solution of sodium fluorescein was measured using a multimodal scanning fiber endoscope (mmSFE). The 1.6-millimeter diameter mmSFE scans 424nm laser light and collects wide-field reflectance for navigational purposes in grayscale at 30 Hz. Two fluorescence channels centered at 520 and 549 nm are acquired and ratiometric analysis produces a pseudo-color overlay of pH. In vitro measurements calibrate the pH heat maps in the range 4.7 to 7.2 pH (0.2 standard deviation). In vivo measurements of a single case study provides informative images of interproximal biofilm before and after a sugar rinse. Post processing a time series of images provides a method that calculates the average pH changes of oral biofilm, replicating the Stephan Curve. These spatio-temporal records of oral biofilm pH can provide a new method of assessing the risk of tooth decay, guide the application of preventative therapies, and provide a quantitative monitor of overall oral health. The non-contact in vivo optical imaging of pH may be extended to measurements of wound healing, tumor environment, and other food processing surfaces since it relies on low power laser light and a US FDA approved dye.

Dynamic contrast-enhanced (DCE) MRI is increasingly used to stage and evaluate rectal cancer extent in vivo in order to plan and target interventions for locally advanced tumors. The major clinical challenge faced with rectal cancers today is to personalize interventions through early identification of patients will benefit from neoadjuvant chemoradiation (nCRT) alone and who will benefit from aggressive surgery (with adjuvant radiation) instead; via baseline imaging. In this study, we evaluated texture kinetic features of rectal tumors using baseline DCE MRI scans, in order to predict pathologic tumor stage regression in response to nCRT. Our texture kinetics approach utilized a combination of texture features (from multiple DCE uptake phases) and polynomial curve fitting to uniquely quantify spatiotemporal patterns of lesion texture during contrast uptake and diffusion that were different between responders and non-responders to nCRT. We utilized a cohort of 48 rectal cancer patients for whom pre-nCRT 3 T DCE MRI was available, including pre-, early-, and delayed-enhancement phases. All DCE MRI phases were processed for motion and spatial alignment artifacts via rigid co-registration, and the tumor ROI on all 3 contrast phases was normalized with respect to non-enhancing muscle. 191 texture features were extracted from each of 3 contrast phases separately, following which each feature was plotted with respect to time to yield a feature enhancement curve. Polynomial fitting was applied to each feature enhancement curve to result in a vector of coefficients which was considered the texture kinetic representation of that feature. All 191 features were evaluated in terms of their texture kinetic representation as well as the raw feature enhancement, for predicting pathologically regressed tumor stages (ypT0-2) from non-regressed tumors (ypT3-4) via a cross-validated QDA classifier. Texture kinetics of gradient XY enhancement yielded the best overall AUC=0:762±0:053, which was significantly higher than any feature enhancement representation (best AUC=0:696±0:050). Texture kinetic representations also outperformed their corresponding raw feature enhancement representations in 54.5% of the features compared, and performed significantly worse in only 13% of the comparisons. Non-invasive guidance of interventions in rectal cancers could therefore be enhanced through the use of texture kinetic features from DCE MRI, which may better characterize spatiotemporal differences between responders and non-responders on baseline imaging.

The human organism is a highly complex system that is prone to various diseases. Some diseases are more dangerous than others, especially those that affect the circulatory system or the aorta in particular. The aorta is the largest artery in the human body. Its wall comprises several layers. When the intima, i.e. the innermost layer of the aortic wall, tears, blood enters and propagates between the layers causing them to separate. This is known as aortic dissection (AD). Without immediate treatment, an AD may kill 33% of patients within the first 24 hours, 50% of patients within 48 hours, and 75% of patients within 2 weeks. However, proper treatment is still subject to research and active discussion. By providing a deeper understanding of aortic dissections, this work aims to contribute to the continuous improvement of AD diagnosis and treatment by presenting AD in a new, immersive visual experience: Virtual Reality (VR). The visualization is based on Computed Tomography (CT) scans of human patients suffering from an AD. Given a scan, relevant visual information is segmented, refined and put into a 3D scene. Further enhanced by blood flow simulation and VR user interaction, the visualization helps in better understanding AD. The current implementation serves as a prototype and is considered to be extended by minimizing user interaction when new CT scans are loaded into VR (i) and by providing an interface to feed the visualization with simulation data provided by mathematical models (ii).

The use of percutaneous and bronchoscopic microwave ablation to treat both primary and secondary lung tumors has been growing recently. These ablation systems are typically accompanied by an ablation planning system to optimize the treatment outcome by ensuring adequate margin in the expected ablation zone during the planning phase. The planning system utilizes pre-operative CT scan to identify the tumor and recommend microwave probe position. Radiomics is a process of converting medical images into higher-dimensional data and subsequent mining of data to reveal underlying pathophysiology for enhancing clinical decision support making. Radiomics analysis have shown promises in capturing distinct tumor characteristics and predicting prognosis of the tumor. Here, we present a new method to predict microwave ablation zones by supplementing a bioheat transfer model of microwave tissue ablation with microwave sensitive radiomics features. We hypothesize that supplementing traditional bioheat transfer modeling with microwave sensitive radiomics features will generate a more accurate and personalized ablation prediction that will lead to better treatment outcome. Inputs to the bioheat transfer modeling approach include the geometry of the target tumor, physical characteristics of the tissue, and dimensions of the microwave ablation applicator. The radiomics algorithm extracts characteristics of the targeted tumor’s size and shape, as well as texture characteristics, from pre-operative CT images. We employed cascaded segmentation based on RetinaNet and U-Net to obtain a tumor’s size and shape. Then, a segmented tumor is employed for texture analysis through a set of regression convolutional neural networks. These tumor characteristics are employed as radiomics features for more accurate dose prediction and margin for microwave ablation treatment. We present the preliminary results of a study using images from clinical lung tumor cases to predict ablation treatment outcome, with patient-specific tissue biophysical properties based on radiomics features.

Medical image processing and analysis operations, particularly segmentation, can benefit a great deal from prior information encoded to capture variations over a population in form, shape, anatomic layout, and image appearance of objects. Model/atlas-based methods are extant in medical image segmentation. Although multi-atlas/ multi-model methods have shown improved accuracy for image segmentation, if the atlases/models do not cover representatively the distinct groups, then the methods may not be generalizable to new populations. In a previous study, we have given an answer to address the following problem at image level: How many models/ atlases are needed for optimally encoding prior information to address the differing body habitus factor in a population? However, the number of models for different objects may be different, and at the image level, it may not be possible to infer the number of models needed for each object. So, the modified question to which we are now seeking an answer to in this paper is: How many models/ atlases are needed for optimally encoding prior information to address the differing body habitus factor for each object in a body region? To answer this question, we modified our method in the previous study for seeking the optimum grouping for a given population of images but focusing on the individual objects. We present our results on head and neck computed tomography (CT) scans of 298 patients.

Publisher’s Note: This paper, originally published on 16 March 2020, was replaced with a corrected/revised version on 28 April, 2020. If you downloaded the original PDF but are unable to access the revision, please contact SPIE Digital Library Customer Service for assistance.

Publisher’s Note: This paper, originally published on 16 March 2020, was replaced with a corrected/revised version on 28 December 2020. If you downloaded the original PDF but are unable to access the revision, please contact SPIE Digital Library Customer Service for assistance.