We develop a novel strategy to generate synthetic tumors. Unlike existing works, the tumors generated by our strategy have two intriguing advantages: (1) realistic in shape and texture, which even medical professionals can confuse with real tumors; (2) effective for training AI models, which can perform liver tumor segmentation similarly to the model trained on real tumors—this result is unprecedented because no existing work, using synthetic tumors only, has thus far reached a similar or even close performance to the model trained on real tumors. This result also implies that manual efforts for developing per-voxel annotation of tumors (which took years to create) can be considerably reduced for training AI models in the future. Moreover, our synthetic tumors have the potential to improve the success rate of small tumor detection by automatically generating enormous examples of small (or tiny) synthetic tumors.

Automated analysis of data with arbitrarity in intensity, protocols, and field-of-view is an important task in the modern biomedical imaging pipeline. Recently, spiking computation has been leveraged to reduce the computational overhead of neural networks in the domain of medicine. However, state-of-the-art methods are on the trend of energy for accuracy, a lack of scalability due to the non-differentiable activations of spiking neurons, shallow architectures, and unsophisticated tasks. To avoid native spiking neural network (SNN) design, a pipeline for end-to-end vertebrae segmentation, identification, and localization from Computed Tomography acquisitions is denoted wherein the learned parameters of deep convolutional analog networks are transferred to equivalent-accurate spiking ones. We report that complex architectures such as autoencoders can be run as rate-based SNNs with a major reduction in latency to achieve error rates close to conventional deep methods. The networks are also able to forego normalization to a common space, making our approach ideal in an intraoperative context and for longitudinal neuroanatomical studies. To the best of our knowledge, this paper is the first to study biologically-inspired clinical systems on such massive heterogeneous data.

Deep learning techniques have greatly benefited computer-aided diagnostic systems. However, unlike in other fields, in medical imaging, acquiring large fine-grained annotated datasets such as 3D tumour segmentation is challenging due to the high cost of manual annotation and privacy regulations. This has given interest to weakly-supervise methods to utilize the weakly labelled data for tumour segmentation. In this work, we propose a weakly supervised approach to obtaining regions of interest using binary class labels. Furthermore, we propose a novel objective function to train the generator model based on a pretrained binary classification model. Finally, we apply our method to the brain tumour segmentation problem in MRI.

Source-free unsupervised domain adaptations (SFUDA) have been a predominant solution for transferring knowledge inherent in the model parameters trained with a privately labeled source domain to apply to an unlabeled target domain. In the case of missing source domain labeled data training, unfortunately, the conventional SFUDA approaches can be easily caught in the pitfall of "winner takes all", i.e., the majority class dominates the predictions of the deep segmentation model in a class-imbalanced task while the minority classes are overlooked. In this work, we provide a complementary self-training (CST) approach for SFUDA segmentation to get over these challenges, since it can be much easier to exclude certain classes with low probabilities than to predict the correct one. Specifically, we resort to the complementary pseudo-label, which can be easier to learn and able to keep low noise level. Its superior performance has been evidenced in a CT-to-MR cardiac anatomical segmentation task with throughout quantitative evaluation.

Recent work has investigated the distributions of learned convolution filters through a large-scale study containing hundreds of heterogeneous image models. Surprisingly, on average, the distributions only show minor drifts in comparisons of various studied dimensions including the learned task, image domain, or dataset. However, among the studied image domains, medical imaging models appeared to show significant outliers through "spikey" distributions, and, therefore, learn clusters of highly specific filters different from other domains. Following this observation, we study the collected medical imaging models in more detail. We show that instead of fundamental differences, the outliers are due to specific processing in some architectures. Quite the contrary, for standardized architectures, we find that models trained on medical data do not significantly differ in their filter distributions from similar architectures trained on data from other domains. Our conclusions reinforce previous hypotheses stating that pre-training of imaging models can be done with any kind of diverse image data.

Normative modelling is an emerging method for understanding the underlying heterogeneity within brain disorders like Alzheimer Disease (AD) by quantifying how each patient deviates from the expected normative pattern that has been learned from a healthy control distribution. Existing deep learning based normative models on magnetic resonance imaging (MRI) neuroimaging data use unimodal autoencoders with a single encoder and decoder that may fail to capture the relationship between brain measurements extracted from different MRI modalities. In this work, we propose multi-modal variational autoencoder (mmVAE) based normative modelling framework that can capture the joint distribution between different modalities and apply it for normative modeling. The deviation maps generated by our proposed multimodal model (mmVAE) are more sensitive to disease staging within AD, have a better correlation with patient cognition and higher number of brain regions with statistically significant deviations compared to a unimodal baseline model with all modalities concatenated as a single input.

The amount of manually labeled data is limited in medical applications, so semi-supervised learning and automatic labeling strategies can be an asset for training deep neural networks. However, the quality of the automatically generated labels can be uneven and inferior to manual labels. This paper suggests a semi-supervised training strategy for leveraging both manually labeled data and extra unlabeled data. In contrast to the existing approaches, we apply a robust loss for the automated labeled data to compensate for the uneven data quality automatically. First, we generate pseudo-labels for unlabeled data using a model pre-trained on labeled data. These pseudo-labels are noisy, and using them along with labeled data for training can severely degrade learned feature representations and the generalization of the model. Here we mitigate the effect of these pseudo-labels by using a robust loss, Beta Cross-Entropy. We show that our proposed strategy improves the model performance by penalizing the labels with a lower likelihood in a segmentation application.

While deep learning models are widely used in medical image segmentation, they are typically not designed to generalize to unseen segmentation tasks involving new anatomies, image modalities, or labels. Generally, given a new segmentation task, researchers will design and train a new model or fine-tune existing models. This is time-consuming, even for machine learning researchers, and poses a substantial barrier for clinical researchers, who often lack the resources or expertise to train new models. In this paper, we present a model that can solve new unseen medical segmentation tasks in a single forward pass at inference without retraining or fine-tuning. Our task-amortization model, UniverSeg, can segment a wide range of datasets as well as generalize to new ones. A UniverSeg network takes as input the target image to be segmented and a small set of example images and label maps representing the desired task and outputs a segmentation map. We train the proposed model on a large collection of over 85 medical imaging datasets with varying anatomies and modalities. This encourages the model to be task-agnostic and instead learn to transfer the relevant information from the example set to the target image, enabling segmentation even in tasks unseen during training. In preliminary experiments, we find that using only one trained UniverSeg model to segment previously unseen tasks can achieve performance close to that of models specifically trained on those new tasks.

Computed tomography (CT) is of great importance in clinical practice, but its potential radiation risk is raising people’s concerns. Deep learning-based methods are considered promising in CT reconstruction, but these methods are usually trained with the measured data obtained from specific scanning protocol and need to centralizedly collect large amounts of data, which will lead to serious data domain shift, and privacy concerns. To relieve these problems, in this paper, we propose a hypernetwork-based federated learning method for personalized CT imaging, dubbed as HyperFed. The basic assumption of HyperFed is that the optimization problem for each institution can be divided into two parts: the local data adaption problem and the global CT imaging problem, which are implemented by an institution-specific hypernetwork and a global-sharing imaging network, respectively. The purpose of global-sharing imaging network is to learn stable and effective common features from different institutions. The institution-specific hypernetwork is carefully designed to obtain hyperparameters to condition the global-sharing imaging network for personalized local CT reconstruction. Experiments show that HyperFed achieves competitive performance in CT reconstruction compared with several other state-of-the-art methods.

Identifying Alzheimer's disease (AD) involves a deliberate diagnostic process owing to its innate traits of irreversibility with subtle and gradual progression, hampering the AD biomarker identification from structural brain imaging (e.g., structural MRI) scans. We propose a novel deep-learning approach through eXplainable AD Likelihood Map Estimation (XADLiME) for AD progression modeling over 3D sMRIs using clinically-guided prototype learning. Specifically, we establish a set of topologically-aware prototypes onto the clusters of latent clinical features, uncovering an AD spectrum manifold. We then measure the similarities between latent clinical features and these prototypes to infer a "pseudo" AD likelihood map. Considering this pseudo map as an enriched reference, we employ an inferring network to estimate the AD likelihood map over a 3D sMRI scan. We further promote the explainability of such a likelihood map by revealing a comprehensible overview from two perspectives: clinical and morphological.

Generative modeling of 3D brain MRIs presents difficulties in achieving high visual fidelity while ensuring sufficient coverage of the data distribution. In this work, we propose to address this challenge with composable, multiscale morphological transformations in a variational autoencoder (VAE) framework. These transformations are applied to a chosen reference brain image to generate MRI volumes, equipping the model with strong anatomical inductive biases. We show substantial performance improvements in FID while retaining comparable, or superior, reconstruction quality compared to prior work based on VAEs and generative adversarial networks (GANs).

Neural networks have proven to be very effective for skin lesion classification, performing equal to, or better than, dermatologists in controlled settings. Recently, various approaches have also been presented to augment neural networks with uncertainty metrics, which improves their usefulness as decision support tool in a clinical practice. However, it is unclear how these techniques match the diagnostic process of a dermatologist. Therefore, we have set up a survey, to compare the diagnosis and uncertainty of a neural network with dermatologists. We found that the neural network and the dermatologists performed similarly. They were however, unsure about different lesions, suggesting they might be complimentary.

Many important problems in medical imaging require analysing the causal effect of genetic, environmental, or lifestyle factors on the normal and pathological variation of anatomical phenotypes. There is, however, a lack of computational tooling to enable causal reasoning about morphological variations of 3D surface meshes. To tackle this problem, we present the framework of deep structural causal shape models (CSMs) using a database of subcortical brain meshes. CSMs enable subject-specific prognoses through counterfactual mesh generation, by utilising high-quality mesh generation techniques, from geometric deep learning, within the expressive framework of deep structural causal models (DSCM).

The success of deep learning relies heavily on the availability of large annotated datasets, but neither sizable data nor annotation is easily accessible for novel diseases. This paper uses the classification of COVID-19 in late 2019 as an example to demonstrate the effectiveness of a novel strategy, named "Label-Assemble". To facilitate the diagnosis of novel diseases, we propose to assemble existing labels from public datasets. Although novel diseases are not in the existing labels, we discover that learning from alternative labels can dramatically improve the diagnosis of the novel disease as these labels can better define the classification boundary of the novel disease. This discovery has the potential to accelerate the development circle of computer-aided diagnosis of novel diseases, in which positive label is hard to collect, yet negative labels are usually available and relatively easier to assemble. Label-Assemble achieves 99.3% accuracy on the COVIDx-CXR2 dataset, which significantly exceeds the previous state of the art (96.3% accuracy) and only uses 3% of the annotated COVID-19 images. We further investigate the implementation of the assembling strategy, showing that assembling pathologically related labels, supplemented by semi-supervised learning, is preferred.

Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labeling instructions are key. Despite the importance of these instructions, their optimization remains largely unexplored. Here, we present the first systematic study of labeling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the MICCAI Society, we uncovered a discrepancy between annotators’ needs for labeling instructions and their current quality and availability. Based on an analysis of 14,040 images annotated by 156 annotators from four professional companies and 708 Amazon Mechanical Turk (MTurk) crowdworkers using instructions with different information density levels, we further found that including exemplary images significantly boosts annotation performance compared to text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform MTurk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labeling instructions.

In this extended abstract, we summarize our recently published work on CNN textural bias in the context of MRI k-space artifacts, namely Gibbs, spike, and wraparound artifacts. We illustrated how carefully simulating artifacts at training time can help reduce textural bias, and consequently lead to CNN models that are more robust to acquisition noise as well as out-of-distribution inference, including data from previously unseen hospitals. We also introduced Gibbs ResUnet; an end-to-end framework that automatically finds optimal combinations of Gibbs artifacts and segmentation model weights. The work was carried out on multimodal and multi-institutional clinical MRI data obtained retrospectively from the Medical Segmentation Decathlon (n=750) and The Cancer Imaging Archive (n=243).

Early detection of cardiac dysfunction through routine screening is vital for diagnosing cardiovascular diseases. An important metric of cardiac function is the left ventricular ejection fraction (EF), which is used to diagnose cardiomyopathy. Echocardiography is a popular diagnostic tool in cardiology, with ultrasound being a low-cost, real-time, and non-ionizing technology. However, human assessment of echocardiograms for calculating EF is both time-consuming and expertise-demanding, raising the need for an automated approach. Earlier automated works have been limited to still images or use echocardiogram videos with spatio-temporal convolutions in a complex pipeline. In this work, we propose to generate images from readily available echocardiogram videos, each image mimicking a M(otion)-mode image from a different scan line through time. We then combine different M-mode images using off-the-shelf model architectures to estimate the EF and, thus, diagnose cardiomyopathy. Our experiments show that our proposed method converges with only ten modes and is comparable to the baseline method while bypassing its cumbersome training process.Keywords: Echocardiography, M-mode Ultrasound, Ejection Fraction

Compression is essential to storing and transmitting medical videos, but the effect of compression on downstream medical tasks is often ignored. Furthermore, systems in practice rely on standard video codecs, which naively allocate bits between medically relevant frames or parts of frames. In this work, we present an empirical study of some deficiencies of classical codecs on gastroenterology videos, and motivate our ongoing work to train a learned compression model for colonoscopy videos. We show that two of the most common classical codecs, H264 and HEVC, compress medically relevant frames statistically significantly worse than medically nonrelevant ones, and that polyp detector performance degrades rapidly as compression increases. We explain how a learned compressor could allocate bits to important regions and allow detection performance to degrade more gracefully. Many of our proposed techniques generalize to medical video domains beyond gastroenterology.

Imaging the electrical activity of the heart can be achieved with invasive catheterisation. However, the resulting data are sparse and noisy. Mathematical modelling of cardiac electrophysiology can help the analysis but solving the associated mathematical systems can become unfeasible. It is often computationally demanding, for instance when solving for different patient conditions. We present a new framework to model the dynamics of cardiac electrophysiology at lower cost. It is based on the integration of a low-fidelity physical model and a learning component implemented here via neural networks. The latter acts as a complement to the physical part, and handles all quantities and dynamics that the simplified physical model neglects. We demonstrate that this framework allows us to reproduce the complex dynamics of the transmembrane potential and to correctly identify the relevant physical parameters, even when only partial measurements are available. This combined model-based and data-driven approach could improve cardiac electrophysiological imaging and provide predictive tools.

Pneumothorax is a potentially life-threatening condition that can be rapidly and accurately assessed via the lung sliding artefact generated using lung ultrasound (LUS). The presence of lung sliding rules out a diagnosis of pneumothorax, while its presence warrants further investigation. We present a method to distinguish present from absent lung sliding that includes conversion of B-mode LUS videos to M-mode images and subsequent prediction using a 2D convolutional neural network. The classifier achieved mean sensitivity for absent lung sliding of 0.935 (SD 0.0034) and 0.832 (SD 0.039) on ten-fold cross validation respectively. On a holdout set consisting of separate patients, the classifier achieved 0.935 sensitivity and 0.875 specificity. The results add to the growing evidence that deep computer vision methods are useful in establishing automated LUS interpretation.

Annotating large medical imaging datasets is an arduous and expensive task, especially when distinct labels must be applied to disjoint subsets of a dataset. When collecting lung ultrasound (LUS) data, the transducer probe is placed in one of two locations on the chest resulting in clips from two distinct views. Each of these views interrogates different anatomic areas of the lungs and must be annotated for separate and distinct pathological features. In this work, we propose a method that exploits this implicit hierarchical organization to optimize annotator efficiency. Specifically, we trained a machine learning model to accurately distinguish between LUS views using an annotated training set of 2908 clips. In a downstream expository view-specific annotation task, we found that automatically partitioning a 780-clip dataset by view saved 42 minutes of manual annotation time and resulted in 55 ± 6 additional relevant labels per hour. We propose that this method can be applied to other hierarchical annotation schemes.

We present a physically-primed deep-neural-network (DNN) architecture for undersampled MRI reconstruction. Our architecture encodes the undersampling mask in addition to the observed data. It employs an appropriate training approach that uses undersampling mask augmentation to encourage the model to generalize the undersampled MRI reconstruction problem. We demonstrated an enhanced generalization capacity which resulted in significantly improved robustness against variations in the acquisition process and the anatomical distribution, especially in pathological regions.

Automated region of interest detection in histopathological image analysis is a challenging and important topic with tremendous potential impact on clinical practice. The deep-learning methods used in computational pathology help us to reduce costs and increase the speed and accuracy of regions of interest detection and cancer diagnosis. In this work, we propose a patch-based region of interest detection method for melanocytic skin tumor whole-slide images. We work with a dataset that contains 165 primary melanomas and nevi Hematoxylin and Eosin whole-slide images and build a deep-learning method. The proposed method performs well on a hold-out test data set including five TCGA-SKCM slides (accuracy of 93.94% in slide classification task and intersection over union rate of 41.27% in the region of interest detection task), showing the outstanding performance of our model on melanocytic skin tumor. Even though we test the experiments on the skin tumor dataset, our work could also be extended to other medical image detection problems, such as various tumors' classification and prediction, to help and benefit the clinical evaluation and diagnosis of different tumors.

Flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. A large international expert consortium now created Metrics Reloaded, a comprehensive framework guiding researchers towards problem-aware metric selection. The framework is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects relevant for metric selection, from the domain interest to properties of the target structure(s), data set and algorithm output. It supports image-level classification, object detection, semantic and instance segmentation tasks. Users are guided through the process of selecting and applying appropriate validation metrics while being made aware of pitfalls. To improve the user experience, we implemented the framework in an online tool, which also provides a common point of access to explore metric weaknesses and strengths. An instantiation of the framework for various biomedical image analysis use cases demonstrates its broad applicability across domains.

Classification of large and dense networks based on topology is very difficult due to the computational challenges of extracting meaningful topological features from real-world networks. In this paper we present a computationally tractable approach to topological classification of networks by using principled theory from persistent homology and optimal transport to define a novel vector representation for topological features. The proposed vector space is based on the Wasserstein distance between persistence barcodes. The 1-skeleton of the network graph is employed to obtain 1D persistence barcodes that represent connected components and cycles. These barcodes and the corresponding Wasserstein distance can be computed very efficiently. The effectiveness of the proposed vector space is demonstrated using support vector machines to classify brain networks.

Contrastive self-supervised learning has emerged in the field of digital pathology, which leverages unlabeled data for learning domain invariant representations in pathology images. However, the downstream models, trained using these representations, often fail to generalize to out-of-distribution (OOD) domains due to differences in scanner, stain, or other site-specific sources of variation. We investigate different considerations in contrastive self-supervised learning to improve downstream model generalization performance. Specifically, we evaluate how different augmentations and training time during SimCLR training can affect the generation of task-specific, domain-invariant features. The trained SimCLR feature extractors were evaluated for downstream melanoma classification. The results show that optimizing SimCLR improves the out-of-distribution melanoma detection task by 21% and 56%, according to classification accuracy and sensitivity respectively. The improved OOD performance can benefit melanoma patient care.

Trusting the predictions of deep learning models in safety critical settings such as the medical domain is still not a viable option. Distentangled uncertainty quantification in the field of medical imaging has received little attention. In this paper, we study disentangled uncertainties in image to image translation tasks in the medical domain. We compare multiple uncertainty quantification methods, namely Ensembles, Flipout, Dropout, and DropConnect, while using CycleGAN to convert T1-weighted brain MRI scans to T2-weighted brain MRI scans. We further evaluate uncertainty behavior in the presence of out of distribution data (Brain CT and RGB Face Images), showing that epistemic uncertainty can be used to detect out of distribution inputs, which should increase reliability of model outputs.

We propose a structured variational inference model for disentangling observable evidence of disease (e.g. brain lesions or atrophy) from subject-specific anatomy in brain MRIs. With flexible, partially autoregressive priors, our model (1) addresses the subtle and detailed dependencies that typically exist between anatomical and pathological generating factors of an MRI to ensure the validity of generated samples; (2) preserves and disentangles finer pathological details pertaining to one’s disease state. We additionally demonstrate, by providing supervision to a subset of latent units, that (1) a partially supervised latent space achieves a higher degree of disentanglement between evidence of disease and subject-specific anatomy; (2) when the prior is formulated with an autoregressive structure, knowledge from the supervision can propagate to the unsupervised latent units, resulting in more informative latent representations capable of modelling anatomy-pathology interdependencies.

Segmentation of Multiple Sclerosis (MS) lesions is a challenging problem. Several deep-learning-based methods have been proposed in recent years. However, most methods tend to be static, that is, a single model trained on a large, specialized dataset, which does not generalize well. Instead, the model should learn across datasets arriving sequentially from different hospitals by building upon the characteristics of lesions in a continual manner. In this regard, we explore experience replay, a well-known continual learning method, in the context of MS lesion segmentation across multi-contrast data from 8 different hospitals. Our experiments show that replay is able to achieve positive backward transfer and reduce catastrophic forgetting compared to sequential fine-tuning. Furthermore, replay outperforms multi-domain training, thereby emerging as a promising solution for the segmentation of MS lesions.The code is open-source available at https://github.com/naga-karthik/continual-learning-ms.

A radiology report typically comprises multiple sentences covering different aspects of an imaging examination. With some preprocessing effort, these sentences can be regrouped according to a predefined set of topics, allowing us to implement a straightforward two-stage model for chest X-ray radiology report generation. Firstly, a topic classifier detects relevant findings or abnormalities in an image. Secondly, a conditional report generator outputs sentences from an image conditioned on a given topic. We present experimental results on the test split of the MIMIC-CXR dataset for each stage separately and the system as a whole. Most notably, the proposed model outperforms previous works on several medical correctness metrics based on the CheXpert labeler, establishing a new state-of-the-art. The source code is available at https://github.com/PabloMessina/MedVQA/

We here summarise and discuss our published work on semantic segmentation of 3D neonatal brain MRI with deep networks. In addition to developing an accurate, end-to-end segmentation pipeline specifically designed for neonatal brain MRI, we investigated two approaches that can help alleviate the problem of label scarcity often faced in neonatal imaging. First, we examined different strategies of distributing a limited budget of annotated 2D slices over 3D whole-brain images. In the second approach, we compared the segmentation performance of pre-trained models with different strategies of fine-tuning on a small subset of preterm infants. We illustrated our findings using publicly available MRI scans obtained retrospectively from the Developing Human Connectome Project (ages at scan: 26-45 weeks).

Applications of deep learning solutions, which are usually trained with large amount of dataset, in controlling the spread of Coronavirus Disease 2019 (COVID-19) have shown promising results. Motivated by the lack of large number of well-annotated dataset during the onset of a novel disease, we present a high-performing, interpretable few-shot learning network that detects positive COVID-19 cases with limited examples of ultrasound images. Extensive experiments are conducted to evaluate model performance under different encoder architectures, number of training shots and classification problem complexity. When trained with only 5-shots, network classifies between positive and negative COVID-19 cases with 99.3% overall accuracy, 99.5% recall and 99.25% precision for positive cases. Network explainability is evaluated with two visual explanation tools and reviewed by a practicing clinician to ensure validity of network's decision-making process.

Modern 3D Medical Image Segmentation is typically done using a sliding window approach due to GPU memory constraints. However, this presents an interesting trade-off between the amount of global context the network sees at once, versus the proportion of foreground voxels available in each training sample. It is known already that UNets perform worse with low global context, but enlarging the context comes at the cost of heavy class imbalance between background (typically very large) and foreground (much smaller) while training. In this abstract, we analyze the behavior of Transformer-based (UNETR) and attention gated (Attention-Unet) models along with vanilla-Unets across this trade-off. We explore this using a synthetic data set, and a subset of the spleen segmentation data set from the Medical Segmentation Decathlon to demonstrate our results. Beyond showing that all three types of networks prefer more global context rather than bigger foreground-to-background ratios, we find that UNETR and attention-Unet appear to be less robust than vanilla-Unet to drifts between training versus test foreground ratios.

We present a new approach to distance based cell instance segmentation. Specifically, we design a new loss that more faithfully matches the shapes in segmentation geometry based on computational topology. This loss takes advantage of regularity of the distance maps that require learning. We test our approach using microscopy images consisting of many tissue types in human cells. The results indicate that the new formulation consistently improves the segmentation performance of commonly used network architectures, and the best result advances state-of-the-art.

A fundamental task of a robotic scrub nurse is surgical instrument detection. Errors in the detection process can severely interrupt the surgical workflow of a medical procedure, and therefore, near-perfect performance is required. Despite constituting powerful potential solutions, deep learning methods are subject to imperfections. Even if a scene remains unchanged, the predictions of a trained model can vary from one frame to the next. Moreover, for the instrument detection task, the presence of similar-looking instruments and changing lighting conditions promote misclassifications. In this work, we rely on an RGB-D camera, mounted on a robot manipulator in an eye-in-hand configuration, and we propose a multi-frame voting scheme for instrument detection using Mask R-CNN [1]. The predictions on multiple individual frames of a given scene are used as votes and employed for the determination of a final prediction. We consider different robot poses with the goal of capturing the important visual features of the instruments in the scene. From every pose, the predicted data of the instruments are projected onto a common plane, where the predictions are matched for our voting scheme. With this strategy, we expect to significantly improve the performance of a previously trained model and contribute toward the goal of error-free instrument detection.

Diffuse myocardial diseases can be diagnosed using T1 mapping technique based on T1 relaxation times from MRI data. The T1 relaxation parameter is acquired through pixel-wise fitting of the MRI signal. Hence, pixels misalignment resulted by cardiac motion leads to an inaccurate T1-mapping. Therefore, registration is needed. However, due to the intensity differences between the different time-points, recent unsupervised deep-learning approaches based on minimizing the mean-squared-error (MSE) between the images cannot be utilized directly. To overcome this challenge, we propose a new double-stage method, in which a style-transfer is used to harmonize the signal intensities over time, followed by an unsupervised deep-learning based minimization of the MSE between the images. We evaluated our approach on a publicly available cardiac T1 mapping database of 210 subjects. Our approach achieved the best median model-fitting R^2 compared to baseline methods (0.9794, vs. 0.9651/0.9744/0.9756) and T1 values which are much closer to the the expected myocardial T1 value. Furthermore, both metrics have less variability compared to the other methods.

Breast cancer is the second most common type of cancer in women in Canada and the United States, representing over 25% of all new female cancer cases. Neoadjuvant chemotherapy treatment has recently risen in usage as it may result in a patient having a pathologic complete response (pCR), and it can shrink inoperable breast cancer tumors prior to surgery so that the tumor becomes operable, but it is difficult to predict a patient’s pathologic response to neoadjuvant chemotherapy. In this paper, we investigate the efficacy of leveraging learnt volumetric deep features from a newly introduced magnetic resonance imaging (MRI) modality called synthetic correlated diffusion imaging (CDI^s) for the purpose of pCR prediction. More specifically, we leverage a volumetric convolutional neural network to learn volumetric deep radiomic features from a pre-treatment cohort and construct a predictor based on the learnt features using the post-treatment response. As the first study to explore the utility of CDI^s within a deep learning perspective for clinical decision support, we evaluated the proposed approach using the ACRIN-6698 study against those learnt using gold-standard imaging modalities, and found that the proposed approach can provide enhanced pCR prediction performance and thus may be a useful tool to aid oncologists in improving recommendation of treatment of patients. Subsequently, this approach to leverage volumetric deep radiomic features (which we name Cancer-Net BCa) can be further extended to other applications of CDI^s in the cancer domain to further improve prediction performance.

Usually, parameters of a machine learning (ML) model are used to fine-tune a new ML model using a new dataset. Since a ML model can generate other useful information, such as features, we propose a new method that extracts locally diverse features sets by using different ML models, then applies feature selection (FS) methods to identify the best globally diverse hybrid features, and finally uses them to build an accurate hybrid classifier. These ML models may be pretrained and/or non-pretrained. Simulation results using the medical image dataset DermaMNIST (from MedMNIST2D) indicate that the new hybrid classifiers using the hybrid features extracted by a fine-tuned pretrained ResNet model and the Vision Transformer (ViT) can outperform both the fine-tuned pretrained ResNet model and the ViT, and also perform more accurately than the five commonly used image classifiers (ResNet18, ResNet50, auto-sklearn, AutoKeras, and Google AutoML Vision). New optimization methods will be developed to extract highly informative feature sets from more fine-tuned pretrained ML models and other non-pretrained ML models, select best features, and build a highly accurate, fast, energy-efficient, and memory-efficient classifier for image recognition.

Computed tomography (CT) has been widely explored as a COVID-19 screening and assessment tool to complement RT-PCR testing. To assist radiologists with CT-based COVID-19 screening, a number of computer-aided systems have been proposed. However, many proposed systems are built using CT data which is limited in both quantity and diversity. Motivated to support efforts in the development of machine learning-driven screening systems, we introduce COVIDx CT-3, a large-scale multinational benchmark dataset for detection of COVID-19 cases from chest CT images. COVIDx CT-3 includes 431,205 CT slices from 6,068 patients across at least 17 countries, which to the best of our knowledge represents the largest, most diverse dataset of COVID-19 CT images in open-access form. Additionally, we examine the data diversity and potential biases of the COVIDx CT-3 dataset, finding that significant geographic and class imbalances remain despite efforts to curate data from a wide variety of sources.

Quantitative Susceptibility Mapping is a parametric imaging technique to estimate the magnetic susceptibilities of biological tissues from MRI phase measurements. This problem of estimating the susceptibility map is ill posed. Regularized recovery approaches exploiting signal properties such as smoothness and sparsity improve reconstructions, but suffer from over-smoothing artifacts. Deep learning approaches have shown great potential and generate maps with reduced artifacts. However, for reasonable reconstructions and network generalization, they require numerous training datasets resulting in increased data acquisition time. To overcome this issue, we proposed a subject-specific, patch-based, unsupervised learning algorithm to estimate the susceptibility map. We make the problem well-posed by exploiting the redundancies across the patches of the map using a deep convolutional neural network. We formulated the recovery of the susceptibility map as a regularized optimization problem and adopted an alternating minimization strategy to solve it. We tested the algorithm on a 3D invivo dataset and, qualitatively and quantitatively, demonstrated improved reconstructions over competing methods.

The interpretability of deep learning (DL) for Alzheimer's disease (AD) prediction has provided supporting evidence for the timely intervention of disease progression. In particular, counterfactual reasoning is gradually being employed in the medical field, providing refined visual explanatory maps. However, most visual explanatory maps still rely on visual inspection without quantifying their validity, being a barrier for non-expert individuals. To this end, we propose a novel framework to analyze the counterfactual reasoning-based visual explanation by transforming them into quantitative features. Furthermore, we develop a simple shallow linear classifier to boost the effectiveness of quantitative features while promoting the model's interpretability and achieving superior predictive performance compared to the DL model. By doing so, our method further provides an ADness index that can be used to intuitively comprehend a patient's brain status with respect to AD.

In this paper, we develop attribution-based confidence (ABC) metric to detect black-box adversarial attacks in breast histopathology images used to detect cancer. Due to the lack of data for this problem, we subjected histopathological images to adversarial attacks using the state-of-the-art technique Meta-Learning the Search Distribution (Meta-RS) and generated a new dataset. We adopt the Sobol Attribution Method to the problem of cancer detection. The output helps the user to understand those parts of the images that determine the output of a classification model. The ABC metric characterizes whether the output of a deep learning network can be trusted. We can accurately identify whether an image is adversarial or original with the proposed approach. The proposed approach is validated with eight different deep learning-based classifiers. The ABC metric for all original images is greater or equal to 0.8 and less for adversarial images. To the best of our knowledge, this is the first work to detect attacks on medical systems for breast cancer detection using the ABC metric.

Unpaired image-to-image translation methods aim at learning a mapping of images from a source domain to a target domain. Recently, these methods showed to be very useful in biological applications to display subtle phenotypic cell variations otherwise invisible to the human eye. However, while most microscopy experiments remain limited in the number of images they can produce, current models require a large number of images to be trained. In this work, we present an improved CycleGAN architecture that employs self-supervised discriminators to alleviate the need for numerous images. We demonstrate quantitatively and qualitatively that the proposed approach outperforms the CycleGAN baseline including when it is combined with differentiable augmentations. We also provide results obtained with small biological datasets on obvious and non-obvious cell phenotype variations demonstrating a straightforward application of this method.

The main aim of this work is to improve the visualization of abnormalities in Optical Coherence Tomography (OCT) images of the human retina. OCT images have substantial noise, which can affect the classification and visualization performance of a neural network. In this work, we show that denoising improves visualization without affecting the classification performance considering the specific pathology of Pigmented Epithelial Detachment (PED). The noise in OCT images may lead to unstable training and poor classification/visualization performance. Hence, the need for image quality enhancement. We consider several image denoising techniques, namely, K-Singular Value Decomposition (K-SVD), Bilateral filter optimized using Poisson Unbiased Risk Estimate (PURE), Poisson Unbiased Risk Estimate - Linear Expansion of Thresholds (PURE-LET), Guided filter, Rolling Guidance filter, Gated Convolutional and Deconvolutional Structure (GCDS), Denoising Convolution Neural Network (DnCNN), and DnCNN with skip connections. We compare the classification and visualization results obtained from the model trained on noisy images with that trained on enhanced images. Several class activation mapping (CAM) based techniques have been developed, for instance, Grad-CAM, Grad-CAM++, Score-CAM, Ablation-CAM, and Self-Matching CAM, which are also the visualization techniques that we employ in this paper. Our results show that denoising improves visualization performance by a factor of approximately10 in Jaccard Index, taking the expert segmentation as the ground-truth.

Real-time margin assessment in breast cancer surgeries is critical to reduce positive margin rates. The iKnife is an intra-operative modality that captures the molecular signature of tissues and can be paired with AI to facilitate real-time tissue characterization. As training these AI models is typically done with homogeneous ex-vivo iKnife data, intra-operative deployment is challenging because of tissue heterogeneity and unseen classes. In this study, we explore different mechanisms to address the intra-operative deployment challenge. Using cross validation and comparison to baseline methods, we show that the intermediate attention of graph transformer model as well as the uncertainty estimation of Bayesian neural network can be used to to reduce false positive rate of breast cancer surgery. We conclude that the class prediction output is not enough for successful deployment and additional interpretability features are needed to improve the performance.

We study the precise counting of Helicobacter Pylori (HP), which is important for diagnosis of gastric cancer. The crowd counting technique is adapted for a precise quantitative analysis. The challenge of training an HP counting model lies in scarcity of labels. We use a DCGAN for the generative modelling of HP morphology and perform high-fidelity data augmentation. The comparative results show our method outperforms the object detection and semantic segmentation baselines. The proposed framework is potential useful in quantitative analysis of other bacteria in histology images. The dataset is available at https://cyxhello.github.io/HPCDataset/.

In patients with coronary artery disease the prediction of future cardiac events such as myocardial infarction (MI) remains a major challenge. In this work, we propose a novel anatomy-informed multimodal deep learning framework to predict future MI from clinical data and Invasive Coronary Angiography (ICA) images. The performance of our framework on a clinical study of 445 patients admitted with acute coronary syndromes confirms that multimodal learning increases the predictive power and achieves a relatively good performance (AUC: $0.67\pm0.04$ \& F1-Score: $0.36\pm0.12$), which outperforms the prediction obtained by each modality independently as well as that of two interventional cardiologists. To the best of our knowledge, this is the first and promising attempt towards combining multimodal data through a deep learning framework for future MI prediction.

Deep Learning is now well established as the most efficient method for medicalimage segmentation. Yet, it requires large training sets and ground-truth labels,annotated by clinicians in a time-consuming process. We propose an unsupervisedsegmentation method using multi-atlas registration. The architecture of our regis-tration model is composed of cascaded networks that produce small amounts ofdisplacement to warp progressively the moving image towards the fixed image.Once the networks are trained, multiple annotated magnetic resonance (MR) fetalbrain images and their labels are registered with the image to segment, the resultingwarped labels are then combined to form a refined segmentation. Experiments showthat our cascaded architecture outperforms the state-of-the-art registration methodsby a significant margin. Furthermore, the derived segmentation method obtainssimilar results as one of the most robust state-of-the-art segmentation methods,without using any labels during training.

Diffusion Magnetic Resonance Imaging (dMRI) allows probing the tissue microstructure in-vivo using biophysical models. In large group-level analyses, the estimation of these models is often biased due to inter- and intra-scanner variability which results in spurious group differences. Thus to correct the distribution shift in the data acquired at different sites, data harmonization is a crucial step in the dMRI analysis pipeline. Apart from the issue of bias, dMRI data also suffers from a limited signal-to-noise ratio (SNR) which also corrupts the signal, leading to spurious estimates of the underlying tissue micro-architecture. In this work, we explore the interaction of denoising with retrospective harmonization of dMRI data. Specifically, we make use of state-of-the-art denoising and harmonization methods which work directly with the 4D dMRI data of the target and reference sites. Our results show that denoising tends to improve the harmonization performance on account of the reduced variance and improved spherical harmonics-based representations of the signal. We show this using fractional anisotropy measures derived from target and reference sites in the DIAGNOSE-CTE cohort.

Deep Ensemble Convolutional Neural Networks has become a methodology of choice for analyzing medical images with a diagnostic performance comparable to a physician, including the diagnosis of Diabetic Retinopathy. However, commonly used techniques are deterministic and are therefore unable to provide any estimate of predictive uncertainty. Quantifying model uncertainty is crucial for reducing the risk of misdiagnosis. A reliable architecture should be well-calibrated to avoid over-confident predictions. To address this, we propose a UATTA-ENS: Uncertainty-Aware Test-Time Augmented Ensemble Technique for 5 Class PIRC Diabetic Retinopathy Classification to produce reliable and well-calibrated predictions.

Despite being a non-invasive imaging modality, coronary computed tomography angiography (CCTA) is still not the clinical gold-standard modality for the diagnosis and evaluation of Coronary Artery Diseases (CAD), which is typically performed with an invasive coronary angiography (ICA). In this work, we aim at bringing CCTA diagnosis performance closer to the level of the ICA. We propose a deep attention learning framework that takes as an input non-invasive CCTA images and is able to predict a clinical decision, such as revascularization, that is typically based on invasive modalities such as ICA. We represent the CCTA volumetric imaging by two cross-sectional views that follow the curvature of the coronary artery, and we use an attention mechanism that learns a fused representation for better diagnosis. Experimental results on a clinical study of 80 patients indicate that the learned fused model achieves a significant gain in the performance (F1-score: 0.53 +- 0.11) with respect to the CT fractional-flow-reserve (FFR_CT), a clinical baseline estimating the drop of flow from CCTA (F1-score: 0.46+-0.09). These preliminary results confirm that a data-driven approach can boost the diagnosis power of CCTA and eventually contribute towards the wider adoption of this non-invasive imaging modality in clinical settings.

Quantification of the volume of ascities can be an accurate predictor of clinical outcomes to certain pathological setting, e.g., cases of ovarian cancer. Due to the properties of ascities being a liquid, accurate segmentation can be quite a challenging task. In this paper, we show that by tuning nnU-Net, a model that learns the heuristics of the data, it is possible to achieve state-of-the-art segmentation performance. Our trained model, was able to achieve a segmentation Dice score of 0.7, with 0.75 precision and 0.69 recall on pathological test cases. This is a distinct improvement over current state-of-the-art.

We propose to use discrete optimal transport normalizing flows (OT-NF) for the simultaneous synthesis of brain images through years, and the explicit control of such progression. The OT-NF formulation, based on the minimization of the sliced-Wasserstein distance, allows to infer such trajectories in the absence of longitudinal data. The proposed framework could allow the imputation of brain images conditioned on non-constant insults or stimuli.

Susceptibility tensor imaging (STI) is a magnetic resonance imaging technique that can provide important information for reconstruction of neural fibers and detection of myelination changes in the brain. However, the application of STI in human in vivo has been practically infeasible because of its time-consuming acquisition that requires sampling at multiple (usually more than six) head orientations and the challenging dipole inversion problem involved in image reconstruction. Here, we tackle these issues by presenting a novel image reconstruction algorithm for STI that leverages data-driven priors. Our method, called DeepSTI, learns the data prior implicitly via a deep neural network that resembles the proximal operator of a regularizer function. The dipole inversion problem is then solved iteratively using the learned proximal network. Experimental results demonstrate superior performance of DeepSTI over state-of-the-art methods for STI reconstruction and fiber tractography. DeepSTI is the first to achieve high quality results for in vivo human STI with fewer than six orientations.

Multi-institutional collaborations are key for learning generalizable MRI synthesis models that translate source- onto target-contrast images. To facilitate collaboration, federated learning (FL) adopts decentralized training and mitigates privacy concerns by avoiding sharing of imaging data. However, FL-trained synthesis models can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident when common or variable translation tasks are prescribed across sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) to improve reliability against domain shifts. pFLSynth is based on an adversarial model that produces latents specific to individual sites and source-target contrasts, and leverages novel personalization blocks to adaptively tune the statistics and weighting of feature maps across the generator stages given latents. To further promote site specificity, partial model aggregation is employed over downstream layers of the generator while upstream layers are retained locally. As such, pFLSynth enables training of a unified synthesis model that can reliably generalize across multiple sites and translation tasks. Comprehensive experiments on multi-site datasets clearly demonstrate the enhanced performance of pFLSynth against prior federated methods in multi-contrast MRI synthesis

Deep learning models are effective for medical image analysis tasks such as segmentation. However, training these models requires substantial amounts of labeled data, most often annotated manually. Segmenting new medical images to create labeled training data is a tedious and time-consuming process for human annotators. Interactive segmentation tools seek to alleviate this problem, most often by predicting completed segmentations from limited user inputs. This works reasonably well for some domains and for well-defined tasks. But for a new domain or task, the segmentation task is ambiguous. We hypothesize than in such situations proposing multiple partial segmentations is more useful than proposing a single complete segmentation. We propose a probabilistic partial segmentation model, that takes an input image and partial segmentation, and predicts possible next steps for the segmentation. The proposed model can be used iteratively to help annotators accurately and efficiently segment new medical images. The user can choose among multiple predicted larger segmentations and perhaps make a small number of corrections before inputting the updated segmentation back into the system. By predicting multiple larger partial segmentations at each iteration rather than attempting to fully complete the segmentation in one step, the system can enable users to produce accurate segmentations for new medical image domains with fewer corrections. We use synthetic data to demonstrate the proposed model and show a proof-of-concept for the system.

Generative adversarial networks (GAN) that learn a one-shot mapping from source to target images have been established as state-of-the-art in medical image synthesis tasks. GAN models implicitly characterize the target image distribution, so they can suffer from limited sample fidelity and diversity. Here, we propose a novel method based on diffusion modeling, SynDiff, for improved reliability and performance in medical image synthesis. To learn a direct correlate of the image distribution, SynDiff employs conditional diffusion to gradually map noise and source images onto the target image. For fast sampling during inference, large step sizes are coupled with adversarial projections for reverse diffusion. To enable training on unpaired datasets, a cycle-consistent architecture is introduced with coupled diffusion processes that synthesize the target given source and vice versa. Experiments on a public multi-contrast MRI dataset indicate the superiority of SynDiff against competing GAN and diffusion models.

Semi-supervised learning (SSL) promises gains in accuracy compared to training classifiers on small labeled datasets by also training on many unlabeled images. Unfortunately, modern deep SSL often makes accuracy worse when given uncurated unlabeled sets. In realistic applications like medical imaging, unlabeled sets are often uncurated and thus possibly different from the labeled set in represented classes. Recent remedies suggest filtering approaches that detect out-of-distribution (OOD) unlabeled examples and then discard or downweight them. Instead, we view all unlabeled examples as potentially helpful. We introduce a procedure called Fix-A-Step that can improve heldout accuracy of common deep SSL methods despite lack of curation. Our first key insight is that unlabeled data, even OOD, can usefully inform augmentations of labeled data. Our second innovation is to modify gradient descent updates to prevent following the multi-task SSL loss from hurting abeled-set accuracy. Though our method is simpler than alternatives, we show consistent accuracy gains on common CIFAR-10 benchmarks across all levels of contamination. We further suggest a new medically-focused robust SSL benchmark called Heart2Heart, where the core task is recognizing the view type of ultrasound images of the heart. On Heart2Heart, Fix-A-Step can learn from 353,500 truly uncurated unlabeled images to deliver gains that generalize across hospitals.

Automatic cancer localization of irregular shaped abnormalities from mammogram images has remained challenging. This is mainly because annotated mammograms are scarce for training learning models to analyze such high resolution and complex images. In clinical settings for mammogram screening, radiologists not only examine images obtained during the examination, but also compare the current and prior mammogram images to make a clinical decision. To have an automatic breast cancer localization system and to address the problem of lack of annotated mammograms, in this study, we develop an unsupervised feature correlation deep learning model. The proposed model compares unannotated current and previous mammograms and employs an attention-based U-Net based network to identify and generate a map for abnormalities.

A 50-gene subtype predictor called PAM50 was developed to predict breast cancer prognosis and chemotherapy benefit by classifying cancer as luminal A, luminal B, or others. We propose a machine-learning pipeline analyzing H&E stained TMA images to discriminate between luminal A and luminal B subtypes in a way that removes the closely-associated tumor-grade effects.

After more than two years since the beginning of the COVID-19 pandemic, the pressure of this crisis continues to devastate globally. The use of chest X-ray (CXR) imaging as a complementary screening strategy to RT-PCR testing is not only prevailing but has greatly increased due to its routine clinical use for respiratory complaints. Thus far, many visual perception models have been proposed for COVID-19 screening based on CXR imaging. Nevertheless, the accuracy and the generalization capacity of these models are very much dependent on the diversity and the size of the dataset they were trained on. Motivated by this, we introduce COVIDx CXR-3, a large-scale benchmark dataset of CXR images for supporting COVID-19 computer vision research. COVIDx CXR-3 is composed of 30,386 CXR images from a multinational cohort of 17,026 patients from at least 51 countries, making it, to the best of our knowledge, the most extensive, most diverse COVID-19 CXR dataset in open access form. Here, we provide comprehensive details on the various aspects of the proposed dataset including patient demographics, imaging views, and infection types. The hope is that COVIDx CXR-3 can assist scientists in advancing machine learning research against both the COVID-19 pandemic and related diseases.

Diabetic retinopathy (DR) is the main cause of visual impairment for the international working-age population. Federated machine learning models trained for DR grade classification using smartphone-based fundus imaging (SBFI) have the potential to enable equitable access for autonomous DR screening services while helping to protect patient data privacy. However, gradient inversion attacks have been shown to be able to reconstruct SBFI data from federated parameter gradient updates, posing a serious threat to patient data privacy. Therefore, it is critical that the gradient inversion attack mechanism is thoroughly understood so that robust defense strategies can be developed to alleviate potential privacy threats. The purpose of this work was to investigate whether it is possible to use estimates of model prediction uncertainty computed using the Bayesian Active Learning by Disagreement (BALD) score metric to identify specific images in an SBFI dataset, which are especially vulnerable to being reconstructed by gradient inversion attacks. Therefore, Spearman’s rank correlation coefficients were calculated to examine the relationship between BALD scores and several metrics measuring the gradient inversion attack reconstruction quality. Experimental results based on 46 images from the Fine-Grained Annotated Diabetic Retinopathy (FGADR) dataset demonstrate that there is a statistically significant moderate negative correlation (rho = -0.629) between BALD score and peak signal-to-noise ratio (PSNR), implying that images with lower BALD scores may be more vulnerable to gradient inversion attacks.

Neuroimaging processing tasks like segmentation and registration are fundamental in a broad range of neuroscience research studies. These tasks are increasingly solved by machine learning based methods. However, given the heterogeneity of medical imaging modalities, many existing methods are not able to generalize well to new modalities or even slight variations of existing modalities, and only perform well on the type of data they were trained on. Most practitioners have limited training data for a given task, limiting their ability to train generalized networks. To enable neural networks trained on one image type or modality to perform well on other imaging contrasts, we propose CIFL: contrast invariant feature learning. CIFL uses synthesized images of varying contrasts and artifacts, and an unsupervised loss function, to learn rich contrast-invariant image features. The resulting representation can be used as input to downstream tasks like segmentation or registration given some modality available at training, and subsequently enables performing that task on contrasts not available during training. In this abstract, we perform preliminary experiments that show this process in neuroimaging segmentation and registration.

Accurate delineation of fine-scale structures is a very important yet challenging problem. Existing methods use topological information as an additional training loss, but are ultimately making pixel-wise predictions. In this abstract, we propose the first deep learning based method to learn topological/structural representations. We use discrete Morse theory and persistent homology to construct an one-parameter family of structures as the topological/structural representation space. Furthermore, we learn a probabilistic model that can perform inference tasks in such a topological/structural representation space. Our method generates true structures rather than pixel-maps, leading to better topological integrity in automatic segmentation tasks. It also facilitates semi automatic interactive annotation/proofreading via the sampling of structures and structure-aware uncertainty.

We define and tackle region-of-interest adaptive (RoI-adaptive) acquisition for accelerated MRI. Existing methods for identifying k-space sampling patterns in accelerated MRI are optimized for the quality of the entire image or a general image-wide task. However, MRI is often acquired to image a specific RoI, such as a suspected pathology. We demonstrate that a sampling strategy that serves for a general multi-purpose task is often suboptimal for each individual objective. We propose a framework that efficiently learns MRI sampling masks specific to the RoI, leading to substantially faster acquisition that still enables accurate analysis of the RoI. We show empirically that our RoI-adaptive acquisition approach significantly outperforms general acquisition baselines in the RoI reconstruction and segmentation tasks.

Communication within or between complex systems is commonplace in the natural sciences and fields such as graph neural networks. The brain is a perfect example of such a complex system, where communication between brain regions is constantly being orchestrated. To analyze communication, the brain is often split up into anatomical regions that each perform certain computations. These regions must interact and communicate with each other to perform tasks and support higher-level cognition. On a macroscale, these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances. When and what types of signals are communicated over time is an unsolved problem and is often studied using either functional or structural data. In this paper, we propose a non-linear generative approach to communication from functional data. We address three issues with common connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity. To evaluate our model, we simulate temporal data that has sparse communication between nodes embedded in it and show that our model can uncover the expected communication dynamics. Subsequently, we apply our model to temporal neural data from multiple tasks and show that our approach models communication that is more specific to each task. The specificity of our method means it can have an impact on the understanding of psychiatric disorders, which are believed to be related to highly specific communication between brain regions compared to controls. In sum, we propose a general model for dynamic communication learning on graphs, and show its applicability to a subfield of the natural sciences, with potential widespread scientific impact.

One barrier to the clinical deployment of deep learning-based models is the presence of images at runtime that lie far outside the training distribution of a given model. We aim to detect these out-of-distribution (OOD) images with a generative adversarial network (GAN). Our training dataset was comprised of 3,234 liver-containing computed tomography (CT) scans from 456 patients. Our OOD test data consisted of CT images of the brain, head and neck, lung, cervix, and abnormal livers. A StyleGAN2-ADA architecture was employed to model the training distribution. Images were reconstructed using backpropagation. Reconstructions were evaluated using the Wasserstein distance, mean squared error, and the structural similarity index measure. OOD detection was evaluated with the area under the receiver operating characteristic curve (AUROC). Our paradigm distinguished between liver and non-liver CT with greater than 90% AUROC. It was also completely unable to reconstruct liver artifacts, such as needles and ascites.

We have developed an Out-of-Distribution (OOD) detection algorithm based on HVAEs with logistic mixture likelihoods, with a likelihood ratio score threshold at test time. We evaluate the method on data from the 2022 Medical OOD (MOOD)and ISLES 2015 challenges.

Glioblastoma (GBM) is an aggressive brain tumor with median patient survival of about 15 months. The key reason for our poor understanding of GBM is that it is a highly heterogeneous disease with molecular heterogeneity and spatiotemporal heterogeneity. Radiogenomics is a rapidly emerging field that seeks to develop non-invasive imaging signatures associated with genetic mutations of such cancers from magnetic resonance imaging (MRI) scans. This paper develops a technique to quantify the molecular heterogeneity of GBM using radiogenomic features. We fit a probabilistic model that predicts the likelihood of 13 different genetic mutations and use a technique called Intensive Principal Component Analysis (InPCA) to visualize the predictions of this model. The principal components of InPCA form an interpretable coordinate system for characterizing the imaging signatures of different GBM pathways; this coordinate system is consistent with clinical research. We quantify the overlap of different pathway groups to characterize the molecular heterogeneity of GBM. Such analysis can potentially be used in the future for targeted treatments, e.g., when patients present with one or more of these overlapping pathways.

We propose performing imbalanced classification by regrouping majority classes into small classes so that we turn the problem into balanced multiclass classification. This new idea is dramatically different from popular loss reweighting and class resampling methods. Our preliminary result on imbalanced medical image classification shows that this natural idea can substantially boost the classification performance as measured by average precision (approximately area-under-the-precision-recall-curve, or AUPRC), which is more appropriate for evaluating imbalanced classification than other metrics such as balanced accuracy. Future versions of this work will be posted online at \url{https://arxiv.org/abs/2210.12234}.

Segmentation and classification of nuclei of various cell types in histology images is a fundamental task in the emerging area of computational pathology (CPath). Deep Learning (DL) methods tend to perform well but generally require large annotated datasets, which are time-consuming and costly to obtain. Semi-supervised learning (SSL) can help mitigate this challenge by exploiting a large amount of unlabeled data for model training alleviating the need for large annotated data. However, SSL models may exhibit poor generalization due to overly reliance on context resulting in a loss of self-awareness. In this paper, we propose a semi-supervised method that learns robust features from both labeled and unlabeled images. Enforces context-aware cross-consistency training in an unsupervised manner. The proposed model incorporates context-awareness consistency by contrasting pairs of overlapping images in a pixel-wise manner from different contexts resulting in robust and consistent context-aware features. Additionally, to improve the prediction confidence, cross-consistency regularization, and entropy minimization are employed on the unlabeled data, as shown by extensive comparative evaluation on a publicly available MoNuSeg dataset.

Recent studies introduced deep models to synthesize clinical contrast-weighted images from magnetic resonance fingerprinting (MRF). While these studies reported high synthesis quality, they require supervision from fully-sampled training data of clinical contrasts that might be challenging to collect due to scan time considerations. To avoid reliance on full supervision, we propose a semi-supervised model (ssMRF) that can be trained directly using accelerated references. To achieve this, ssMRF introduces a semi-supervised loss function based only on acquired k-space samples of target contrasts. ssMRF further leverages complementary Poisson disc masks in a multi-task learning framework for synergistic synthesis of multiple contrasts. Retrospective experiments demonstrate the efficacy of ssMRF where the method yields high-quality synthesis performance across different clinical contrasts on par with the fully-supervised alternative.

Owing to success in the data-rich domain of natural images, Transformers have recently become popular in medical image segmentation. However, the pairing of Transformers with convolutional blocks in varying architectural permutations leaves their relative effectiveness to open interpretation. We introduce Transformer Ablations that replace the Transformer blocks with plain linear operators to quantify this effectiveness. With experiments on 8 models on 2 medical image segmentation tasks, we explore - 1) the replaceable nature of Transformer-learnt representations, 2) Transformer capacity alone cannot prevent representational replaceability and works in tandem with effective design, 3) The mere existence of explicit feature hierarchies in transformer blocks is more beneficial than accompanying self-attention modules, 4) Major spatial downsampling before Transformer modules should be used with caution.

Dry Eye Disease (DED) is one of the most common ocular diseases: over fivepercent of US adults suffer from DED. Tear film instability is a known factorfor DED, and is thought to be regulated in large part by the thin lipid layer thatcovers and stabilizes the tear film. In order to aid eye related disease diagnosis,this work proposes a novel paradigm in using computer vision techniques tonumerically analyze the tear film lipid layer (TFLL) spread. Eleven videos ofthe tear film lipid layer spread are collected with a micro-interferometer and asubset are annotated. A tracking algorithm relying on various pillar computervision techniques is developed. Our method can be found at https://easytear-dev.github.io/.

Chest X-ray (CXR) datasets hosted on Kaggle, though useful from a data sciencecompetition standpoint, have limited utility in clinical use because of their narrowfocus on diagnosing one specific disease. In real-world clinical use, multiplediseases need to be considered since they can co-exist in the same patient. Inthis work, we demonstrate how federated learning (FL) can be used to makethese toy CXR datasets from Kaggle clinically useful. Specifically, we train asingle FL classification model (‘global‘) using two separate CXR datasets – oneannotated for presence of pneumonia and the other for presence of pneumothorax(two common and life-threatening conditions) – capable of diagnosing both. Wecompare the performance of the global FL model with models trained separatelyon both datasets (‘baseline‘) for two different model architectures. On a standard,naive 3-layer CNN architecture, the global FL model achieved AUROC of 0.84and 0.81 for pneumonia and pneumothorax, respectively, compared to 0.85 and0.82, respectively, for both baseline models (p>0.05). Similarly, on a pretrainedDenseNet121 architecture, the global FL model achieved AUROC of 0.88 and0.91 for pneumonia and pneumothorax, respectively, compared to 0.89 and 0.91,respectively, for both baseline models (p>0.05). Our results suggest that FL can beused to create global ‘meta‘ models to make toy datasets from Kaggle clinicallyuseful, a step forward towards bridging the gap from bench to bedside.

Identifying Alzheimer's disease (AD) involves a deliberate diagnostic process owing to its innate traits of irreversibility with subtle and gradual progression, hampering the AD biomarker identification from structural brain imaging (e.g., structural MRI) scans. We propose a novel deep-learning approach through eXplainable AD Likelihood Map Estimation (XADLiME) for AD progression modeling over 3D sMRIs using clinically-guided prototype learning. Specifically, we establish a set of topologically-aware prototypes onto the clusters of latent clinical features, uncovering an AD spectrum manifold. We then measure the similarities between latent clinical features and these prototypes to infer a "pseudo" AD likelihood map. Considering this pseudo map as an enriched reference, we employ an inferring network to estimate the AD likelihood map over a 3D sMRI scan. We further promote the explainability of such a likelihood map by revealing a comprehensible overview from two perspectives: clinical and morphological.

Quantitative Susceptibility Mapping is a parametric imaging technique to estimate the magnetic susceptibilities of biological tissues from MRI phase measurements. This problem of estimating the susceptibility map is ill posed. Regularized recovery approaches exploiting signal properties such as smoothness and sparsity improve reconstructions, but suffer from over-smoothing artifacts. Deep learning approaches have shown great potential and generate maps with reduced artifacts. However, for reasonable reconstructions and network generalization, they require numerous training datasets resulting in increased data acquisition time. To overcome this issue, we proposed a subject-specific, patch-based, unsupervised learning algorithm to estimate the susceptibility map. We improved the conditioning of the problem by exploiting the redundancies across the patches of the map using a deep convolutional neural network. We formulated the recovery of the susceptibility map as a regularized optimization problem and adopted an alternating minimization strategy to solve it. We tested the algorithm on a 3D invivo dataset and, qualitatively and quantitatively, demonstrated improved reconstructions over competing methods.

Multi-institutional collaborations are key for learning generalizable MRI synthesis models that translate source- onto target-contrast images. To facilitate collaboration, federated learning (FL) adopts decentralized training and mitigates privacy concerns by avoiding sharing of imaging data. However, FL-trained synthesis models can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident when common or variable translation tasks are prescribed across sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) to improve reliability against domain shifts. pFLSynth is based on an adversarial model that produces latents specific to individual sites and source-target contrasts, and leverages novel personalization blocks to adaptively tune the statistics and weighting of feature maps across the generator stages given latents. To further promote site specificity, partial model aggregation is employed over downstream layers of the generator while upstream layers are retained locally. As such, pFLSynth enables training of a unified synthesis model that can reliably generalize across multiple sites and translation tasks. Comprehensive experiments on multi-site datasets clearly demonstrate the enhanced performance of pFLSynth against prior federated methods in multi-contrast MRI synthesis

Diabetic retinopathy (DR) is the main cause of visual impairment for the international working-age population. Federated machine learning models trained for DR grade classification using smartphone-based fundus imaging (SBFI) have the potential to enable equitable access for autonomous DR screening services while helping to protect patient data privacy. However, gradient inversion attacks have been shown to be able to reconstruct SBFI data from federated parameter gradient updates, posing a serious threat to patient data privacy. Therefore, it is critical that the gradient inversion attack mechanism is thoroughly understood so that robust defense strategies can be developed to alleviate potential privacy threats. The purpose of this work was to investigate whether it is possible to use estimates of model prediction uncertainty computed using the Bayesian Active Learning by Disagreement (BALD) score metric to identify specific images in an SBFI dataset, which are especially vulnerable to being reconstructed by gradient inversion attacks. Therefore, Spearman’s rank correlation coefficients were calculated to examine the relationship between BALD scores and several metrics measuring the gradient inversion attack reconstruction quality. Experimental results based on 46 images from the Fine-Grained Annotated Diabetic Retinopathy (FGADR) dataset demonstrate that there is a statistically significant moderate negative correlation (rho = -0.629) between BALD score and peak signal-to-noise ratio (PSNR), implying that images with lower BALD scores may be more vulnerable to gradient inversion attacks.

Accurate delineation of fine-scale structures is a very important yet challenging problem. Existing methods use topological information as an additional training loss, but are ultimately making pixel-wise predictions. In this abstract, we propose the first deep learning based method to learn topological/structural representations. We use discrete Morse theory and persistent homology to construct an one-parameter family of structures as the topological/structural representation space. Furthermore, we learn a probabilistic model that can perform inference tasks in such a topological/structural representation space. Our method generates true structures rather than pixel-maps, leading to better topological integrity in automatic segmentation tasks. It also facilitates semi automatic interactive annotation/proofreading via the sampling of structures and structure-aware uncertainty.

Communication within or between complex systems is commonplace in the natural sciences and fields such as graph neural networks. The brain is a perfect example of such a complex system, where communication between brain regions is constantly being orchestrated. To analyze communication, the brain is often split up into anatomical regions that each perform certain computations. These regions must interact and communicate with each other to perform tasks and support higher-level cognition. On a macroscale, these regions communicate through signal propagation along the cortex and along white matter tracts over longer distances. When and what types of signals are communicated over time is an unsolved problem and is often studied using either functional or structural data. In this paper, we propose a non-linear generative approach to communication from functional data. We address three issues with common connectivity approaches by explicitly modeling the directionality of communication, finding communication at each timestep, and encouraging sparsity. To evaluate our model, we simulate temporal data that has sparse communication between nodes embedded in it and show that our model can uncover the expected communication dynamics. Subsequently, we apply our model to temporal neural data from multiple tasks and show that our approach models communication that is more specific to each task. The specificity of our method means it can have an impact on the understanding of psychiatric disorders, which are believed to be related to highly specific communication between brain regions compared to controls. In sum, we propose a general model for dynamic communication learning on graphs, and show its applicability to a subfield of the natural sciences, with potential widespread scientific impact.