Deep learning models have reached or surpassed human-level performance in the field of medical imaging, especially in disease diagnosis using chest x-rays. However, prior work has found that such classifiers can exhibit biases in the form of gaps in predictive performance across protected groups. In this paper, we benchmark the performance of nine methods in improving the fairness of these classifiers. We utilize the minimax definition of fairness, which focuses on maximizing the performance of the worst-case group. Our experiments show that certain methods are able to improve worst-case performance for selected metrics and protected attributes. However, we find that the magnitude of such gains is limited. Finally, we provide best practices for selecting fairness definitions for use in the clinical setting.

In this position paper, we will present and discuss opportunities and challenges brought about by a new deep learning method by AUC maximization (aka \underline{\bf D}eep \underline{\bf A}UC \underline{\bf M}aximization or {\bf DAM}) for medical image classification. Since AUC (aka area under ROC curve) is a standard performance measure for medical image classification, hence directly optimizing AUC could achieve a better performance for learning a deep neural network than minimizing a traditional loss function (e.g., cross-entropy loss). Recently, there emerges a trend of using deep AUC maximization for large-scale medical image classification. In this paper, we will discuss these recent results by highlighting (i) the advancements brought by stochastic non-convex optimization algorithms for DAM; (ii) the promising results on various medical image classification problems. Then, we will discuss challenges and opportunities of DAM for medical image classification from three perspectives, feature learning, large-scale optimization, and learning trustworthy AI models.

Reduced sharing of affect and differences in use of facial expressions for communication are core symptoms of ASD [1] and are assessed as a part of standard diagnostic evaluations [2]. Children with ASD more often display ambiguous expressions compared to children with other developmental delays and typically developing (TD) children [3]. Standardized observational assessments of ASD symptoms require highly trained and experienced clinicians [4] that usually involves manually coding the observations of facial expressions from recorded videos using time-intensive facial action coding systems. Computer vision (CV) can be used to overcome this challenge with automatic extraction of facial landmarks that can be used to quantify the atypicality of facial expressions [5] and emotional competence [6] in ASD. Understanding the range of facial landmarks’ movement and its dynamics across a time domain [7] can serve as a distinctive behavioral biomarker towards early screening of ASD. In our work, we studied the facial landmarks’ dynamics of the toddlers with ASD versus TD, quantified in terms of a complexity estimate derived from Multiscale entropy (MSE) [8] analysis. We hypothesized that the toddlers with ASD would exhibit higher complexity (i.e., less predictability) in their landmarks’ dynamics associated with eyebrows and mouth regions.

This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments and assess the generalization capabilities of our agents to clinical CT volumes. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques, enabling cross-modality applications. Further, we solely condition our agents on 2D slices, breaking grounds for 3D guidance in much more difficult imaging modalities, such as ultrasound imaging. This is an important step towards user guidance during the acquisition of standardised diagnostic view planes, improving diagnostic consistency and facilitating better case comparison.

In patients with stable Coronary Artery Disease (CAD), the identification of lesions which will be responsible of a myocardial infarction (MI) during follow-up remains a daily challenge. In this work, we propose to predict culprit stenosis by applying a deep learning framework on image stenosis obtained from patient data. Preliminary results on a data set of 746 lesions obtained from angiographies confirm that deep learning can indeed achieve significant predictive performance, and even outperforms the one achieved by a group of interventional cardiologists. To the best of our knowledge, this is the first work that leverages the power of deep learning to predict MI from coronary angiograms, and it opens new doors towards predicting MI using data-driven algorithms.

A lines and B lines are artifacts visible in lung ultrasound (LUS) examinations that are routinely used to identify normal and pathological lung tissue respectively. We present a method to distinguish between normal and abnormal LUS clips that includes a convolutional neural network for image classification and a custom clip prediction method that ingests the network’s outputs. The image classifier achieved a mean AUC of 0.964 (SD 0.019) upon cross validation and an AUC of 0.926 on a holdout set from an external centre. With particular hyperparameter values, the clip-based algorithm achieves a recall of 0.90 and true negative rate of 0.92 on the external dataset. The results warrant further investigation of diagnostic tools aided by computer vision in LUS interpretation.

In multi-center randomized clinical trials imaging data can be diverse due to acquisition technology or scanning protocols. Models predicting future outcome of patients are impaired by this data heterogeneity. Here, we propose a prediction method that can cope with a high number of different scanning sites and a low number of samples per site. We cluster sites into pseudo-domains based on visual appearance of scans, and train pseudo-domain specific models. Results show that they improve the prediction accuracy for steatosis after 48 weeks from imaging data acquired at an initial visit and 12-weeks follow-up in liver disease.

Recent line of work indicated strong improvement for transfer learning and model generalization when increasing model, data and compute budget scale in the pre-training. To compare effect of scale both in intra- and inter-domain full and few-shot transfer, in this study we combine for the first time large openly available medical X-Ray chest imaging datasets to reach a dataset scale comparable to ImageNet-1k. We then conduct pre-training and transfer to different natural or medical targets while varying network size and source data scale and domain, being either large natural (ImageNet-1k/21k) or large medical chest X-Ray datasets. We observe strong improvement due to larger pre-training scale for intra-domain natural-natural and medical-medical transfer. For inter-domain natural-medical transfer, we find improvements due to larger pre-training scale on larger X-Ray targets in full shot regime, while for smaller targets and for few-shot regime the improvement is not visible. Remarkably, large networks pre-trained on very large natural ImageNet-21k are as good or better than networks pre-trained on largest available medical X-Ray data when performing transfer to large X-Ray targets. We conclude that high quality models for inter-domain transfer can be also obtained by substantially increasing scale of model and generic natural source data, removing necessity for large domain-specific medical source data in the pre-training. (Code is available at: https://github.com/SLAMPAI/large-scale-pretraining-transfer)

In this work, we consider semi-supervised segmentation as a dense prediction problem using prototype vector correlation and propose a simple way to represent each segmentation class with multiple prototype vectors. To avoid degenerate solutions, two regularization strategies are applied on unlabeled images, based on mutual information maximization and orthogonality. The first one ensures that all prototype vectors are considered by the network, while the other one explicitly enforces prototypes to be orthogonal by decreasing their cosine distance. Experimental results on two benchmark medical segmentation datasets reveal our method's effectiveness in improving segmentation performance when few annotated images are available.

Quantifying confidence in predictions made by deep learning based segmentation systems is of utmost importance for clinical decision making as it can help quantify support for a decision. For Meningioma (a primary brain tumor) segmentation methods, uncertainty estimation is particularly interesting for efficient precision therapy, tumor-growth estimation, and patient-specific treatment planning. While models exists to estimate uncertainty for Glioma tissues, no study has been done to assess the confidence of a model's segmentation prediction for Meningiomas. In this paper, we train the first 3D Bayesian deep neural network (BNN) to segment Meningioma and simultaneously provide an uncertainty estimate. Using 10,674 MRI sequences (T1-w non-tumor and T1-w contrast-enhanced with tumor), we explore optimal training strategies and architectures for BNNs. We obtain a dice score of 0.828 on a held out dataset of 74 sequences. By predicting voxel-level uncertainty, we determine model's confidence in finding tumor regions with a precision which can further assist in downstream tasks such as radiation therapy planning. Our findings also serve as a proof-of-concept to access the quality of meningioma segmentations, which can potentially be used to flag outputs with high-errors that need further human review.

Estimators based on non-convex sparsity-promoting penalties were shown to yield state-of-the-art solutions to the magneto-/electroencephalography (M/EEG) brain source localization problem. In this paper we tackle the model selection problem of these estimators: we propose to use a proxy of the Stein's Unbiased Risk Estimator (SURE) to automatically select their regularization parameters. The effectiveness of the method is demonstrated on realistic simulations and 30 subjects from the Cam-CAN dataset. To our knowledge, this is the first time that sparsity promoting estimators are automatically calibrated at such a scale. Results show that the proposed SURE approach outperforms cross-validation strategies and state-of-the-art Bayesian methods both computationally and statistically.

Over the last years, many 'explainable artificial intelligence' (xAI) approaches have been developed, but these have not always been objectively evaluated. To evaluate the quality of heatmaps generated by various saliency methods, we developed a framework to generate artificial data with synthetic lesions and a known ground truth map. Using this framework, we evaluated two data sets with different backgrounds, Perlin noise and 2D brain MRI slices, and found that the heatmaps vary strongly between saliency methods and backgrounds. We strongly encourage further evaluation of saliency maps and xAI methods using this framework before applying these in clinical or other safety-critical settings.

Deep learning for medical applications faces many unique challenges. A major challenge is the large amount of labelled data for training, while working in a relatively data scarce environment. Active learning can be used to overcome the vast data need challenge. A second challenged faced is poor performance outside of a experimental setting, contrary to the high requirement for safety and robustness. In this paper, we present a novel framework for estimating uncertainty metrics and incorporating a similarity measure to improve active learning strategies. To showcase effectiveness, a medical image segmentation task was used as an exemplar. In addition to faster learning, robustness was also addressed through adversarial perturbations. Using epistemic uncertainty and our framework, we can cut number of annotations needed by 39% and by 54% using epistemic uncertainty and a similarity metric.

Machine learning models commonly exhibit unexpected failures post-deployment due to either data shifts or uncommon situations in the training environment. Domain experts typically go through the tedious process of inspecting the failure cases manually, identifying failure modes and then attempting to fix the model. In this work, we aim to standardise and bring principles to this process by answering a critical question: how do we know that we have identified meaningful and distinct failure types? We suggest that the quality of the identified failure types can be validated by measuring the intra- and inter-type generalisation after fine-tuning and introduce metrics to compare different subtyping methods. In addition, we propose a data-driven method for identifying failure types based on clustering in the gradient space. We evaluate its utility on a classification and an object detection tasks and we show that gradient clustering was able to not only identify failure types with the highest quality according to our metrics but also to identify clinically important failures like undetected catheters close to the ultrasound probe in intracardiac echocardiography.

We present results of a fully automated computer vision pipeline for the analysis of interactions between caregivers and young children in a free play setting. For both caregiver and child, we extract binary time-series signal of ‘reaching’ (1) and ’not reaching’ (0) to the toy and perform dyadic analysis of these data using Markovian models. Our results show that caregiver-child dyads can be clustered into two groups that differ by probabilities of transitions between the model states, serving as a proxy for leading-following behavior characteristics. We found that these two cluster groups differ in terms of the child’s level of social skills as measured by clinical Vineland Adaptive Behavior Scales - Socialization and Communication Subscale. Our results suggest the potential of digital assessment of caregiver-child interactions via computer vision analysis and using it as a tool for screening and providing behavioral biomarkers for neurodevelopmental disorders.

This paper concerns how machine learning explainability advances understanding of self-supervised learning for ultrasound video. We define the explainability as capturing anatomy-aware knowledge and propose a new set of quantitative metrics to evaluate explainability. We validate our proposed explainability approach on medical fetal ultrasound video self-supervised learning and demonstrate how it can guide the choice of self-supervised learning method. Our approach is attractive as it reveals biologically meaningful patterns which may instil human (clinician) trust in the trained model.

Self-supervised learning has been demonstrated to be a powerful way to use unlabeled data in computer vision tasks. In this study, we propose a self-supervised pretraining approach to improve the performance of deep learning models that detect left ventricular systolic dysfunction from 12-lead electrocardiography data. We first pretrain an encoder that can extract rich features from unlabeled electrocardiography data using self-supervised contrastive learning, and then fine-tune the model on the downstream dataset using the pretrained encoder. In experiments, our proposed approach achieved higher performance than the supervised baseline method, using only 28% of the labels used by the baseline method.

Texture smoothing has recently become a promising data augmentation method to enhance the performance of deep learning segmentation methods in medical image analysis. However, a deeper understanding of this phenomenon has not been investigated. In this study, we investigated this phenomenon using a controlled experimental setting, using datasets from the Human Connectome Project, in order to mitigate the inhomogeneity of data confounders to the network, and investigate possible explanations as to why model performance changes when applying different levels of total variation smoothing during data augmentation. Through experiments we confirm previous findings regarding the benefits of smoothing during data augmentation, but further report that the regime of improvement is limited and it changes in relation to the selected imaging protocol. We also found that smoothing during data augmentation produces a spatial attention shift also associated with different performance levels of the trained segmentation model.

Automated segmentation of fundus photographs would improve the quality, capacity, and cost-effectiveness of eye care screening programs. However, current segmentation methods are not robust towards the diversity of images typical for clinical applications. To overcome this, we used contrastive self-supervised learning to pre-train an encoder of a U-Net on a large variety of unlabeled fundus images from the EyePACS dataset. We demonstrate for the first time that the pre-trained network learns to recognize blood vessels, optic disc, fovea, and various lesions without being provided any labels. Furthermore, when fine-tuned on a downstream blood vessel segmentation task, such pre-trained networks achieve state-of-the-art domain transfer performance. The pre-training also leads to an improved few-shot performance and shorter training times on downstream tasks. Altogether, our results showcase the potential benefits of contrastive self-supervised pre-training for real-world clinical applications.

Bone mineral density (BMD) is a critical clinical indicator of osteoporosis, measured by dual-energy X-ray absorptiometry (DXA). Due to the limited access to DXA machines, osteoporosis is often under-diagnosed and under-treated, leading to an increased risk of fragility fractures. In this work, we present an alternative deep learning-based method for BMD estimation and opportunistic osteoporosis screening using plain film radiography. Our method achieves high performance on three datasets of hip, lumbar spine, and chest radiographs, indicating the strong feasibility of opportunistic osteoporosis screening using plain film radiography.

Annotating cancerous regions in whole-slide images (WSIs) plays a critical role in clinical diagnosis and biomedical research, but generating such exhaustive and accurate annotations is labor-intensive and costly. We present a method, named Label Cleaning Multiple Instance Learning (LC-MIL), to refine coarse and approximate annotations – much easier and less costly to obtain – to produce more accurate ones, on a single WSI without the need of external training data. Our experiments on a heterogeneous set of diverse cancer types demonstrate that LC-MIL is a promising and light-weight tool to provide fine-grained and accurate annotations from coarsely annotated pathology sets.

In recent years, generative adversarial networks (GAN) with their sensitivity to detailed structures have been established as state-of-the-art in various synthesis tasks in medical imaging. However, GAN models based on convolutional neural network (CNN) backbones perform local processing with small filters. This inductive bias compromises the learning of long-range spatial dependencies. Here, we propose a novel generative approach for medical image synthesis, ResViT, to combine local precision of convolution operators with contextual sensitivity of vision transformers. ResViT employs an encoder-decoder architecture with a central bottleneck composed of novel aggregated residual transformer blocks (ART) that are expressive for both local and long-distance interactions among image features. Demonstrations on MRI datasets indicate the superiority of ResViT in terms of qualitative observations and quantitative metrics.

Training computer-vision related algorithms on medical images for disease diagnosis or image segmentation is difficult due to the lack of training data, labeled samples, and privacy concerns. For this reason, a robust generative method to create synthetic data is highly sought after. However, most three-dimensional image generators require additional image input or are extremely memory intensive. To address these issues we propose adapting video generation techniques for 3-D image generation. Using the temporal GAN (TGAN) architecture, we show we are able to generate realistic head and neck PET images. We also show that by conditioning the generator on tumour masks, we are able to control the geometry and location of the tumour in the generated images. To test the utility of the synthetic images, we train a segmentation model using the synthetic images. Synthetic images conditioned on real tumour masks are automatically segmented, and the corresponding real images are also segmented. We evaluate the segmentations using the Dice score and find the segmentation algorithm performs similarly on both datasets (0.65 synthetic data, 0.70 real data). We also perform data augmentation experiments to evaluate how synthetic data may be used to improve segmentation model performance. Our findings show that augmenting small datasets with synthetic data results in higher segmentation Dice scores.

Chest X-rays are one of the most common medical images in the medical domain. Reading chest X-rays are often regarded as one entry-level task for radiologist trainees. Traditionally, radiomics, as a field of medical study that aims to extract a large number of quantitative features from medical images, is very popular among the medical-related researchers before the deep learning era. With the rise of deep learning, this task has drawn an increasing attention from AI researchers. Yet, the interpretability of learning-oriented machine intelligence of understanding chest X-rays remains poor and non-transparent compared to radiomics. Motivated to solve the above challenge, we focus on combining radiomics with deep learning to improve the black-box's interpretability. Specifically, we propose a novel training strategy for deep neural networks to learn from radiomics on Chest X-rays to extract robust features without loss of interpretability.

Deep convolutional neural networks (CNNs) have become an essential tool in the medical imaging-based computer-aided diagnostic pipeline. However, training accurate and reliable CNNs requires large fine-grain annotated datasets. To alleviate this, weakly-supervised methods can be used to obtain local information from global labels. This work proposes the use of localized perturbations as a weakly-supervised solution to extract segmentation masks of brain tumours from a pretrained 3D classification model. Furthermore, we propose a novel optimal perturbation method that exploits 3D superpixels to find the most relevant area for a given classification using a U-net architecture. Our method achieved a Dice similarity coefficient (DSC) of 0.44 when compared with expert annotations. When compared against Grad-CAM, our method outperformed both in visualization and localization ability of the tumour region, with Grad-CAM only achieving 0.11 average DSC.

We hypothesize that probabilistic voxel-level classification of anatomy and malignancy in prostate MRI, although typically posed as near-identical segmentation tasks via U-Nets, require different loss functions for optimal performance due to inherent differences in their clinical objectives. We investigate distribution, region and boundary-based loss functions for both tasks across 200 patient exams from the publicly-available ProstateX dataset. For evaluation, we conduct a thorough comparative analysis of model predictions and calibration, measured with respect to multi-class volume segmentation of the prostate anatomy (whole-gland, transitional zone, peripheral zone), as well as, patient-level diagnosis and lesion-level detection of clinically significant prostate cancer. Notably, we find that distribution-based loss functions (in particular, focal loss) are well-suited for diagnostic or panoptic segmentation tasks such as lesion detection, primarily due to their implicit property of inducing better calibration. Meanwhile, (with the exception of focal loss) both distribution and region/boundary-based loss functions perform equally well for anatomical or semantic segmentation tasks, such as quantification of organ shape, size and boundaries.

Computed Tomography (CT) is a key tool for COVID-19 pneumonia staging due to the possibility to quantify relevant imaging findings such as Ground Glass Opacity (GGO) and consolidation. While automatic lung and COVID-19 infection segmentation has been successfully tackled using Deep Learning models, for the infection classification into these imaging findings, the use of fixed thresholds remains the preferred method in literature. However, this method does not consider the evolutionary pathological processes involved in the disease. We perform automatic segmentation of lung and COVID-19 infection through a 3D-UNet and propose the use of Gaussian Mixture Models (GMM) to characterize the GGO and consolidation on each CT scan from a probabilistic perspective. The segmentation of lung and COVID-19 infection achieved a Dice Similarity Coefficient of 0.973±0.015 and 0.817±0.119 (mean±SD), respectively. Using the probability distributions obtained through GMM a dynamic decision boundary was defined for each CT for GGO and consolidation voxel-wise classification. Visual comparison of the use of dynamic and fixed thresholds was performed by 3 experts, revealing similar results for most of the studied CT scans. Currently, the clinical validation of the model is in progress.

Creating ground truth segmentations for medical imaging is labour and time intensive. While promising, contemporary contrastive learning techniques commonly overlook the ultrasound domain. We investigate the potential benefits of using ultrasound's real-time trait through different contrastive learning sampling strategies in multi-class semantic segmentation. First, we perform a head-to-head label efficiency comparison between two state of the art algorithms, one for contrastive learning and the other fully supervised, to demonstrate the efficiency gains from contrastive learning. Next, we leverage the notion of temporal coherency which is the notion that frames within an ultrasound cine that are close together share structural similarities. Using data from over 500 patients, our preliminary results indicate that temporal partitioning has potential improvements to the learned embeddings. Future work is needed to investigate the changes to intra-class compactness and inter-class separability for these embeddings, as well as identifying downstream tasks which may benefit the most from temporal coherency.

Pulmonary embolism (PE) is a common life-threatening condition with a challenging diagnosis, as patients often present with nonspecific symptoms. Prompt and accurate detection of PE and specifically an assessment of its severity are critical for managing patient treatment. We introduce diverse multimodal fusion models that are capable of utilizing weakly-labeled multi-modal data, combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. The best performing multimodality model is an intermediate fusion model that achieves an AUC of 0.96 for assessing PE severity, with a sensitivity of 90\% and specificity of 94%. To the best of our knowledge, this is the first study that attempted to automatically assess PE severity.

Recent developments have shown multiple ways to tackle whole-slide image clas- sification with weak labels, a challenging task due to memory constraints. A recent example is Clustering-constrained Attention Multiple instance learning (CLAM), which encodes whole-slide images (WSI) into a smaller set of features. The down- side of this approach is that the encoder uses ImageNet pre-trained weights for feature extraction, which might result in suboptimal features for downstream clas- sification tasks. In this study we propose to train the CLAM model end-to-end using streaming stochastic gradient descent, which can train deep neural networks at near static memory cost regardless of image input size. This way the encoder can learn task-specific feature representations of whole-slide images. We show that it is possible to train with images of 65536 × 65536 at 0.5µm, and obtain improved results for public datasets of metastasis detection in breast cancer.

Implicit neural representations (INRs) have recently achieved impressive results in image representation. This work explores the uncertainty quantification quality of INRs for medical imaging. We propose the first uncertainty aware, end-to-end INR architecture for computed tomography (CT) image reconstruction. Four established neural network uncertainty quantification techniques -- deep ensembles, Monte Carlo dropout, Bayes-by-backpropagation, and Hamiltonian Monte Carlo -- are implemented and assessed according to both image reconstruction quality and model calibration. We find that these INRs outperform traditional medical image reconstruction algorithms according to predictive accuracy; deep ensembles of Monte Carlo dropout base-learners achieve the best image reconstruction and model calibration among the techniques tested; activation function and random Fourier feature embedding frequency have large effects on model performance; and Bayes-by-backpropogation is ill-suited for sampling from the INR posterior distributions. Preliminary results further indicate that, with adequate tuning, Hamiltonian Monte Carlo may outperform Monte Carlo dropout deep ensembles.

Brain tumor segmentation is a critical task for tumor volumetric analyses and AI algorithms. However, it is a time-consuming process and requires neuroradiology expertise. While there has been extensive research focused on optimizing brain tumor segmentation in the adult population, studies on AI guided pediatric tumor segmentation are scarce. Furthermore, MRI signal characteristics of pediatric and adult brain tumors differ, necessitating the development of segmentation algorithms specifically designed for pediatric brain tumors. We developed a segmentation model trained on magnetic resonance imaging (MRI) of pediatric patients with low-grade gliomas (pLGGs) from our local research hospital. The proposed model utilizes deep Multitask Learning (dMTL) by adding tumor's genetic alteration classifier as an auxiliary task to the main network, ultimately improving the accuracy of the segmentation results.

Here, we present a severity assessment technique based on an interpretable artificial intelligence (AI) method. Our model builds on an authentic multi-reader dataset of 1208 chest X-rays (CXRs) from 396 patients at Emory University affiliated hospitals with confirmed RT-PCR tests within the course of the study. All the CXRs have been labeled by 6 expert chest radiologists and 2 in-training residents into normal, mild, moderate, and severe classes depending on the consolidation and opacity degrees. We train a convolutional neural network (CNN) using a two-stage transfer-learning approach and show that the model outperforms radiologists and residents over unseen data with an average area under the curve (AUC) of 0.97, 0.92, 0.86, and 0.96 for the normal, mild, moderate, and severe classes, respectively. Finally, we visualize the outputs of the most important filters of the CNN using a pruning method to unlock the black box and provide intuition about the decision-making process of the CNN.

Convolution neural network(CNN) is the de-facto standard for medical imaging tasks. However, CNNs are known to be computation expensive. To take advantage of GPU's computing power, techniques such as Image Block to Column(im2col) are adopted at the cost of increasing memory usages. Such reasons prevent convolution-based architectures from training with large batch sizes and applying to high-definition and high-resolution input. In this work, we propose PISTACHIO: patch importance sampling to accelerate CNNs via a hash index optimizer. With efficient hashing-based sampling, we reduce the memory consumption of CNNs while preserving the final training accuracy.

Reliable and precise diagnoses are essential to mitigate severe outcomes of COVID-19. Here we develop a deep learning workflow for robust detection of opacities and sub-classification of COVID-19 anomalies additionally supervised by segmentation of opacity locations. For the classification, we propose ensemble of convolutional neural networks with auxiliary branches that learn to segment the opacity regions for enhanced features. To detect opacities, we used ensemble of detectors trained in a semi-supervised manner. Our workflow was evaluated in the SIIM-FISABIO-RSNA COVID-19 challenge (https://www.kaggle.com/c/siim-covid19-detection). Our method was ranked 5th on the public (mAP 64.8%) and 7th (top 1%) on the private (mAP 62.6%) test sets out of a total of 1305 competing teams. Notably, we did not use any external datasets. Interestingly, geometrical augmentations significantly boosted our model performance, and CheXpert pretraining (much smaller than ImageNet) achieves comparable results to that of ImageNet pre-trained models. In summary, incorporating opacity segmentation branches directly into the classification model architecture appears to be a powerful strategy.

There are limited works showing the efficacy of unsupervised Out-of-Distribution (OOD) methods on complex medical data. Here, we present preliminary findings of our unsupervised OOD detection algorithm, SimCLR-LOF, as well as a recent state of the art approach (SSD), applied on medical images. SimCLR-LOF learns semantically meaningful features using SimCLR and uses LOF for scoring if a test sample is OOD. We evaluated on the multi-source International Skin Imaging Collaboration (ISIC) 2019 dataset, and show results that are competitive with SSD as well as with recent supervised approaches applied on the same data.

Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that enables full-brain coverage at relatively high spatial resolution (millimeters), though its temporal resolution is somewhat limited due to the sluggishness of the hemodynamic response . Conversely, electroencephalography (EEG) is a neuroimaging modality with high temporal resolution (milliseconds) and low spatial resolution as it records electrical signals from electrodes on the surface of the scalp. In light of the complementarity between the two modalities, when acquired simultaneously, EEG and fMRI potentially can compensate for the shortcoming in one modality using the merits of the other. Here we propose a model that enables high spatiotemporal resolution recovery of the latent neural source space via transcoding of simultaneous EEG/fMRI data. Specifically a latent source space with millimeter and millisecond resolution is generated through a hierarchical deep transcoding process based on a cyclic Convolutional Neural Network (CNN). An important property of the model is that instead of it being a "black box", it is interpretable and can be seen to extract meaningful features, such as hemodynamic impulse response functions (HRF) from the data.

We present a novel ranking loss based Multiple Instance Learning (rankMIL) method which uses the routine H&E-stained slide images to predict the human papillomavirus (HPV) infection status of head and neck cancer patients. Experiments were conducted on the publicly available TCGA-HNSC cohort and the proposed method achieved the new state-of-the-art performance (AUC=0.92) compared to previous methods.

Biomarker development is increasingly focusing on heterogeneous sources of data including brain images, biological samples and social data. Biobanks give access to tens of thousands of brain images and other social and biomedical data. These large-scale datasets make it possible to model biomedical outcomes using machine learning. To interpret predictive models, it is crucial to understand how input features influence the prediction. Over the past decades, a wide range of methods has been developed for ranking variables according to their importance in predictive models. Given the variety of settings (e.g. dimensionality or non-linearities, classification vs regression) it remains unclear which method provides the most accurate feature rankings. Benchmarks have been conducted for multiple methods using simulations and empirical validation, yet, these efforts have been disconnected so far because of the diversity of research settings. As a result, some of the most popular methods for estimating variable importance have never been compared. In this work, we extend the literature by systematically comparing the most popular methods for linear and non-linear inputs in classification and regression tasks. For methods providing assessment of statistical significance, we checked if the p-values are well calibrated. We confronted performance metrics with computation time. Deep Neural Networks (DNN) were most reliable at ranking variables according to their importance. SHAP values did not provide reliable population-level importance scores, whereas BART and MDI provided a reasonable tradeoff between computation time and reliability while not providing statistical guarantees. Marginal selection, knockoffs and d0CRT did not generalize well when data were non-linear or correlated. Applied to biomarker learning, DNN and BART provided overall similar importance rankings. Our results emphasize the importance of systematic empirical benchmarks across applied contexts.

Oral epithelial dysplasia (OED) is a pre-cancerous histopathological diagnosis given to a range of head and neck lesions. OED is distinguished by architectural and cytological changes of the epithelium reflecting the loss of normal growth and stratification pattern. Therefore, segmentation of the epithelium layer into three distinct layers can be considered as a first step towards identification of OED by quantifying and comparing nuclear features between the different layers. However, semantic segmentation of regions of interest in large multi-gigapixel histology images remains a challenge due to the sheer size of the histology images and also due to the complexity and variety of histological patterns present in these images. We propose a solution for designing neural networks for effective semantic segmentation of epithelial layers in OED. Our preliminary results reveal that the model achieved using an optimal network architecture approach outperforms all other state-of-the-art models for the semantic segmentation task.

Systemic lupus erythematosus (SLE) is an autoimmune disease in which the immune system of the patient starts attacking healthy tissues of the body. Lupus Nephritis (LN) refers to the inflammation of kidney tissues resulting in renal failure due to these attacks. The International Society of Nephrology/Renal Pathology Society (ISN/RPS) has released a classification system based on various patterns observed during renal injury in SLE. Traditional methods require meticulous pathological assessment of the renal biopsy and are time-consuming. Recently, computational techniques have helped to alleviate this issue by using virtual microscopy or Whole Slide Imaging (WSI). With the use of deep learning and modern computer vision techniques, we propose a pipeline that is able to automate the process of 1) detection of various glomeruli patterns present in these whole slide images and 2) classification of each image using the extracted glomeruli features.

Generating annotated pairs of realistic tissue images along with their annotations is a challenging task in computational histopathology. Such synthetic images and their annotations can be useful in training and evaluation of algorithms in the domain of digital pathology. To address this, we present a framework to generate pairs of realistic colon cancer histology images with corresponding tissue component masks from the input glandular structure layout. The framework shows the ability to generate realistic qualitative tissue images preserving morphological characteristics including stroma, goblet cells and glandular lumen. We also validate the quality of generated annotated pair with the help of a gland segmentation algorithm.

In this work we explore methods of performing 3D segmentation using single-plane 2D projections of target segmentation maps for supervision. Specifically, we attempt to segment the spine in 3D MR scans using annotations derived from registered 2D coronal DXA scans of the same patient. By exploiting prior knowledge of the 3D shape and appearance of the spine, we propose several methods to perform this task. We test these methods empirically using DXA-Dixon MRI scan pairs from the UK Biobank. The best-performing segmentation model achieves good agreement with manual 3D annotations, with a 3D Dice score of 0.642. By performing this segmentation, one can estimate the 3D curve of the spine, which has been shown to improve monitoring and prediction of scoliosis progression.

Contrastive Learning has shown impressive results on natural and medical images, without requiring annotated data. However, a particularity of medical images is the availability of meta-data (such as age or sex) that can be exploited for learning representations. Here, we show that the recently proposed contrastive y-Aware InfoNCE loss, that integrates multi-dimensional meta-data, asymptotically optimizes two properties: conditional alignment and global uniformity. Similarly to \cite{wang2020}, conditional alignment means that similar samples should have similar features, but conditionally on the meta-data. Instead, global uniformity means that the (normalized) features should be uniformly distributed on the unit hyper-sphere, independently of the meta-data. Here, we propose to define conditional uniformity, relying on the meta-data, that repel only samples with dissimilar meta-data. We show that direct optimization of both conditional alignment and uniformity improves the representations, in terms of linear evaluation, on both CIFAR-100 and a brain MRI dataset.

Graph Neural Networks (GNNs) have been shown to be a powerful tool for generating predictions from biological data. Their application to neuroimaging data such as functional magnetic resonance imaging (fMRI) scans has been limited. However, applying GNNs to fMRI scans may substantially improve predictive accuracy and could be used to inform clinical diagnosis in the future. In this paper, we present a novel approach to representing resting-state fMRI data as a graph containing nodes and edges without omitting any of the voxels and thus reducing information loss. We compare multiple GNN architectures and show that they can successfully predict the disease and sex of a person. We hope to provide a basis for future work to exploit the power of GNNs when applied to brain imaging data.

Surgical phase recognition from endoscopic video could enable numerous context-aware technologies that impact efficiency and performance of surgeons and minimally invasive care teams. Surgical phases can vary greatly (from seconds to minutes) due to patient factors and surgeon workflows along with many other reasons. However, the performance of activity recognition models on the varying statistics of surgical phase durations is poorly understood and tested. To address this problem, we ensemble neural networks and other machine learning models with different architectures and temporal parameters to recognize surgical phases. The probability estimates of the ensemble per second of an entire case are then used as a sequence of observations for forward-backward smoothing to generate posterior beliefs of surgical phases over time. We demonstrate the performance of this modeling process on three data sets: 1) robot-assisted inguinal hernia (five phases), 2) robot-assisted training in a porcine model (seven phases), and 3) laparoscopic cholecystectomy (Cholec80, seven phases). The results suggest that this novel method to address varying phases in different procedures holds promise for the future of surgical phase recognition.

Learning models that generalize under different distribution shifts in medical imaging has been a long-standing research challenge. There have been several proposals for efficient and robust visual representation learning among vision research practitioners, especially in the sensitive and critical biomedical domain. In this paper, we propose an idea for out-of-distribution generalization of chest X-ray pathologies that uses a simple balanced batch sampling technique.
We observed that balanced sampling between the multiple training datasets improves the performance over baseline models trained without balancing.

Training deep learning models on medical datasets that perform well for all classes is a challenging task. It is often the case that a suboptimal performance is obtained on some classes due to the natural class imbalance issue that comes with medical data. An effective way to tackle this problem is by using targeted active learning, where we iteratively add data points to the training data that belong to the rare classes. However, existing active learning methods are ineffective in targeting rare classes in medical datasets. In this work, we propose a framework for targeted active learning that uses submodular mutual information functions as acquisition functions. We show that Tailsman outperforms the state-of-the-art active learning methods by ~10%-12% on the rare classes accuracy and ~4%-6% on overall accuracy for Path-MNIST and Pneumonia-MNIST image classification datasets.

The healthcare domain is characterized by heterogeneous data modalities, such as imaging and physiological data. In practice, the variety of medical data assists clinicians in decision-making. However, most of the current state-of-the-art deep learning models solely rely upon carefully curated data of a single modality. In this paper, we propose a dynamic training approach to learn modality-specific data representations and to integrate auxiliary features, instead of solely relying on a single modality. Our preliminary experiments results for a patient phenotyping task using physiological data in MIMIC-IV & chest radiographs in the MIMIC-CXR dataset show that our proposed approach achieves the highest area under the receiver operating characteristic curve (AUROC) (0.764 AUROC) compared to the performance of the benchmark method in previous work, which only used physiological data (0.740 AUROC). For a set of five recurring or chronic diseases with periodic acute episodes, including cardiac dysrhythmia, conduction disorders, and congestive heart failure, the AUROC improves from 0.747 to 0.798. This illustrates the benefit of leveraging the chest imaging modality in the phenotyping task and highlights the potential of multi-modal learning in medical applications.

Deep diffeomorphic registration faces significant challenges for high dimensional images, especially in terms of memory limits. To mitigate this, we propose a Dividing and Downsampling mixed Registration network (DDR-Net), a general architecture that preserves most of image information at multiple scales with reducing memory and inference time cost. In particular, DDR-Net leverages the global context via downsampling the input and utilizes the local details by dividing the input images to subvolumes. Such design fuses global and local information and obtains both coarse-level and fine-level alignments in the final deformation fields. We apply DDR-Net to OASIS dataset. The proposed method is a general method and could be extended to other registration architectures for better performance.

Deep learning models used as computer-assisted diagnosis systems in a medical context achieve high accuracy in numerous tasks; however, explaining their predictions remains challenging. Notably in the medical domain, we aspire to have accurate models that can also provide explanations for their outcomes. In this work we propose a deep learning-based framework for medical image analysis that is inherently explainable while maintaining high prediction accuracy. To this end, we introduce a hybrid approach which uses human-interpretable as well as machine-learned features while minimizing their mutual information. Using images of skin lesions we empirically show that our approach achieves human-level performance while being intrinsically interpretable.

Magnetic Resonance Imaging (MRI) can accurately identify pathologies, their use for population-level screening of diseases has remained infeasible due to the high costs associated with their operations. A large portion of the cost is attributed to the slow process of acquiring enough data to generate images for the human eye to read and identify clinically relevant variables. Existing methods focus on reducing costs by accelerating the data acquisition process. However, the requirement to generate a high-fidelity image imposes certain constraints on the acquisition process, limiting the speedups achievable. We propose the AcceleRated MRScreener (ARMS), which learns to infer the clinically relevant variable directly from raw measurements acquired by the scanner, achieving a speedup of 20x in the data acquisition process, thereby bringing the technology closer to screening. We test the efficacy of our method on the task of identifying clinically significant prostate tumors in the MR scans of the abdomen and in identifying ACL sprains and Meniscus tears in the knee MR scans.

Electrocardiogram (ECG) abnormalities are linked to cardiovascular diseases, but may also occur in other non-cardiovascular conditions such as mental, neurological, metabolic and infectious conditions. However, most of the recent success of deep learning (DL) based diagnostic predictions in selected patient cohorts have been limited to a small set of cardiac diseases. In this study, we use a population-based dataset of >250,000 patients with >1000 medical conditions and >2 million ECGs to identify a wide range of diseases that could be accurately diagnosed from the patient’s first in-hospital ECG. Our DL models uncovered 128 diseases and 68 disease categories with strong discriminative performance.

Quantitative Diffusion-Weighted MRI (DW-MRI) may be deployed to collect quantitative information about tissue properties, such as diffusion and perfusion fractions, and produce quantitative (rather than qualitative) MRI maps. Such a protocol typically relies on fitting bio-physical models to large data sets of MR measurements. Classical fitting methods require long acquisition and computation times in order to obtain reliable measurements of tissue properties. Recent deep-learning methods have the potential to produce faster tissue properties measurements, however, in-consistency between scanners and acquisition protocols, or unknown variations in the physical acquisition parameters, may result with highly unstable predictions. To address this limitation, we focus our interest on the Intra-Voxel Incoherent Motion (IVIM) signal decay model, and propose a novel DNN model: “MELoDee”, for which the IVIM acquisition parameters are incorporated as part of the network’s architecture. In addition, a training protocol that is appropriately tailored to the proposed architecture is introduced. We demonstrate the improved performance of MELoDee compared to previous DNN-based methods through simulation studies for the IVIM model as well as in-vivo DW-MRI data.

This paper presents LESSER, the deep learning-based mobile gaze tracking system for the diagnosis of dyslexia. Dyslexia has been studied extensively across the domain since it is one of the most common learning disabilities found in general population. However, strenuous gaze tracking step with bulky and costly equipment has been hindering the timely diagnosis of dyslexia. With the use of the minimal input which includes coordinate information of both pupil, LESSER outperforms existing approaches which use complete images of both eyes in terms of capability of generalization.

Artificial intelligence in medical imaging has emerged to be a topic with high demand in medical practice in the recent years. However, limited data availability due to strict patient privacy policy becomes a main barrier in this area. Federated learning enables multiple parties to collaboratively train a machine learning/deep learning model without sharing their local data. Model-contrastive federated learning, as a novel federated learning framework, is designed to handle the heterogeneity of local data distribution by using contrastive learning across parties. In this work, we applied the model-contrastive federated learning in multiple chest x-ray datasets to derive a global model for disease diagnosis. Our experiment shows that using federated learning only on two datasets, our model outperforms the model trained in one single dataset by 4%, which indicates the potential to apply federated learning on several chest x-ray datasets to achieve higher accuracy without the need to share local data.

In patients with stable Coronary Artery Disease (CAD), the identification of lesions which will be responsible of a myocardial infarction (MI) during follow-up remains a daily challenge. In this work, we propose to predict culprit stenosis by applying a deep learning framework on image stenosis obtained from patient data. Preliminary results on a data set of 746 lesions obtained from angiographies confirm that deep learning can indeed achieve significant predictive performance, and even outperforms the one achieved by a group of interventional cardiologists. To the best of our knowledge, this is the first work that leverages the power of deep learning to predict MI from coronary angiograms, and it opens new doors towards predicting MI using data-driven algorithms.

Learning models that generalize under different distribution shifts in medical imaging has been a long-standing research challenge. There have been several proposals for efficient and robust visual representation learning among vision research practitioners, especially in the sensitive and critical biomedical domain. In this paper, we propose an idea for out-of-distribution generalization of chest X-ray pathologies that uses a simple balanced batch sampling technique.
We observed that balanced sampling between the multiple training datasets improves the performance over baseline models trained without balancing. Code for this work is available on GitHub at https://github.com/etetteh/OoDGen-ChestXray.