Skip to main content
Department of Information Technology

Methods for Image Data Analysis

The MIDA group focuses on development of general methods for image data analysis. Our aim is to devise generally applicable methods, which work well independent of the particular application and types of images used. We therefore strive for robust methods which are performing well under varying conditions. Also aiming for practically useful methods, we essentially always collaborate with other groups, including Social Robotics Lab, Quantitative Microscopy and MedIP - Medical Image Processing.

MIDA_Group.gif




Graph-based methods for improved cancer treatment

Interpretable AI-based multispectral 3D-analysis of cell interrelations in cancer microenvironment

MIDA_OncoSpace.jpg


Immunotherapy has become a life-saving option for advanced cancer patients. However, only a minority of patients develop a durable response. Despite great efforts to explain the variable responses to immunotherapy and to optimize patient selection, current diagnostic tools cannot sufficiently guide clinical practice. This project combines state-of-the-art multiplexed microscopy with the latest techniques of image processing and deep learning to radically advance the understanding of how cell interrelations in the tumor microenvironment affect the disease progression and treatment efficacy, ultimately leading to improved treatments and saved lives.
Starting from a large collection of acquired multispectral histology images, the project aims to develop advanced interpretable AI-driven approaches for image data analysis, for characterization of the structural 3D organization and interrelations of different cell types, enabling reliable and explainable prediction of patient disease progression.

The project heavily relies on interdisciplinary competences and is conducted in close collaboration with Patrick Micke's group at the Department of Immunology, Genetics and Pathology (IGP) at Uppsala University.

This research is funded by

Explainable Artificial Intelligence

Interpretation of classification behaviour of deep neural network models

Interp_of_NNs.png


Deep convolutional neural networks demonstrate state-of-the-art performance in many image analysis tasks, however, their opacity does not allow to infer how they arrive at a decision. We are aiming at detection of oral cancer at an early stage, and it is particularly important to develop a reliable algorithm. In our workflow, trained deep convolutional neural networks are used to differentiate cytological images into normal and abnormal classes. We examine methods that could elevate understanding of the deep learning classification properties and enable interpretation of data classification. Furthermore, we would like to increase understanding of the premalignant state by exploring and visualizing what parts of cell images are considered as most important for the task.

This research is funded by

Related publications

  • M. Fraile, C. Fawcett, J. Lindblad, N. Sladoje, G. Castellano. End-to-End Learning and Analysis of Infant Engagement During Guided Play: Prediction and Explainability. In Proc. of the 2022 Intern. Conf. on Multimodal Interaction (ICMI), 2022. Online
  • M. Fraile, J. Lindblad, C. Fawcett, N. Sladoje, and G. Castellano. Automatic analysis of infant engagement during play: An end-to-end learning and Explainable AI pilot experiment. In Proc. of ICMI 2021 Workshop on Bridging Social Sciences and AI for Understanding Child Behavior, (WoCBU), 2021. Online
  • N. Koriakina, N. Sladoje, E. Wetzer, J. Lindblad. Uncovering hidden reasoning of convolutional neural networks in biomedical image classification by using attribution methods. 4th NEUBIAS Conference, Bordeaux, France, March 2020.
  • N. Koriakina, N. Sladoje, E. Bengtsson, E. Darai Ramqvist, J-M. Hirsch, C. Runow Stark, J. Lindblad. Visualization of convolutional neural network class activations in automated oral cancer detection for interpretation of malignancy associated changes. 3rd NEUBIAS Conference, Luxembourg, Feb. 2019.Poster

Trustworthy AI-based decision support in cancer diagnostics

To reach successful implementation of AI-based decision support in healthcare it is important to have trust in the system outputs. One reason for lack of trust is the lack of interpretability of the complex non-linear decision making process. A way to build trust is thus to improve humans’ understanding of the process, which drives research within the field of Explainable AI. For a successful implementation of AI in healthcare and life sciences, it is imperative to acknowledge the need for cooperation of human experts and AI-based decision making systems: Deep learning methods, and AI systems, should not replace, but rather augment clinicians and researchers. This project aims to facilitate understandable, reliable and trustworthy utilization of AI in healthcare, empowering the human medical professionals to interpret and interact with the AI-based decision support system. Watch here an overview of how deep learning can aid screening programs for early oral cancer detection, presented at IT20, the 20th anniversary celebrations of Uppsala University's IT Department Video.

This research is funded by

Related publications

  • N. Koriakina, N. Sladoje, V. Baši?, J. Lindblad. Oral cancer detection and interpretation: Deep multiple instance learning versus conventional deep single instance learning, 2022. Preprint
  • A. Andersson, N. Koriakina, N. Sladoje, J. Lindblad. End-to-end Multiple Instance Learning with Gradient Accumulation. In Proc. of the IEEE Intern. Conf. on Big Data, 2022. Online
  • N. Koriakina, J. Lindblad, and N. Sladoje. The Effect of Within-Bag Sampling on End-to-End Multiple Instance Learning. In Proceedings of the 12th IEEE Intern. Symp. on Image and Signal Processing and Analysis (ISPA), 2021. Online
  • J. Lu, N. Sladoje, C. Runow Stark, E. Darai Ramqvist, J-M. Hirsch, J. Lindblad. A Deep Learning based Pipeline for Efficient Oral Cancer Screening on Whole Slide Images. In Proc. of the Intern. Conf. of Image Analysis and Recognition, ICIAR 2020, Lecture Notes in Computer Science - LNCS 12132, pp 249-261, Springer 2020. Online

AIDA_Trustworthy_Lindblad.jpg

Image Registration

InspireAnimSmall.gif
In this project we study general-purpose registration methods based on distances incorporating both intensity and spatial information. Thus far, we have developed methods that increase convergence regions when using gradient-based optimization, while improving the accuracy, as well as having similar or shorter run-time than comparable intensity-based methods. Through our work on the multimodal representation learning, we have shown that these registration methods can be extended from mono-modal to multi-modal scenarios. Our current research in this project aims to find paths towards greater generality in applicability, increased computational efficiency, and methods to overcome the idiosyncratic challenges of various applications (such as local discontinuities, missing structures, noise and other corruption).

Related publications

  • J. Öfverstedt, J. Lindblad, and N. Sladoje. INSPIRE: Intensity and Spatial Information-Based Deformable Image Registration. To appear, 2023. Preprint
  • J. Öfverstedt, J. Lindblad, and N. Sladoje. Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment. Pattern Recognition Letters 159, 2022.OnlinePreprint
  • J. Öfverstedt, J. Lindblad, and N. Sladoje. Cross-Sim-NGF: FFT-Based Global Rigid Multimodal Alignment of Image Volumes using Normalized Gradient Fields. 10th Workshop on Biomedical Image Registration, WBIR. Lecture Notes in Computer Science, 13386, 2022. Online
  • J. Öfverstedt, J. Lindblad, and N. Sladoje. Fast Computation of Mutual Information with Application to Global Multimodal Image Alignment of Micrographs. COMULIS Conference 2021 Poster
  • J. Öfverstedt, J. Lindblad, N. Sladoje. Fast and Robust Symmetric Image Registration Based on Distances Combining Intensity and Spatial Information. IEEE Trans. on Image Processing, Vol.28(7), 2019. Online

Representation Learning and Image-to-Image Translation

Multimodal Representation Learning

Multimodal images refer to images produced by multiple imaging techniques, such as different sensors. Combining the information of different imaging modalities offers complimentary information about the properties of the imaged specimen. Often these modalities need to be captured in different machines, such that the resulting images need to be registered in order to map the corresponding signals to each other. This can be a very challenging task due to the varying appearance of the specimen in different sensors.
We use contrastive learning to find representations of both modalities which are abstract and very similar with respect to a similarity measure, such that the images of different modalities are mapped into the same representational space. There are requirements that these representations need to fulfil for the downstream task, such as rotational equivariance or intensity similarity, which can be enforced through modifications of the contrastive loss. Finally, in this abstract space, common methods for monomodal registration (intensity-based as well as feature-based) can be used to align the corresponding images. The transformation found this way can be applied to the original images, which overcomes the problem of multimodal registration.

This research is funded by

Related publications

  • L. Nordling, J. Öfverstedt, J. Lindblad, N.Sladoje. Contrastive Learning of Equivariant Image Representations for Multimodal Deformable Registration. IEEE Int. Symp on Biomedical Imaging, ISBI, 2023. To appear.
  • J. Lu, J. Öfverstedt, J. Lindblad, N. Sladoje. Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study. PLoS ONE 17(11): e0276196. Open Access
  • E. Breznik, E. Wetzer, J. Lindblad, N. Sladoje. Cross-Modality Sub-Image Retrieval using Contrastive Multimodal Image Representations. Preprint
  • E. Wetzer, N. Pielawski, E. Breznik, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, N. Sladoje. Contrastive Learning for Equivariant Multimodal Image Representations. "The Power of Women in Deep Learning" Workshop at the "Mathematics of Deep Learning" Programme at the Isaac Newton Institute 2021 Poster
  • E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, N. Sladoje. Registration of Multimodal Microscopy Images using CoMIR – learned structural image representations. COMULIS Conference 2021 Poster
  • N. Pielawski, E. Wetzer, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, N. Sladoje. CoMIR: Contrastive Multimodal Image Representation for Registration. In Proc. of NeurIPS 2020 OnlinePoster Video
  • E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lindblad, I. Floroiu, A. Dumitru, M. Costache, R. Hristu, S. G. Stanciu, N. Sladoje. Cross-modal Representation Learning for Efficient Registration of Multiphoton and Brightfield Microscopy Images of Skin Tissue. 4th NEUBIAS Conference, Bordeaux, March 2020
CoMIR2.png

Image-to-Image Translation

Generative Adversarial Networks (GANs) can be used to create mappings from image domain A to B, such that the distribution of the mapped, -or translated- images of domain A are indistinguishable from the distribution of B. This approach can be used for style transfer and in consequence to translate from one imaging modality to another; or for data augmentation, in particular in fields in which labelled data is expensive, such as biomedical applications.

Related publications

  • J. Lu, J. Öfverstedt, J. Lindblad, and N. Sladoje. Image-to-Image Translation in Multimodal Image Registration: How Well Does It Work? COMULIS Conference 2021. Poster
  • T. Majtner, B. Bajic, J. Lindblad, N. Sladoje, V. Blanes-Vidal, E. S. Nadimi. On the Effectiveness of Generative Adversarial Networks as HEp-2 Image Augmentation Tool. In Proc. of the Scandinavian Conference on Image Analysis, SCIA2019, Lecture Notes in Computer Science, LNCS-11482, 2019. Online
MIDA I2I.jpg

Robust learning of geometric equivariances

The project builds on, and extends recent works on geometric deep learning and aims at combining it with Manifold learning, to produce truly learned equivariances without the need for engineered solutions and maximize benefits of shared weights (parameters to learn). A decrease of the numbers of parameters to learn leads to increased performance, generalizability and reliability (robustness) of the network. An additional gain is in reducing a risk that the augmented data incorporates artefacts not present in the original data. A typical example is textured data, where interpolation performed in augmentation by rotation and scaling unavoidably affects the original texture and may lead to non-reliable results. Reliable texture-based classification is, on the other hand, in many cases of high importance in biomedical applications.

This research is funded by the Wallenberg Autonomous Systems and Software Program, WASP, within AI-Math initiative

Related publications

  • K. Bengtsson Bernander, J. Lindblad, R. Strand, I. Nyström. Replacing data augmentation with rotation-equivariant CNNs in image-based classification of oral cancer. In Proceedings of the 25th Iberoamerican Congress on Pattern Recognition (CIARP), 2021.
image.png

Multi-layer object representations for texture analysis

MultiscaleDetection.png


Texture features such as local binary patterns have shown to provide complementary information to that of plain intensity images in learning algorithms. We investigate methods on the fusion of texture and intensity sources, as well as the problems connected to the fact that many texture descriptors are unordered sets and require suitable (dis-)similarity measures in order to be processes by for example convolutional neural networks. We develop strategies to integrate more complex texture features into learning methods and evaluate their performance on various biomedical images. Such hybrid object representations show promising results in, e.g., detection and segmentation of high resolution transmission electron microscope (TEM) images, taking it one step closer to automation of pathological diagnostics.

This research is funded by

Related publications

  • E. Wetzer, J. Gay, H. Harlin, J. Lindblad, and N. Sladoje. When texture matters: Texture-focused CNNs outperform general data augmentation and pretraining in Oral Cancer Detection. In Proceedings of the 17th IEEE International Symposium on Biomedical Imaging (ISBI), IEEE, pp. 517-521, Iowa City, USA, April 2020. Online
  • E. Wetzer, J. Gay, H. Harlin, J. Lindblad, and N. Sladoje. Texture-Based Oral Cancer Detection: A Performance Analysis of Deep Learning Approaches. 3rd NEUBIAS Conference, Luxembourg, Feb. 2019.Poster
  • B. Bajic, T. Majtner, J. Lindblad, N. Sladoje. Generalized Deep Learning Framework for HEp-2 Cell Recognition Using Local Binary Pattern Maps. IET Image Processing, Vol. 14, No. 6, pp. 1201-1208, 2020. Online
  • E. Wetzer, J. Lindblad, I.-M. Sintorn, K. Hultenby, N. Sladoje. Towards automated multiscale imaging and analysis in TEM: Glomerulus detection by fusion of CNN and LBP maps. In Proc. of the ECCV 2018, Workshop on BioImage Computing, Lecture Notes in Computer Science, LNCS-11134, pp. 465-475, Munich, Germany, Sept. 2018. Online Poster

Image Similarity and Distance Measures

Stochastic Distance Transform

sdt.jpg


Distance transforms (DTs) are based on minimal distances and are thus noise sensitive; a single noisy point can change the distance substantially. The Stochastic Distance Transform (SDT) is a distance transform based on stochastic modelling of the binary images using the theory of discrete random sets. In this project, we explore theoretical properties of the SDT, efficient algorithms that enable the computation of these distance transforms, and how the accuracy of the resulting distances are substantially improved by adopting the SDT in favor of classical DTs.

Related publications

  • J. Öfverstedt, J. Lindblad, and N. Sladoje. Stochastic Distance Transform: Theory, Algorithms and Applications. Journal of Mathematical Imaging and Vision 62(5), 2020. Online
  • J. Öfverstedt, J. Lindblad, N. Sladoje. Stochastic Distance Transform. In Proceedings of the 21th International Conference on Discrete Geometry for Computer Imagery, DGCI2019, Lecture Notes in Computer Science, LNCS-11414, 2019. Online

Combining Shape and Intensity Information

AffineSpots.gif


Similarity (or distance) measures between images are fundamental components in image analysis, and are used in many tasks such as

  • template matching,
  • image registration,
  • classification,
  • objective functions for training various types of Neural Networks.

We study measures which combine image intensity and spatial information efficiently and aim to demonstrate that they lead to practical, robust, high performance methods for these and other common tasks.

Related publications

  • J. Öfverstedt, N. Sladoje, and J. Lindblad. Distance Between Vector-valued Fuzzy Sets based on Intersection Decomposition with Applications in Object Detection. In Proc. of the 13th International Symposium on Mathematical Morphology, ISMM2017, Lecture Notes in Computer Science, LNCS-10225, 2017. Online
  • N. Sladoje and J. Lindblad. Distance Between Vector-valued Representations of Objects in Images with Application in Object Detection and Classification. In Proc. of the 18th International Workshop on Combinatorial Image Analysis, IWCIA2017, Lecture Notes in Computer Science, LNCS-10256, 2017. Online

Updated  2023-02-26 23:02:55 by Natasa Sladoje.