Holographic computing at the Heidelberg University Hospital

Augmented Reality by Heart

You're my heart, you're my soul..​

Cutting off the blood flow, isolating arteries and rotating the whole organ 180 - how do you completely dissect a real heart without endangering the person it's beating for? In the cooperation project with the Heidelberg University Hospital, we make the impossible possible.

DT
AR
SC

Technologies

  • Augmented Reality
  • Unity (+MRTK)
  • Digital Twin
  • 3D Modeling (GLTF 2.0, Blender)
  • Networking (SignalR)

Benefits

  • Diffuse anatomical structures become understandable.
  • Automated workflow of CT data in 3D model

Year

  • since 2019

Innovation

  • „Multiplayermode“
  • Digital Twin & Remoting

The Challenge

The Challenge

The Challenge

Turning CT into 3D!

    Informatics4life is an initiative of cardiologists, computer scientists, and mathematicians who aim to create benefits for medicine by letting algorithms work for them. In our joint project HoloHeart, dreams come true.

    Holographic hearts for cardiology - a dream from the point of view of a data scientist. In patients with congenital heart defects, ventricular fibrillation and other dysfunctions, standard computed tomography provides wonderful data landscapes that only need to be tapped into. From a UX point of view, however, half the battle. Because in medical teaching, the use of the HoloLens only makes sense if certain processes are implemented that are not yet available off the peg.

    How do we go beyond software and create a solution that actually helps students and researchers research, teach and learn better? Can open AR standards be established that also benefit other disciplines? These and other questions concern us in the ongoing cooperation project HoloHeart.

    ‍


    The Solution

    Die Lösung

    A holographic heart

    In a semi-automated workflow we separate from the CT data what does not belong to the heart: bones, muscles, blood vessels, etc., are of no interest to us. A bit of code here, a bit of math there, help us do magic to hundreds of two-dimensional layers until the heart is visible in all its glory in 3D.

    Using a secure network, the Mixed Reality Toolkit (MRTK) and the newly available Model Loader, we manage to create a process in which the application can be loaded directly from the data glasses’ menu instead of having to rely on cloud-based structures.

    The Benefit

    The Benefits

    I will be holding you forever

    It was love at first sight when Prof. Dr. Eberhard Scholz first experienced the HoloHeart. HoloLens on, Systems on and after a few "clicks" it appears: magnificent, colorful and very impressive. A heart in the middle of the room and within reach. The wearer of glasses can enlarge it, reduce it, turn it, turn it upside down. Soon, individual segments will be able to be pulled out and viewed in isolation. The model of a real organ will be in front of them, based on T-data from a living person.

    I'll keep it shining everywhere I go

    Soon it will be possible to follow the lecturer handling the ventricle, aorta and artery via laptop, tablet or smartphone. Particularly diffuse anatomical structures, such as complex heart defects, will become tangible, even in times of minimum distance. The "hands-on" model helps understand what is otherwise difficult to grasp. E.g. how close together risk areas are on the heart. This is relevant not only for students, but also for interdisciplinary exchange to evaluate positional relationships and discuss surgical interventions.

    For HoloLens wearers, reality does not disappear in the process. They can interact with the model in space without losing a sense of their physical surroundings. This is where the music plays!

    ‍

    The Extras

    Outlook

    The Extra

    Simulations and 4D

    In December 2020, we are working on refining the multiplayer session, which will enable multiple HoloLens wearers to access the model simultaneously and make manipulations on the heart, such as extracting, rotating and scaling individual segments, visible to all.

    Ideas for creating new added value abound. For example, by virtually visualizing the propagation of excitation: comparable to a three-dimensional "map," our HoloHeart could represent the propagation of electrical stresses in the heart and simulate incisions.

    Very profitable would also be the integration of 4D: the contraction of the muscle, the squeezing of the chamber and finally the outflow of blood. A model that holographically represents the heartbeat from start to finish, encompassing all four dimensions of visual perception.

    In the project, we process technologies entirely at the passage of time. Our HoloHeart has the potential to create great benefits for medicine that will need to be opened up to other specialties. You're my brain, you're my wits... well. Presumably other organs are also suitable as holographic, but not as song lyrics.

    INSPIRATIONLABS GmbH

    Rohrbacher Straße 79
    69115 Heidelberg

    +49 6221 42 99 222
    hello@inspirationlabs.com

    Deutsche Version