Speech-based Medical Decision Support in VR using a Deep Neural Network (Demonstration)

1 Introduction

In this demo paper, we present a novel real-time decision support system for the radiology domain, where the radiologist can visualise and interact with patient data in a VR environment by using natural speech and hand gestures. Our multimodal dialogue system is an extension of previous work [Luxenburger et al., 2016] where we used an Oculus Rift DK2 with integrated eye tracking technology 1 in a remote collaboration setting. In this scenario the radiologist starts with a patient examination form he or she fills in by using a mobile tablet with integrated stylus. The data are transcribed using handwriting recognition, then analysed and stored based on common medical ontologies [Sonntag et al., 2009]. In our scenario, the doctor uses the VR application and interacts with patient records using our dialogue system. The main contribution is the end-to-end system by providing a GPU-accelerated model for automated decision support that computes therapy predictions in real-time.

2 Medical Dialogue System

We implemented a Patient Data Provider service for connecting to PACS (picture archiving and communication system) and RIS (radiology information system). The mobile application retrieves patient data and medical images through this interface and sends back completed reports from the radiologist. As depicted in figure 1, all devices and services are connected through a central hub, the Proxy Server. Services can be registered through our API, and clients can use these services using the XML-RPC protocol. This video 2 shows the complete workflow.

1 Oculus Rift DK 2 with SMI Eye Tracking Upgrade

For real-time data acquisition, we use a mobile device with a digital pen [Sonntag et al., 2014]. The speech-based dialogue system supports three basic types of commands: interaction with the patient data shown on the virtual display (e.g., "Open the patient file for Gerda Meier.", "Show the next page."); question answering functionality about factiod contents of a patient record (e.g., "When was the last examination?"); and the RNN-based therapy prediction component ("Which therapy is recommended?"). In this demo, we implemented the following linguistic phenomena:

• Anaphoric reference resolution: "What is her current medication?"
• Elliptic speech input: "... and the age?"
• Multimodal deictic reference resolution: "Zoom in here" [user points on a region on the display]
• Cross-modal reference resolution (e.g., "What is the second best therapy recommendation?")

We used SiAM-dp [Neßelrath, 2015], an open development platform and workbench for multimodal dialogue systems, for implementation. We integrated the dialogue system into the VR application and connected our backend services.

2 link to video

3 Decision Support Model

Our medical dialogue system facilitates the doctor's decision about which therapy is most suitable for a given patient when examining the patient history. We integrated a prediction model for clinical decision support based on deep learning [Esteban et al., 2016] as backend service, running on a dedicated GPU server. [Esteban et al., 2016] presented a recurrent neural network (RNN) to include sequences of examinations which was modified to take new patient data as additional input. We used their proposed model (RNN/GRU + static input) trained on an anonymised dataset of 475 real patients, containing a total of 19438 diagnoses, 15352 procedures, 59202 laboratory results and 13190 medications from a clinical data intelligence project [Sonntag et al., 2016]. Personal data, such as names, date of birth, and patient identifier were anonymised, and all date and time data were shifted.

For our speech-based multimodal dialogue system, fast response times are of particular interest. We use TensorFlow to enable GPU-accelerated predictions on a scalable platform. Tests show prediction time capabilities in the 100 milliseconds range. Our service runs on a high-performance computer and is accessible to the dialogue system through our Proxy Server.

In our current scenario we created a Unity3D application 3 that resembles a doctor's office. The user can move inside the room (by position tracking) and look around using head tracking. A virtual screen at the wall is used to show the patient file, the previously annotated digital form, or the therapy prediction (see figure 2). The image viewer is currently limited to 2D visualisations, however, for future work we plan to include 3D medical image data. Navigation inside the documents, like zooming or scrolling, is achieved by multimodal interaction, i.e., natural speech and the Oculus Touch controllers.

4 Conclusions and FutureWork

A multimodal dialogue system in combination with data visualisation in virtual reality provides an intuitive and immersive user experience for the doctor. Currently, we are exploring how displaying complex 3D medical images (e.g., DICOM) in virtual reality can improve the diagnostic process. Future work includes additional modalities such as eye-tracking to improve the interaction in VR.

3 http://unity3d.com/

Speech dialogue extensions include ambiguity resolution by asking clarification questions. If a patient name in a user input is ambiguous, the system could proactively ask for clarification (U: "Open the patient file of Mrs. Meier." S: "Gerda Mayer or Anna Maier?"). In addition, users should be able to change or add patient data through natural speech when applicable [Sonntag et al., 2012].

Acknowledgments

This research is part of the project "clinical data intelligence" (KDI) which is founded by the Federal Ministry for Economic Affairs and Energy (BMWi). Further, we'd like to thank Yinchong Yang and Volker Tresp for providing their therapy prediction model and for their guidance during the model integration process.