Developing a speech-based VR application

Prototyping a note-taking tool for a VR application to learn about neuroanatomy

For their capstone project, four students in the MEL in Dependable Software Systems program developed a working prototype of a note-taking tool for a virtual reality application that will be used by UBC medical students to learn about neuroanatomy.

UBC MEL DSS Student Project Developing a speech-based VR application

Historically, medical students have relied on lectures, textbooks and cadavers to learn about human anatomy. Virtual reality and augmented reality applications offer a new approach where students can explore the body’s systems and functions in a completely immersive way, gaining competence and knowledge before they begin working with patients.

VIRTUAL REALITY FOR MEDICAL TRAINING SIMULATION

One virtual reality application under development by UBC HIVE (the campus’s Hackspace for Innovation and Visualization in Education) is SimBrain VR, a training simulation for neurological clinical exams.

For their final-term capstone project, four students in the MEL in Dependable Software SystemsCalvin Fong, Laurie Law, Rebecca Meritz and Niha Thodika – were asked to add new functionality to SimBrain VR. Specifically, the team at UBC Hive were looking for ways to enable medical students to take notes while in the neuroanatomy area of the simulation to help students reinforce and retain important information.

“In a virtual reality environment, it’s impossible to take notes without taking off your headset,” says Niha. “Our challenge was to come up with a way to solve this issue so students could take notes while in the virtual space.”

Equally important was the requirement that the application enable students to “bring their notes back to offline environments so they could revisit the material at home,” says Laurie.

The students worked closely with two team members from UBC HIVE to deepen their understanding of what the project was trying to achieve so they could spec out the system requirements and design the system architecture.

CREATING RELIABLE SYSTEMS

“Our solution was to create an application so students in the virtual reality space could point to an area on the 3D model and then record a speech note that would be converted to text and saved so they could review and modify it later on a website,” says Niha. “The system needed to be reliable from start to end, so that the transcribed speech was accurate and their notes were accessible.”

UBC MEL DSS Student Project Developing a speech-based VR application - diagram

The students used their skills in project management to complete an ambitious amount of work in a very constrained three-month period. “We planned out a road map with numerous milestones to keep us on track, and met every other week with the team at UBC HIVE to share our progress and get feedback,” says Calvin.

SPEECH-TO-TEXT FUNCTIONALITY

Working with virtual reality applications was a new experience for all the group members, as was working with Unity, the cross-platform game engine that powers SimBrain.

“Once we started working with the tools and seeing it from the user perspective, we identified other issues that needed to be addressed,” says Niha. “For example, some of the APIs weren’t as developed as we had thought.”

At the end of the three-month project, the team had developed a working prototype that they shared with their class.

“UBC HIVE has a long and ambitious roadmap of the many tools they want to develop, and this is just one of the features that could then be connected to other projects,” says Niha. “Our work was a proof of concept to show it is possible to do add speech-to-text functionality within their virtual reality applications.”

EXAMINING FAILURE DETECTION, FAULT TOLERANT, COST AND SCALABILITY

Their design evaluation process included an examination of failure detection, fault tolerance, cost and scalability. The students also outlined opportunities for future work and development.

According to the group members, the capstone project was an important opportunity to draw on the knowledge they had acquired over the course of the MEL in software development, distributed systems, security and testing.

SOLVING REAL-WORLD PROBLEMS

“This capstone project was very useful for practising what we had been learning in our classes,” says Laurie. “It’s a different level of learning when you are not just going over the theory but actually implementing something and developing a prototype that offers a feasible solution to a real problem.”

Calvin adds that the business and leadership courses gave them shared skills to work on the capstone project as a team. “The business classes covered concepts around how to build effective teams, communication and project management – and we benefited from these leadership skills.”

 

UBC Crest The official logo of the University of British Columbia. Arrow An arrow indicating direction. Arrow in Circle An arrow indicating direction. Caret An arrowhead indicating direction. E-commerce Cart A shopping cart. Time A clock. Chats Two speech clouds. Facebook The logo for the Facebook social media service. Social Media The globe is the default icon for a social media platform. TikTok The logo for the TikTok social media platform. Calendar Location Home A house in silhouette. Information The letter 'i' in a circle. Instagram The logo for the Instagram social media service. Linkedin The logo for the LinkedIn social media service. Location Pin A map location pin. Mail An envelope. Telephone An antique telephone. Play A media play button. Search A magnifying glass. Arrow indicating share action A directional arrow. Speech Bubble A speech bubble. Star An outline of a star. Twitter The logo for the Twitter social media service. Urgent Message An exclamation mark in a speech bubble. User A silhouette of a person. Vimeo The logo for the Vimeo video sharing service. Youtube The logo for the YouTube video sharing service. Future of work A logo for the Future of Work category. Inclusive leadership A logo for the Inclusive leadership category. Planetary health A logo for the Planetary health category. Solutions for people A logo for the Solutions for people category. Thriving cities A logo for the Thriving cities category. University for future A logo for the University for future category.