Developing a speech-based VR application
Prototyping a note-taking tool for a VR application to learn about neuroanatomy
For their capstone project, four students in the MEL in Dependable Software Systems program developed a working prototype of a note-taking tool for a virtual reality application that will be used by UBC medical students to learn about neuroanatomy.
Historically, medical students have relied on lectures, textbooks and cadavers to learn about human anatomy. Virtual reality and augmented reality applications offer a new approach where students can explore the body’s systems and functions in a completely immersive way, gaining competence and knowledge before they begin working with patients.
VIRTUAL REALITY FOR MEDICAL TRAINING SIMULATION
One virtual reality application under development by UBC HIVE (the campus’s Hackspace for Innovation and Visualization in Education) is SimBrain VR, a training simulation for neurological clinical exams.
For their final-term capstone project, four students in the MEL in Dependable Software Systems – Calvin Fong, Laurie Law, Rebecca Meritz and Niha Thodika – were asked to add new functionality to SimBrain VR. Specifically, the team at UBC Hive were looking for ways to enable medical students to take notes while in the neuroanatomy area of the simulation to help students reinforce and retain important information.
“In a virtual reality environment, it’s impossible to take notes without taking off your headset,” says Niha. “Our challenge was to come up with a way to solve this issue so students could take notes while in the virtual space.”
Equally important was the requirement that the application enable students to “bring their notes back to offline environments so they could revisit the material at home,” says Laurie.
The students worked closely with two team members from UBC HIVE to deepen their understanding of what the project was trying to achieve so they could spec out the system requirements and design the system architecture.
CREATING RELIABLE SYSTEMS
“Our solution was to create an application so students in the virtual reality space could point to an area on the 3D model and then record a speech note that would be converted to text and saved so they could review and modify it later on a website,” says Niha. “The system needed to be reliable from start to end, so that the transcribed speech was accurate and their notes were accessible.”
The students used their skills in project management to complete an ambitious amount of work in a very constrained three-month period. “We planned out a road map with numerous milestones to keep us on track, and met every other week with the team at UBC HIVE to share our progress and get feedback,” says Calvin.
SPEECH-TO-TEXT FUNCTIONALITY
Working with virtual reality applications was a new experience for all the group members, as was working with Unity, the cross-platform game engine that powers SimBrain.
“Once we started working with the tools and seeing it from the user perspective, we identified other issues that needed to be addressed,” says Niha. “For example, some of the APIs weren’t as developed as we had thought.”
At the end of the three-month project, the team had developed a working prototype that they shared with their class.
“UBC HIVE has a long and ambitious roadmap of the many tools they want to develop, and this is just one of the features that could then be connected to other projects,” says Niha. “Our work was a proof of concept to show it is possible to do add speech-to-text functionality within their virtual reality applications.”
EXAMINING FAILURE DETECTION, FAULT TOLERANT, COST AND SCALABILITY
Their design evaluation process included an examination of failure detection, fault tolerance, cost and scalability. The students also outlined opportunities for future work and development.
According to the group members, the capstone project was an important opportunity to draw on the knowledge they had acquired over the course of the MEL in software development, distributed systems, security and testing.
SOLVING REAL-WORLD PROBLEMS
“This capstone project was very useful for practising what we had been learning in our classes,” says Laurie. “It’s a different level of learning when you are not just going over the theory but actually implementing something and developing a prototype that offers a feasible solution to a real problem.”
Calvin adds that the business and leadership courses gave them shared skills to work on the capstone project as a team. “The business classes covered concepts around how to build effective teams, communication and project management – and we benefited from these leadership skills.”
Application Deadlines
The online application portal for the January 2025 intake is now open!
Applications are due June 25, 2024.
How to ApplyJoin us for an
Info Session
Sign up for our latest online information sessions and discover what our programs have to offer.
Sign Up NowAssess your Eligibility
Determine if your professional experience and academic background make you a fit for your desired program.
Start Assessment
Dependable Software Systems
Gain the technical, business and project management skills to design and maintain reliable software systems.
Read More
Featured Alumni
Vinod Kotiya
With more than a decade of high-level experience in information technology – including working for India’s and Bangladesh’s largest power company – Vinod Kotiya returned to school to update his technical skills and gain new expertise in business, strategy and leadership.