AR/VR for Neuroscience

KIN470 is a 4th year Kinesiology seminar course. The F2017 section of KIN 470 is focused on Multisensory Perception and Action. Students used advanced virtual/augmented reality technologies (e.g., Microsoft Hololens, HTC Vive VR Headset) with which they designed a research experiment. In the last class, the students gave a research experiment demonstration on their topic.

Many upper year post-secondary students are required to present their ideas through oral and written communication forms. These are recognized as core goals to achieve in education as they help students to refine their ability to distill their ideas and disseminate them as effectively as possible. With the advent of computing and user-friendly business communication software, education has encouraged the use of digital technology to provide an additional multisensory element to the presentation of ideas (e.g., Microsoft PowerPoint). The use of communication software in the classroom translates well as a tangible skill required within higher levels of academia and research, as well as within government and industry.

While Microsoft PowerPoint and similar products enable students to rapidly construct multimodal presentations with relatively little technical (programming) skill, the medium is relatively restrictive as it encourages a passive audience with minimal opportunity for engagement and interaction with the speaker. By using extended/mixed reality (XR) technology in the classroom, it is possible to enable a more interactive experience between presenter and audience. If (XR) were to become the next Microsoft PowerPoint, then students would be able to rapidly convey their ideas and interact with the audience in countless ways. With XR technology adoption, it is ever more possible to enter into the mind of the presenter.

As part of a fourth-year Kinesiology seminar taught by Michael Barnett-Cowan at the University of Waterloo on Multisensory Perception and Action, XR technology was used to augment and (in some cases) replace traditional presentation methods (e.g., PowerPoint). The course objectives were to survey the literature on this topic and to construct a unique research proposal based on a literature review in both oral and written format. New this past term was an introduction to the Microsoft Hololens as well as VR HMDs such as the HTC VIVE and Oculus Rift, encouraging students to dream up unique ways that XR technology could be used for research in Kinesiology.

Throughout the course, Π∩ϟ team members worked with the students to teach them the basic principles of the Unreal Engine 4 (UE4) as well as the strengths and limitations of XR technology. We wanted each student to be able to evaluate how XR technologies could be used to better understand how the central nervous system integrates multisensory information. Pairing undergraduate students in Kinesiology with little to no background in computer programming with experienced UE4 developers allowed students to focus on their research concepts and ponder upon potential XR neuroscience experiments.  Students provided ideas for their projects and the developers focused on the rapid development of these ideas in UE4 while teaching students how to code. Students were given some time in the lab to edit these rapidly developed prototypes and present them to their peers as part of the final presentation of their research projects.

From examining the student’s marks, as can be seen in the figure below, the final grades of the students (n=15; F2017) were on average the highest since the class was created in Winter 2014. Indeed, after removing one statistical outlying student (student decided not to hand in some assignments and thus pulled down the class average), the next figure below better demonstrates the increased performance of students in the class.

When interviewing students during and after the class, students clearly expressed that they appreciated learning about and how to use XR technology in the classroom as they felt it better enabled them to convey their thoughts in an immersive virtual environment. Many students expressed a desire to continue working with the technology, with 3 students electing to work in the lab of one of the authors next term so that they could have the opportunity to continue learning how to develop in the UE4 and to translate their ideas in the classroom to research projects.

Previous
Previous

Velocity Labs

Next
Next

McDonald’s BoomBox