Investigating Visual Cue Design for Augmented Reality Piano Playing
Research question: Is added visual cue complexity beneficial for AR-guided piano playing?
We designed a study and protoype to compare three different designs of augmented reality (AR) visual cues for the purpose of guided piano playing. Despite the lack of statistically significant differences between the visual cue designs across performance and subjective measures, we identified several trends which may be relevant for future AR design decisions and research.
- Duration
- 4 months
- Team
- 2 researchers
- Method
- Within-subjects experiment
- Tools
- Apparatus
My role included designing the experiment, designing and implementing the AR visual cues, creating the Qualtrics survey, and carrying out the quantitative analysis in R. See details
The Problem
Previous research has shown that AR, virtual reality, and mixed reality music-education systems can improve learning outcomes, user engagement, and increase learning motivation. But there is still a need for a better understanding of how best to design these AR systems to optimize usability and user experience.
Much of the focus of previous research has been at the system level but the effectiveness and usability of such systems are in part determined by the components of the system. Specific to music learning, AR overlays can greatly improve the music learning process by providing learners with various visual cues that indicate which notes to play, when to play them, and how to play them on the physical instrument.
However, these visual cues can vary greatly in terms of complexity, and there is a gap in our understanding of how these designs affect users. To address this gap, we investigated how visual cue design complexity affected usability, performance, and user experience in a simple AR-guided piano playing prototype.
Method
Three visual cues varying in complexity evaluated in this study.
Cues one piano key at a time.
Cues one piano key and provides a predictive cue for the general location of the next key through a directional arrow.
Cues three piano keys at once, using conventional stoplight design to indicate keypress order (Green > Yellow > Red).
Apparatus
Prototype developed in Unity using the Meta XR SDK to create the AR piano overlay and capture information for each keypress on the piano. MidiJack to track MIDI output from the piano and facilitate communication between the piano and the prototype.
Deployed on a Meta Quest 3S with Meta Quest Touch Plus controllers to navigate the prototype and configure the AR overlay.
Data recorded to Excel, statistical analysis done in R, Qualtrics survey used to capture subjective and qualitative measures.
Study Design
Participants
9 participants recruited using convenience sampling and supplemented with snowball sampling. Selection criteria included 18 years or older and normal or corrected-to-normal vision.
Mean age of 24.5; 8 reported previous experience with AR/VR, 5 with previous piano playing experience, 4 with previous experience in both.
Experiment Design
Within-subjects experiment where all participants experienced all three visual cue designs (i.e., the 3 conditions): Single, Direction, Stoplight. This design was chosen to maximize statistical power for the anticipated small sample size.
Condition order counterbalanced using a balanced Latin-square design (3 repeated sequences) to mitigate order and practice effects.
Measures
Quantitative: Response time (RT) and accuracy.
Subjective: NASA-TLX scores to measure cognitive workload and System Usability Scale (SUS) scores to measure usability.
Qualitative: Post-experiment feedback from each participant.
Task & Procedure