Advanced mode

See the written notes and indications of keys to play. Green indicates play that note now, yellow means to get ready as it is an upcoming note.

Detail view

To access the details of the song, look upward using the HoloLens. Here you will find the progress bar, tempo, and song details. I placed them above the main view to keep them out of the way and have more real estate for the notes and "sheet music." This menu is accessible in both beginner and advanced modes.

Beginner Mode

Watch as the notes fall towards the keys to press. In the style of Guitar Hero and other instrument games, this is an effective way to begin to learn songs if you do not already know how to read notes.

About

With the use of augmented reality, learn to play piano on your own terms.
Made for Helios using the Microsoft HoloLens.

Duration

12 Weeks

Spring 2017

Role

Designer

Process

Research

Most of AR is focused on training seminars or immersive gaming. The concept of an instrument trainer has been explored, but not fully realized. In the case of a piano, instead of mapping onto an actual piano, tablets are used, which cause confusion and difficulty in having to switch from a tablet to a life size piano. Music & Arts offers 1.5 million lessons a year, making the demographic of those wanting to learn a new instrument very large. However, not everyone can make the buisness hours of the stores, and may only have 5-10 minutes to practice.

Concept

Using the HoloLens, the user can sit directly in front of a piano and learn to read music, as well as the corresponding keys. They can work at their own pace and choose their own songs. By leveraging voice commands, the user can access the menu. By indicating upcoming keystrokes, the user can prepare and match the placement. There are two modes: beginner and advanced. The beginning mode teaches when and how to play the keys, whereas the advanced mode teached the notes with the keys. 

Information Architecture

Sketches

Wireframes

Final Thoughts

Strong Points

The challenge was finding an effective way to present new information through augmented reality. Originally, I wanted to do most instruments in this way. However, to scale down the project, I landed on just piano. And no, I still haven't learned to play piano, so maybe someday this could be produced!

Conclusion

I'm happy with the outcome. It's a simple design, but effective in accomplishing the goal. Soon, I'll work on longer animations, but for now these get the idea across. I would like to do a similar project for a range of instruments. I enjoyed working with AR, and would absolutely like to work more with this technology.

Breanna Marshall, 2023