Designing Wearable Haptic Technology to Support Music Learning for Users with Vision Impairment
Team
-
Research associate (me)
-
Doctoral researcher* (information technology)
-
Faculty supervisor (information technology)
Skills
-
Literature Review
-
Subject matter expert interview
-
Co-design workshop
-
Thematic analysis
Timeframe
-
Dec 2022–Apr 2023
*This project was part of their doctoral research program. I was their mentor, and I equally contributed to major processes.
PROBLEM
Users with vision impairment lack access to hand gestures when learning music
Much musical information written on music sheets is conveyed through hand gestures. A conductor who places their thumb down communicates to a choir to play sharp on the following musical note. But, users who are blind or have low vision (hereafter, BLV) have limited access to these cues and miss out on critical musical information.
A music conductor uses so many hand gestures for you to follow through. But I can't see them, and I am lost for information.
- A BLV user with 12 yrs of music experience from our previous study.
Different hand gestures are used to let players know about their next steps while playing.
There are no assistive technologies that convey gestures to BLV musicians. Assistive technologies in the current market enlarge or verbally read out information written in music sheets (e.g., notes, accents) to BLV users.
SOLUTION
Could wearable haptic technology convey hand gestures to BLV users?
Wearable haptic technology conveys information using vibrations of varying intensity, and a user wears the technology on any body part. We saw the potential in the technology to convey hand gestures to BLV users for three reasons.
-
Prior work has demonstrated the effectiveness of the technology in conveying written musical information (e.g., notes) to sighted musicians.
-
Its physical appearance is non-intrusive and does not draw unwanted social attention to a BLV user.
-
BLV users can receive information about hand gestures without taking their hands off their instruments.
We conducted foundational research to identify user requirements of the technology by asking the following research question.
-
Research Question 1: Which hand gestures do BLV users want wearable haptic technology to communicate when they play music?
Touch is BLV users' most commonly used modality. One must look beyond a technology's functions and consider the materials of the technology to create a positive user experience for BLV users. This is why we asked the second research question.
-
Research Question 2: What materials do BLV users wish to feel when interacting with wearable haptic technology?
One of our research goals was to arrive at design recommendations. The business that develops wearable haptic technology with our recommendations will have a competitive advantage that sets them apart from its competitors.
As statistics show, accessibility is good for business.
Buying power
People with disabilities have a buying power of $50 billion in Canada.
Revenues
Companies that champion accessibility attract people with disabilities and their networks.
Canadians with disabilities
Together with their friends and families, this group is the third-largest market segment in North America.
The predicted outcome for the business are:
-
Attracts new customers with vision impairment through innovative products.
-
Boosts branding and retain existing customers by showing accessibility as a core value.
-
Get competitors' customers with disabilities.
-
Contribute towards making Canadian society more inclusive.
OVERALL PROCESS
We led a 4-month project from research conception to final research report preparation.
Step 1. Scoping
We conducted a literature review, and our goals were: 1) to formulate research questions and determine what needs to be investigated and 2) to quickly get up to speed on co-design techniques.
Doctoral researcher and I equally contributed here.
Microsoft Word
Google Scholar
Step 3. Conducting: Subject Matter Expert Interview
We conducted a 2-hour hybrid subject matter expert interview with a blind music teacher to improve the workshop procedure and refine discussion questions.
Doctoral researcher and I equally contributed here.
Zoom
Google Doc
Step 5. Analyzing & Synthesizing: Thematic Analysis
We conducted a collaborative thematic analysis and found 2 themes that answered to our research questions.
Doctoral researcher and I equally contributed here.
MAXQDA
Miro
Step 2. Research Planning
1. Developed a wearable haptic prototype.
2. Wrote and revised a co-design workshop script based on dry rehearsals.
3. Sorted out study logistics (e.g., participant recruitment, budget, timelines).
I contributed to #2-3. Doctoral researcher #1.
Zoom
Google Doc
Step 4. Conducting: Co-design Workshop
We conducted 5 in-person workshops with 10 BLV music teachers and learners. We recruited participants from a music school for BLV people located in New York.
Doctoral researcher ran the workshops.
Google Doc
Step 6. Dissemination
Final report and presentation: we submitted a polished research report to the HCI conference so that our work could reach out to the wider population.
Doctoral researcher and I equally contributed here.
PowerPoint
Google Doc
SUBJECT MATTER EXPERT INTERVIEW
After we prepared the first draft of a co-design script and developed a wearable haptic prototype, we interviewed a BLV subject matter expert (SME), who is a music teacher in the US, and achieved the following goals:
1. Included new discussion questions that are relevant to BLV music learners' lived experiences.
2. Refined the workshop procedure to be accessible to BLV participants.
3. Finalized which ideation materials to include.
4. Tested out whether the wearable haptic prototype and its preset vibration pattern made sense in relation to music learning.
The doctoral researcher (top panel, left) and I (bottom panel) are conversing with an SME and rehearsing a co-design workshop procedure. That is my "I am serious" face as I make important notes in the script.
-
We included these 12 ideation materials for in the co-design workshops, including soft felt, leather, metal, and wood.
-
We chose these materials because they are generally found in musical instruments and music classrooms.
vibrates
pauses
vibrates
pauses
pauses
vibrates
vibrates
-
The doctoral researcher developed a wearable haptic prototype with one preset vibration pattern called an on-and-off pattern.
-
We included this pattern as a preset because it resembles a metronome, and our SME informed us that a metronome can communicate more than tempo.
-
The pattern can play at different speeds, from 60 to 100 beats per minute (BPM).
RESEARCH SETUP
1. Participant Recruitment
We recruited 10 BLV music students and teachers from a music school in New York, USA (one woman & nine men).
Each co-design workshop had a pair of participants and took place at the school. Group ideation > Individual ideation. Having a pair of participants per session resulted in fruitful insights. Each participant clarified the guiding questions to the other, and they also generated new ideas by reflecting on their shared experiences.
All sessions were video-recorded for accurate transcripts and reliable thematic data analysis.
2. Co-design Script
We had a co-design script to standardize the data collection procedure across five co-design sessions.
The script specified:
1. Greeting statements to participants.
2. Instructions for three activities.
3. Discussion questions in order.
Writing a script is just as important as practising and memorizing a script, from what you need to say to participants and what you need to do. You'd be surprised that you can memorize almost every single word that you need to say to participants with enough practice. It is essential to reduce the presence of any confounding variables across experimental sessions.
CO-DESIGN WORKSHOPS
We had two co-design activities.
Activity 1.
Participants brainstormed which materials were suitable for wearable haptic technology. All ideation materials were laid out from the increasing level of softness, from very soft felt on the left side to very hard metals and wood cork on the right side.
In this photo, 2 participants are given time to feel each material.
Activity 2.
Participants did bodystorming and ideated what non-verbal cues can be communicated with vibrations. Bodystorming is an ideation technique where a participant interacts with a sensorial prototype and generates creative ideas. Participants played their musical instruments, placed the prototype on different body parts, and generated new vibration patterns.
Purple circles highlight the placement of the prototype tried on by BLV participants.
Personal Learning
Our BLV participants did not explore the ideation probe to the extent that we had expected. Their brainstorming centred on applying two preset vibration patterns, and they came up with one new vibration pattern, which is crescendo and decrescendo. This limited exploration with the ideation probe could be due to the individual differences in people's on-the-spot brainstorming skills. Some individuals are naturally good with on-the-stop verbal brainstorming and others are not. To address this individual difference variable, I'd revise the workshop such that BLV participants are given enough time to individually explore the ideation probe and reflect on discussion questions before they join the workshop.
KEY INSIGHTS
Participants saw the value of the technology in communicating basic music information using straightforward vibration patterns.
Theme 1. Communicate Basic
Information
1. Use the "on and off" vibration pattern to tell a BLV user when to start playing their instruments.
"In my jazz band, people yell at me to say it is my turn to play a drum. But I can use this technology to tell me when it is my turn to play and turn to end."
-Y
Participants played the piano upon receiving the on-and-off vibration pattern on their arms.
2. Use the "on and off" vibration pattern to indicate fermata, signalling a BLV user to take a break for an unspecified time.
"Sometimes our conductor would have a fermata, and he would indicate holding his hand out and that close when it is time to pause. We don't see it. The vibration could help". Tell us to hold music."
-K
3. Use the vibration pattern of increasing or decreasing intensity to convey crescendo and decrescendo (i.e., telling a performer to increase the loudness over the duration of a phrase of music.
"Definitely dynamics! If I was playing a phrase that needed to be quieter or I needed to transition to a different dynamic, [vibration] intensity would convey that."
-W
Participants viewed the vibration pattern as an early-warning nudge, signalling "something is coming up in a few seconds." One significant finding was that 4 participants couldn't play their instruments to their expectations because they were focused on trying to detect when they would receive the on-and-off vibration pattern.
Distinguishable
"Technology should be made of hard materials to easy to identify different components of the device. I feel in sense of control when I can identify things."
Dirtiness
"No rough and abrasive materials. Also, no materials like silicon which collects dust and make it feel dirty whenever I touch the technology."
Theme 2. Desired Materials in Technology
There were 4 desired material properties in wearable haptic technology. Materials must be ...
Durable
"The device will take damage. Music [playing] can be very rigorous. Metals and woods are good."
Enjoyable
"I have to enjoy touching the textures as I rub through my fingers. Soft fabrics felt nice."
DESIGN RECOMMENDATIONS
For Functions of Wearable Haptic Technology
-
Generalizing the study findings, vibration patterns can communicate basic musical information, including:
-
Sharp and Flat
-
Tempo
-
Articulation, such as staccato
-
-
Vibration patterns must be straightforward as an "on and off" pattern and distinguishable from one another.
-
Don't let the technology send too many vibration patterns. Otherwise, BLV users cannot focus on the music playing, and they need to pay attention to when they are receiving the patterns.
For Materials of Wearable Haptic Technology
Analyzing two leading companies that offer wearable technology to support music learning revealed that the business should model after the Apple Watch in terms of its material selection. The band options from Apple Watch satisfy all 4 material properties identified in our study.
Specifically,
-
The business should not use silicon because it easily attracts dust and link. The business should use leather, stainless steel, woven nylon, and titanium, as they are durable, distinguishable, and less likely to attract dust.
-
The business needs to offer many band options so that BLV users can choose materials that they enjoy interacting with.
Soundbrenner
Apple Watch
Band Materials
Leather & sillicon band.
Variations of silicon, stainless steel, woven nylon, leather, and titanium.
Watch Face Materials
Stainless steel.
Aluminium, stainless steel, and titanium.
Durability
Sillion attracts dust and lint easily.
Sillion attracts dust and lint easily.
Dirtiness
Sillion attracts dust and lint easily.
Sillion attracts dust and lint easily.
Distinguisability
Users interact with the technology with gestures and by controlling an outer button. The button is made of the material that sets it apart from the watch face. Users can feel a sense of control by efficiently locating the button and the watch face.
Users interact with the technology with gestures, by touching a screen, and by controlling three buttons located on the sides. Each button has a unique texture, and users can feel a sense of control by efficiently locating each button.
Enjoyability
The company offers 2 band options, and this makes it difficult for users to choose band materials that they enjoy.
The company offers 7+ band options, and this allows users to choose band materials that they enjoy.
Comparing materials of wearable technology for music between top companies.
OUTCOMES & LESSONS
The project was full of learning experiences. I learned about bodystorming and expanded my data analytical tool to include Miro.
IMPACT
-
Functional proof-of-concept prototype that can be adopted by other Human-Computer Interaction researchers in their co-design workshops.
-
Submitted two research reports to the top-tier Human-computer Interaction conference (acceptance rate 24%).
-
Design recommendations that outline possible use cases of haptic feedback in different learning scenarios.
Leon Lu, Karen Crochane, Jin Kang, & Audrey Girouard. (revise and resubmit). Learning challenges of BLV musicians. [journal name removed on purpose]
Leon Lu, Jin Kang, Chase Trepin, & Audrey Girouard. (submitted). Designing wearable haptic technology to support music learning for BLV individuals. [journal name removed on purpose]
TAKEAWAY
-
Delegate tasks smartly. Delegating tasks based on each other's skills maximize the team's productivity and allow the team members to learn about other members' skills. I contributed to tasks that I am good at, including research planning, data analysis, and manuscript write-up. The lead investigator's expertise in prototype development allowed me to learn about bodystorming.
-
Be open to new analytical approaches. We conducted most of the analysis using MAXQDA and had finalized codes stored in the software. The lead investigator suggested that we approach theme formation more collaboratively using Miro. At first, I hesitated to try the approach because I was used to forming themes in MAXQDA. I respected his suggestion and gave Miro a try. Guess what? I loved developing themes with Miro. In MAXQDA, theme formation is less collaborative because it does not allow synchronous content editing. Using Miro, I found forming themes more fun and efficient because we did not have to wait until another person finished the file.