Signals to the summit: UTM prof creates sensory vest for deafblind Everest climber

When Karolina Pakėnaitė attempts to scale Mount Everest, every vibration from a high-tech vest she plans to wear will bring her closer to the top. If she succeeds, she’ll become the first deafblind person to reach the summit. 

It’s an extraordinary feat that will be supported by the Haptic Harness – a signal-sending vest co-created by Leon Lu, an assistant professor at the University of Toronto Mississauga. 

“I felt that this was just something meaningful to do,” says Lu, who teaches at UTM’s Institute of Communication, Culture, Information and Technology and works at the intersection of accessibility and human-computer interaction.

“It wasn’t about what we might get from it - but more how to make her expedition possible and supported in some way.”

Leon Lu, an assistant professor at UTM, works on the Haptic Harness. (Photo by Nick Iwanyshyn)

Lu has studied the use of haptics – or vibrations, similar to those from a phone or video game controller – for several years. 

He previously looked at how the technology can be used to help blind and low vision musicians communicate with their instructors during performances and practices.

“Imagine a conductor using their hands or their faces to instruct the musician, or even a teacher communicating to the student ... If you’re blind, you miss this information,” says Lu, who developed wearable haptic devices that sent vibration signals between musicians and teachers in real time to help them communicate. 

After meeting Pakėnaitė at an accessibility conference a few years ago, Lu realized the challenges she would face on Everest – like communicating with her guides – could be helped by haptics. 

“It was actually very similar to what we were addressing (with musicians). It was just in a different context,” says Lu.

The Haptic Harness is adjusted and assembled at UTM. (Photo by Nick Iwanyshyn)

Pakėnaitė, who lives in the United Kingdom, has Usher syndrome – she was born deaf and her sight began deteriorating at 19. Her Everest climb is currently planned for Spring 2027 and she has been preparing by training, running marathons and scaling other peaks. 

While she has some usable vision, her deafness means she won’t be able to hear her guides on her Everest climb or read their lips while ascending to the summit since their faces will be covered by masks. Her limited vision also poses major difficulties for navigating the terrain. 

The Haptic Harness is designed to help her overcome these challenges, says Lu.  

The harness is a two-part system: Pakėnaitė will wear the harness, and the guides will support her by sending haptic signals, or messages, through a custom remote.

The vest has four motors – one on each shoulder, and two above the waist, which can indicate signals to move left, right, up and down. The duration and pattern of the vibrations can indicate more complex messages, which are determined by the guides and Pakėnaitė. 

The Haptic Harness involves several motors that are placed on a fabric vest. (Photo by Nick Iwanyshyn)

The project has involved several collaborators. 

Lu has been working with German-based HapticLabs - a company that designs haptics - as well as Rodolfo Cossovich, an assistant professor from NYU Shanghai, whose background in electrical engineering and accessibility supported the technology’s development. 

For the harness fabric, Lu partnered with Hayley Stolee-Smith, a Hamilton-based fashion designer. 

“We have folks who are all in different places, but we’ve all come together to make this happen,” he says.

The team worked on the device for the past year and a half, with Lu describing the development as “an ongoing process of design and iteration.”

They had to factor in elements like weather and battery life, and how to design assistive technology for extreme conditions. They also learned that there are some factors they don’t need to consider, such as moisture.

“Karolina has a team of mentors who have done this climb before and we asked them if we should be concerned about moisture getting into the circuits. They said no, because there isn’t any moisture up in the mountains,” Lu says. “We would never have known that.”

UTM assistant professor Leon Lu works on the Haptic Harness at UTM with Hayley Stolee-Smith, a Hamilton-based fashion designer. (Photo by Nick Iwanyshyn)

While Lu has been focused on the device created specifically for Pakėnaitė to help her achieve her goal, he says it could potentially have broader use. 

“We haven’t given much thought in terms of how this might get applied in other places, but it could definitely help in any scenario where voice or visual communication is not plausible – either because of the context or the ability of the person,” he says. 

Lu adds that his accessibility work stems from his desire to make a positive impact on people’s lives. 

“Karolina is breaking barriers of what society might think people with disabilities can or cannot do,” he says. “It also means that someone else can look at this and say, 'This person can do this – why can’t I?'”