Seeing with our ears…

What if blind people could "see" using their ears?

That is the challenge being addressed by Jean Rouat, a professor and researcher in the Department of Electrical and Computer Engineering at Université de Sherbrooke, and PhD student Mohammad Adeli in seeking to develop a new generation of visual prostheses that would allow blind people to perceive visual scenes through sounds.

Several scientific studies have shown that humans associate certain shapes with certain sounds.

This type of sensory substitution technology is feasible. Several scientific studies have shown that humans associate certain shapes with certain sounds—and vice-versa—, pointing to a strong connection between the visual and auditory cortices.

Inspired by this research, Rouat and his colleagues from the Neuroscience and Intelligent Signal Processing Research Group have developed a system for converting visual scenes into auditory scenes. To illustrate the concept, they created an iPhone application called "See Different", which can be used to locate simple objects using sounds. The application analyzes the content of images supplied by the phone's camera and responds to differences in contrast and texture.

However, to integrate this principle into a prosthesis, the researchers need a better understanding of how the link between vision and audition works in the brain. For example, do all individuals associate sounds with shapes in the same manner? Jean Rouat and his team conducted a sensory substitution experiment in which 119 subjects had to select the visual equivalent of each of 23 musical timbres derived from eight musical instruments, from among three shapes: rounded, angular or mixed. The findings showed that the correspondence between musical timbre and visual shape was consistent across all subjects in the study. For example, the smooth sounds derived from a piano were all associated with the rounded shape.

This study, the first to consider complex sounds and precise visual elements, constitutes a major step towards the creation of sensorial substitution prostheses.