'Hearing' through sight

You might say the research was inspired by dolphins. In a pioneer project conducted by an interdisciplinary team at NUS, researchers are working on how people with hearing problems might get a new experience - through an interpretation of sounds with the help of visuals.

It's really a spin-off from our study of how dolphins communicate underwater, explained Dr Elizabeth Taylor who heads the Marine Mammal Research Laboratory at the Tropical Marine Science Institute and who has a long-standing interest in multi-sensory perception. In order to facilitate communication with dolphins and a better understanding of their cognitive abilities, we developed a computerised system to send and receive sounds both above and within our hearing system. The hearing range of dolphins extends far into the ultrasonic frequency range and we miss these sounds, except for a few that we might hear through bone conduction when swimming underwater, so the world might appear rather different to dolphins.

Lessons from the dolphins

The ways in which dolphins use sound to investigate their underwater environment, to navigate, find food and to keep in contact with their friends (and away from harm) are far from being completely understood. Sometimes they will listen passively to sounds around them and sometimes it will be more useful to transmit sound and listen to the echoes bouncing off objects and other features such as the sea bed, said Dr Elizabeth Taylor who heads the TMSI's Marine Mammal Research Laboratory. Dolphins use "echolocation" to investigate their environment in conditions where vision is limited. By producing very rapid clicking sounds when inspecting an object closely, they are able to interpret the reflections of these sounds into a perception of the object similar to that perceived by the eyes.

"Nature tends to optimise sensory systems and we are far behind animals in using active sonar. Perhaps creating more complex mental images from the sounds around us is another example of how we can learn from nature. With today's powerful computers, we are able as never before to simulate complicated acoustic environments and the interaction of sound with objects at high frequencies and short ranges," said Dr Taylor, "but the power of the brain in making sense of the cacophony of sounds echoing in very complex environments is exceptional and serves to inspire new research." We hope what we learn will eventually be applied to help protect dolphins who despite their acute sense of hearing, are still being trapped by fishing nets, she said. Said Dr Taylor: "We believe that through their well developed echolocation system, dolphins are able to translate sounds, particularly SONAR echoes into mental perceptions of their environment similar to those obtained through vision. By applying the same logic, perhaps humans can also benefit from this 'cross-modal' sensory experience, translating musical sound into a visual display that would help evoke feelings similar to those the sounds are intended to convey. After all, humans are able to identify objects through touch as well as vision, and can also echolocate to a limited extent and tell a cracked glass from a perfect glass by tapping it."

Dr Taylor wanted to develop this area of research into applications that might help the hearing impaired to enjoy live musical performances, first by providing a visual display that reflects the music in real time, and second by exploring ways to augment this experience by increasing the intensity of vibrations naturally produced by sound.

The idea became the research topic for PhD student, Mr Suranga Chandima Nanayakkara who graduated from NUS in 2005 with a 1st-class Honours degree in Electrical Engineering. As the research stretches across disciplines, Assoc Prof Ong Sim Heng, Department of Electrical and Computer Engineering as well as Assoc Prof Lonce Wyse, Communications and New Media Programme, are also co-leading the project. Prof Ong advises on the signals and image processing aspects while Prof Wyse and Dr Taylor primarily guide the integration of graphics and sounds.

So far, for many people with severe hearing problems, their concept of a piece of music is information obtained from music scores - something like what one would see in the computation of music notes for the automatic piano - a linear series of dots. "But what we want to achieve here is to convey not just information in a timeline kind of presentation, but an aesthetic sensation. If we are able to provide this experience, then we are getting somewhere," said Prof Wyse.

So far, Suranga has come out with an algorithm able to provide an appropriate visual to individual notes of the keyboard. For example, hit the C note, and a red circle or sphere appears on the projected screen. Hit a higher C, and the red circle will go higher up on the screen. The red circle travels to the lower screen if a lower C is hit. Red circles appear on screen in dynamic succession when the note is hit repeatedly. To give an impression of a lingering note, the circle gradually trails off the screen in a sequential arrangement, at a speed that conveys the note's duration. An "explosion" (like a firework) is also used to dramatise notes which are hit with more force and to depict majestic sequences in the music score, Base notes and percussion are represented differently, and so on.


Suranga and Dr Taylor will soon conduct a survey involving people with hearing problems, to find out what they might find useful in helping them to enjoy a concert, and the kind of images that would provide them with a sense of the music being played.

His project will have impact on the hearing impaired. During a live performance, a screen with visual images reflecting the feelings and emotions of a piece being played can be projected, for example. Special chairs reserved for the hearing impaired can also provide an added experience by giving off appropriate vibrations or rhythmic pulses that would follow the music.

"Other research findings have indicated that some deaf people do desire to learn how to play music. What we are working on might also help them in learning to play a musical instrument or perhaps sing in tune," said Suranga.

The team also hopes to develop a portable device which the hearing impaired could carry around with them to enhance their everyday experience, for example by providing them with information about someone out of sight speaking, footsteps approaching, a doorbell ringing or traffic flow. It could be something that delivers vibrations to the body or hands and might be worn like a belt or gloves, Prof Wyse said. Prof Ong is also optimistic about such a device. "The main challenge I can foresee, is analysing the sounds from the environment, and coming out with the appropriate response or display in real time. A high-speed processor will have to be employed to do that," he said.

Enhancing listening pleasure

Taking their research to wider applications, Dr Taylor said that such a device might also provide added sensations or environmental information for people with normal hearing. A rich visual display arising directly from the music as it is played could enhance the listening pleasure of the entire audience. Receiving rapid input on infrasound generated by spaces in and around certain buildings for example, which the human ear cannot hear might be helpful to architects and engineers. Or the interplay of space and sound in the chapel or concert hall which gives the organ its ethereal quality when played, conventionally sensed through pressure waves pressing against the body, could be visualised.

It's all to do with sensing things we can't see with our naked eyes or hearing things we normally cannot pick up with our ears. Perhaps insights from the marine world can be applied to help us, said Dr Taylor.