18a. Seeing with your ears: Peter Meijer's vOICe I

     We saw in the previous page how we can to a certain extent "hear" language sounds with our eyes. For people with normal hearing, this means that what we think we hear can be strongly influenced by the mouth movements we see. And we know that trained deaf or hearing-impaired people can to varying degrees understand what others are saying by lip reading 讀唇 or "speech reading". (Here's another short! and interesting reference; here is a page on how to get started lipreading; this reference on speechreading is longer and more detailed.)

     Deaf people will naturally use the other senses available to them to experience the world, in particular, sight, especially in the form of sign language and lip reading. So what will blind people use? They usually rely heavily on hearing, e.g. regular and synthesized speech, and environmental sounds; and touch, e.g. a walking cane and Braille 點字. Although blind people can get an idea of some of the objects in their surroundings, for example, by the sounds of car engines, doors shutting, and bird calls, they cannot experience the overall appearance and structure of where they are, or of a picture in a book. But there are now several technologies which attempt to help them experience "vision" through other methods or senses. One is the Dobelle implant, which gives a very rough sketch of a visual scene through a grid produced by electrodes surgically attached directly to the brain. Another is a system for using electrical impulses to produce a similar grid through a tactile display overlaid on the tongue.

     We have learned about spectrograms, and how they "paint a picture" of human speech. If you take a spectrogram and input it into a pattern playback machine, you can reconstitute the graphic image into the original sound. (This possibilitiy is not a foregone conclusion – it might have been the case that a spectrogram could show an analysis of speech sounds, but you couldn't convert it back into speech.)

     Imagine now that you have a spectrogram you want to translate back into sound. But instead of being a "picture" of human speech, the spectrogram is instead a real "picture" of your surroundings, or anything else you could photograph with a camera. What would happen if you put this photograph into a spectrographic playback machine? What kind of sound would you hear? Remember that the higher a mark is on the spectrogram, the higher the pitch or frequency; the darker a mark is, the louder it will sound; and the more to the right in the spectrogram a mark is, the later in time it occurs.

     Think about all this for a while. Then, when you are ready, go on to the next page for a deeper exploration of this fascinating idea.
     

Next: Seeing with your ears: Peter Meijer's vOICe II (with applet for drawing soundscapes)

on to next page        back        index I       index II       home