3.3 Seeing using sound's mapping algorithm

The mapping algorithm is the piece of the system that takes in an edge-detected image, and produces a sound clip representing the image. The mapping as we implemented it takes three steps:

• Vertical Mapping
• Horizontal Mapping
• Color Mapping

Vertical mapping

The first step of the algorithm is to map the vertical axis of the image to the frequency content of the output sound at a given time. We implemented this by having the relative pitch of the output at that time correspond to rows in each column that have an edge. Basically, the higher the note you hear, the higher it is in your field of vision, and the lower the note, the lower in your field of vision.

Horizontal mapping

Next, we need some way of mapping the horizontal axis to the output sound. We chose to implement this by having our system "sweep" the image from the outside-in in time (see figure 1). The reasoning behind this is that the focus of the final sound should be the center of the field of vision, so we have everything meeting in the middle. This means that each image will have some period that it will take to be "displayed" as sound. The period begins at some time t0, and, with stereo sound, the left and right channels start sounding notes corresponding to edges on each side of the image, finally meeting in the middle at some time tf.

Color mapping

Using scales instead of continuous frequencies for the notes gives us some extra information to work with. We decided to also try to incorporate the color from the original image of the point at an edge. We were able to do this by letting the brightness of the scale that we use. For example, major scales sound much brighter than minor scales, so bright colors correspond to major scales, and darker ones correspond to minor. This effect is difficult to perceive for those that aren't trained, but we believe that the brain can adapt to this pattern regardless of whether or not the user truly understands the mapping.