Audiovisual Perception

Our senses enable us to take in very diverse information about our surroundings, and they differ from one another in a number of features specific to their individual modes. The eye specializes in the perception of spatial structure, and the ear in the perception of temporal processes. Only in the rarest cases, however, are we confronted with sensory stimuli of only a single modality; we perceive our world through all five senses and hence multimodally. Consequently, our sense organs are not, as is often assumed, isolated from one another, since it is their synergetic interplay that gives human beings their evolutionary advantage. Irrespective of the modality, the most reliable stimulus in a given situation dominates all the others; if one sense provides too little or unclear information, all of the other senses enter as a corrective. The integration of multimodal sense stimuli in meaningful units is called multimodal integration; to a certain extent, it already occurs on the neuronal level and hence unconsciously and passively. Another often-used possibility to link stimuli beyond the boundaries between the senses is known as intermodal analogy. This involves searching consciously and actively for an amodal quality that is present in several sensory regions, such as intensity or brightness, in order to form analogies that transcend the boundaries between the senses. These mechanisms, which are described very briefly here, are, of course, just two elements in the interaction of hearing and seeing. Any consideration of audiovisual perception should at least distinguish between genuine synesthesia and other associative or emotional links between image and sound.