ACCORDING TO THE MODALITY PRINCIPLE, USE OF AUDIO narration to explain a visual presentation enhances understanding and related problem solving, when practical. Specifically, cognitive theory and research indicate that spoken words (audio) can best be used to describe a graphic or how to use a visual component rather than simultaneously delivering that graphic along with the same words in written form. However, audio is often overlooked in favor of printed text when developing e-learning projects because it can be more difficult or expensive to implement than text. 

While it may be more efficient or cost effective to simply present graphics and written words in a multimedia presentation, Clark and Mayer propose that presenting "words as onscreen text to describe a graphic conflicts with the way the human mind works." (Clark and Mayer. p. 89) Based on cognitive learning theory, they point out that people process visual/pictorial and auditory/ verbal information in separate channels, each with a limited capacity to acquire new information at any given time.

When graphics are presented along with explanatory text, both are competing for the same limited visual channel. Essentially, the viewer cannot look at the written description and the graphic at the same time. Using a verbal explanation as speech instead of text reduces the demand on the visual channel.

Cognitive research shows that people process pictorial information in a separate channel from auditory/ verbal information. Since the capacity of each channel is limited, graphics and supporting onscreen text vie for limited visual bandwidth as the viewer cannot look at graphics and supporting text simultaneously. Conversely, when verbal information is presented in audible form it enters the cognitive system in a different channel where it can be comfortably processed simultaneously. Tests have shown that subjects presented with animation and simultaneous narration generated from 41 to 114 percent more solutions than those presented with animation and onscreen text - even though the information presented was identical.

Click on Play Button > to Start

When considering the implications of the modality effect, it is also necessary to be aware of circumstances that may impact its application. These circumstances might include the ability of the hardware to support audio, whether the environment where the information will be viewed is appropriate for audio, budgetary limitations, language limitations and the hearing ability of the audience.

When used gratuitously or redundantly (for self-evident graphics), audio can actually cause a distraction. In some cases, it may be advisable to keep verbiage available as an option in printed form as a memory aid or as an alternative source for the hearing impaired.

A corollary of the Modality Principle is the Redundancy Principle, which is also based on our understanding of cognitive overload. Research (Clark and Mayer. p. 97) has demonstrated that presenting words as audio narration and text simultaneously can actually have an adverse effect on learning.

Mousavi, S., Low, R., and Sweller, J. (1995) Reducing cognitive load by mixing auditory and visual presentation modes. Journal of educational psychology, 87, 319-334.

Susan Connell
Graduate Student
SDSU Educational Technology

Connell, S. (2003). The modality principle. In B. Hoffman (Ed.), Encyclopedia of Educational Technology. Retrieved July 15, 2010, from file:///D:/实验室/eet/articles/modality/start.htm