Multi-Modal Displays

We experience our surrounding with the five human senses: sight, smell, hearing, taste, and touch. Due to evolutionary reasons, they all have their own right to exist, all of them play an important role in how we perceive our environment. Although it is possible not only to survive but live comfortably without one of the other senses, every sense has its specific application area in which it performs best. Its substitution by other senses always means to either lack valuable information, or to significantly increase cognitive load. Although we have five senses, it is obvious that the visual sense tends to be predominant in our conscious perception. Two examples for this phenomenon are the dominance of visual arts in our culture and, also, the current enthusiasm for taking pictures with digital cameras.

On the contrary, other senses are far less important in the view of many people, an observation that can be supported by the small number of people owning e.g. a sound recording device or a digital smell keeper. This is not only due to the technology for smelling devices that is not yet ready for portable devices, but also because people tend to actively perceive only visual cues; other senses are primarily processed in the subconscious part of our mind. This fact, however, does not imply that these modalities are less important for our decision-making, yet their influence is more subtle. From this perspective, also the history in the development of data analysis and data exploration tools make sense: Data is almost always represented in visual graphs or plots, even time-based measurements such as EEG data or audio are often decomposed into frequencies and then turned into visual representations.

The use of other modalities such as sound, however, is not widely accepted in the research community. It often is sufficient to show a graph of data to proof a thesis. This is by no means based in any objectively measurable significance of visual displays that our other senses might lack, but only supported by the fact that we tend to believe more in what we have visually perceived.

A multi-modal representation of data makes sense for exploratory tasks because every sense that is involved adds its characteristic structure recognition and analysis abilities. It allows users to perceive formative multi-modal experiences, rather than joining the value of all incorporated senses. The whole is perceived as something different than the sum of its parts.