Reading Images: Multimodality, Representation and New Media – Kress

“Each mode forces me into making certain kinds of commitments about meaning, intended or not. The choice of mode has profound effects on meaning…” (111).

Kress, Gunther. “Reading Images: Multimodality, Representation and New Media.” Information Design Journal & Document Design 12 2 (2004): 110-19.

In this 2004 article, continues his discussion of multimodality and representation (addressed in my last post). He presents his discussion from he perspective of semiotics and specifically from that of multimodality, “which deals with all he means we have for making meaning–the modes of representation–and considers their specific way of configuring the world” (110).

Meanings are always disseminated through particular media: the medium of the boo; or the CD-ROM, involving still and moving images, speech, writing, cartoon-like characters in comic strips, music, and so on. It might be the medium of the teacher’s body, involving speech, movement and gesture. All media offer specific possibilities to the designer, and to the reader/user in their reading and/or use. (111)

In other words, how information is represented is relative to the mode. For example, one can present a picture of an object to explain it (the mode is a still image), or through video, or text, or oral statement, etc. Each method of sharing/explaining the information represents it is a different manner, it makes meaning differently, and the learner therefore receives it and makes meaning from it differently. What causes one to perceive and make meaning in a certain way given a certain mode is based on personal experience, learning style, interest, etc. Kress would argue that all of these elements are society and culture based. “[M]eanings always relate to specific societies and their cultures, and to the meanings of the members of those cultures” (111). This point is all based on a social semiotics approach. Therefore, affected by our own cultures, we chose forms and modes in a way that we feel will best represent specific information for a specific setting. “[S]igns always express, through their form. The meanings that the makers of signs have wished to make” (111).

Of course, there are limitations to the meaning that can be conveyed through certain (and multiple) modes. It is the task of the social semiotic approach to identify those limitations as well as the potentials of the delivery method.

Speech uses the material of (human) sound; writing uses the material of graphic substance. There are things you can do with sound that you cannot do with graphic substance, either easily or at all; not even imitate all that successfully graphically. The up and down of the voice, which produces the melody of (English) speech, makes many meanings, from straightforward questions to highly modulated ones… (112).

This is point very much supports that which I have discussed many times over in regard to the online video conversation (OVC). There are features and modes of each communication method that provide certain features and benefits (as well as downfalls). The increase/decrease of voice intonation and volume relays emotion that cannot be gleaned in writing. Similarly, facial expression and gestures convey information about the presenter’s emotion and intent. It is important to identify and discuss what affordances like this are provided by each mode and each communication method, particularly in specific settings that might provide different elements to such discussions.

Speech – temporal and vague
Continuing on the topic of speech, Kress offers that “Speech happens in time: one sound, one word, one sentence follows another. The ‘logic’ of temporal sequence is the major principle of ordering of languages such as English” (112). In other words, to say “The dog was hungry. He ate all the food.” suggests something quite different from “He ate all the food. The dog was hungry.” So, sequence is important it implies causality. In my first example, it suggests that the dog ate all the food because he was hungry; the second example suggest that perhaps someone else ate all the food, leaving the dog hungry or perhaps the dog ate it all but was still hungry. Kress goes on to note that words are vague. In the example I provided above, we really do not know anything about the dog, such as color, breed, size, etc. A picture of the dog would likely answer those questions. Granted, we might not have any idea that he is hungry or ho ate the food; but we might, particularly if an empty bowl and bag of food were included.

Therefore, we can often glean more from an image than from text. This point also plays into the visual presence of a speaker who uses facial expression and gesture. In this case, there is a speech element, so one receives the spoken content as well as the visual (and audio) emotion with it. However, more modes in not necessarily better; it is more a point of finding the appropriate mode of communication for the situation and audience. Kress notes that the current social world is marked by fragmentation and individuation. (118). Unlike the 19h and 20th century that had more clearly defined structures and placement of genres and authors within those structures as well as clearer choices for appropriate modes, these structures and genres are less clearly defined. For example, as previously discussed, the idea of what makes an author is not obvious now given that anyone can publish anything on the web.

In periods of stability the question of effective communication is answered by the idea of convention and of competent action in relation to those conventions. In periods of fragmentation and individuation communication is fraught: each environment of communication asks that the social and ‘political’ relations, tastes, needs and desires be newly assessed. (119)

Given our fragmentation and the plethora of communication options available, it become increasingly important to question and select appropriately the mode to effectively convey a certain message to a specific audience in a specific setting. “The question of rhetoric – how to make my communication most effective in relation to this audience, here and now – has moved newly, urgently into the center” (119).

Leave a Reply