Multimodal Semiotics Perspective on Literature Visualization〔*〕

Kang Leiming

(School of Foreign Studies Hefei University of Technology，HefeiAnhui230009)

Ⅰ．Introduction

In the time of preference for multimodality，almost all areas of life are full of diverse images and also dominated by images．Literature is believed to have a tendency of visualization，which come entirely with color illustrations and sophisticated layout and typography，have graphically uniform，either in print or electronic media．Some critics worry that it may weaken the original language system，squeeze the aesthetic space of print culture，and move aes-thetics of language art backstage．The problem is basically that of image－language relation，which is an increasingly common concern among literary theory circle．

Multimodal semiotics refers to the study of multimodality within the theoretical framework of social semiotics．It expands language study to include all meaning－making semiotic resources．Image，writing，layout，music，gesture，speech，moving image，soundtrack are examples of modes used in representation and communication．〔1〕Each mode and the interactions between them have the potential to contribute equally to meaning．The present study of image－language relation emphasizes the limits and differences of the two modes，ignoring the interactive and complementary relation between them ．Multimodal semiotics offers a new perspective and methods for objective understanding of literature visualization．

Ⅱ．Literature in image era

Language and print media have for thousands of years been dominant in cultural communication．But in image era，the society has become“a visual phenomenon that conflates looking，seeing and knowing– a kind of‘visual turn’created through new visualizing technologies”〔2〕，which is desired to“achieve transparency，or‘immediacy’in our mediated communication．”〔3〕Screen，image and other non－linguistic semiotic resources are playing increasingly dominant role in social communication．Literature now is completely new to readers or viewers in terms of production，layout and distribution．

1．Multimodality in production of literature

It is a growing consensus that meanings are made not just through language whether as speech or as writing．Social communication always draws on a multiplicity of modes．Mode is“the culturally and socially produced resources for making meaning”〔4〕，and it includes language，image，gesture and music etc．Each mode in a multimodal ensemble is understood as“realizing different communicative work”．〔5〕In most cases，these modes don’t provide communication in its own right．Instead，they interact with and complement each other to create meaning in context．

Literature with a wealth of image，language and color is entirely common practice in the image era．Monologue and Gesture by Pan Jun is full of paintings，photos and sketches．These images are not jus in a supportive relation to language．Instead，with the distinct narrative means，they function semiotically in the shaping of a coherent novel as a whole．Historically，traditional literary text has never been created only by using language as semiotic resources．The Classic of Mountains and Seas，an encyclopedic record of various beliefs held by the ancient Chinese about the world，has a version with drawings and other semiotic modes in Jin Dynasty．In Tang Dynasty，there are painting poetry and poetic painting．In Ming and Qing Dynasty，adding images in novels is a common practice in the printed culture．

2．Multimediality in the means of literature distribution

There are a variety of reasons for literature visualization，one of which lies in the media of communication．Medium is“the term for the culturally produced means for distribution of these representations－as－meanings，that is，as messages．”〔6〕In the image era，the most popular and easily accepted literature is always the one distributed through more accessible and all－pervading media．

In history，oracle bone，wooden tablets and bamboo slips used to be the means by which writing is done．With the invention of paper，print literature reading becomes the most popular and wide－spreading communication because of its convenience and cheapness．But today，new media have played a major role in the emancipation of the image，which is certainly in part related to the great ease anybody“can manipulate images… and integrate them with text by means of authoring software．The resulting multimodal texts can then be distributed over networks to multiple recipients with equal ease．”〔7〕Featured with directness，convenience and instant consumption，electronic media can bring the audience strong visuality through its vivid immediacy．Therefore，most literary works have been adapted to films and TV series，and there are more and more literary works and net novels stored in Cellphone，E－books and IPAD，which enables more audience to“consume”the literature art．

3．Visualization of language forms in literary works

With the visualization of mass media，literature bears more visual property in its inner narration manner as well as its outer medium form．To be specific，taking in more expressive factors from the semiotic system，the language of literature is visualizing．W．J．T．Mitchell，a leading figure in visual studies，puts forward verbal image specially which is about image of language or Verbal Icon．Mitchell states clearly that the language image involves with literature criticism，and divides the image of language or Verbal Icon into Metaphors and Descriptions．〔8〕Literature itself contains the tendency of verbalization．Through some figures of speech like metaphor and description，the abstract literature language presents itself in a vivid manner．〔9〕For instance，description can be done about the image of one’s feelings，which enables the readers to have a specific image of the characters or events．The narration takes on a look of considerable senses:vivid sketch of characters，visualization of scenes，flowability of story，good－looking plots and astonishing audio－visual effects．By means of electronic technology，their words seem to have shape，color and sound．

Ⅲ．Multimodal semiotics:social semiotics perspectives on multimodality

Multimodal semiotics looks beyond language to analyze multiple modes of communication and meaning－making within the theoretical framework of social semiotics，which has been influenced by the Systemic Functional Linguistics(SFL)developed by Michael Halliday and his social semiotic approach to language and his model with its three metafunctions(ideational，interpersonal and textual)．The systematic study of multimodal semiotics goes back to Roland Barthes，so it endows with rich color of semiotics from the very beginning．Since the end of last century，the most active and leading figures in this field have been function－oriented semioticians．Gunther Kress and Theo Van Leeuwen propose visual grammar，thinking that non－linguistic images，like language，can be configured to realize the same representational，interactional and compositional functions，and they do so “in their own，pictorial ways”．〔10〕Language and non－linguistic modes，like image，color and music etc．，have the same function in communication and have the potential to contribute equally to meaning．

Multimodal research provides robust tools“for analyzing and describing the full repertoire of meaning－making resources which people use to communicate and represent… and how these are‘organized’to make meaning．”〔11〕Representation and communication draw on a multiplicity of modes rather than on language alone．People orchestrate meaning through their selection and configuration of modes．Meanings of multimodal semiotic resources are shaped by the norms and rules operating at the moment of sign－making，influenced by the motivations and interests of a sign－makers in a specific social context．〔12〕Multimodal semiotic resources can be deployed independently and interactively in meaning－making and communication．

Ⅳ．Multimodal semiotics perspective on literature visualization

Literary works，either page－based or screen－based，are increasingly designed as coherent multimodal ensembles by means of layout，involving a complex interplay of written text，images and other graphic，color or sound elements．Reading literature is no longer“necessarily linear，wholly or in part，but may go from centre to margin，or in a circular fashion，or vertically，etc．”〔13〕It forces scholars to think about“the particular characteristics of these modes and the way they semiotically function and combine in the modern discourse worlds．”〔14〕The selection of modes and media are influenced by the motivation creation and interests of readers or viewers in a specific social context．Different modes have the features，functions and other aspects which overlap or cannot be substituted．The overall meanings are constructed through the interaction and complement among modes and media．

1．Image’s relative Independence in Making Meaning

Historically，image－language competition is always there．There seems to be no compromise about people’s choice concerning the dominance of either image or language in meaning－making and communication，in which there are clearcut boundaries．As Kress and Van Leeuwen put it，“Language and visual communication can both be used to realize the same fundamental systems of meaning that constitute our culture，but that each does so by means of its own specific forms，does so differently，and independently．”〔15〕Verbal texts are structured by linguistic norms and rules，while images make meaning by principles of visual composition，modality and framing．Visual modes are configurated into overall entities through an organic and coherent system，“connected with the verbal text，but in no way dependent on it–and similarly the other way around．”〔16〕

It is believed that human beings have had pictographic representation and communication long before the emergence of the earliest scripts such as the cuneiform script of the Sumerians in Mesopotamia around 3500 BC．Writing systems such as Chinese character are considered to have pictographic origins．〔17〕Even in the printed page，the layout and different type sizes have potentials for making meaning，serving to create pleasing visual patterns．For example，on the title page of Sir Thomas Elyot’s The Boke Named the Gouernor printed in 1534，the word“the”was printed in the largest type to create an inverted pyramid pattern．〔18〕Dating back to ancient China，scholars recognized image’s independence in meaning－making．Wang bi，for example，has a penetrating elucidation of the relationship between yan，xiang and yi，which are three key categories in classical Chinese philosophy and poetics．Xiang is viewed as“a relative independent cognitive sign，a concept close to image in modern semiotics”．〔19〕

2．Interactions Between Modes in Making Meaning

It is a growing consensus that multimodal texts drawing on more than one semiotic resource can apparently do more than language alone，and“are assuming their central role in information dissemination in the modern world”〔20〕Without the division of semiotic labor，text with language by itself or with image alone would not work in terms of meaning－conveying and communication．The two modes，image and language，are not simply alternative means of realizing the same communicative work，and it is easy to overemphasize either the similarity or the difference between them．In fact，the meanings in any mode“are always interwoven with the meanings made with those of all other modes co－present and‘co－operating’in the communicative event．The interaction between modes is itself a part of the production of meaning．”〔21〕Modes are often configurated intentionally to satisfy the need of message－conveying in specific social context．

There is no denying that images and language have their specific meaning potentials，but meaning representation and communication are always the result of the interaction or“the simultaneous orchestration”〔22〕among different semiotic modes．Some kinds of images may be better at creating direct emotional impact，and language may be more suited to carrying out logical analysis．〔23〕Despite the definite match between the signifier and the signified，language is slightly weaker than image due to its abstractness and association nature in projecting meaning．Although featured with directness and vivid immediacy，visual meaning is too indefinite．Roland Barthes argues that all images，by themselves，are too“polysemous”，too open to a variety of possible meaning．Visual meaning implies a“floating chain”of signifieds，readers able to choose some and ignore others．To arrive at a definite meaning，language must come to the rescue to help“fix the floating chain of signifieds in such a way as to counter the terror of uncertain signs”〔24〕Image－language interactions in making meaning are embodied in the means of complementing and strengthening，in which one mode extends the meaning of the other，or vice versa;one elaborates the other，or vice versa，based on which literary works now are shaped．

Historically，images have long been closely nested in the construction of literature．Any attempt to exclude images from literature would be work in vain．Literature has a diachronical evolution from “co－present of language and images”to“image－language separation”to“image－language conflation”．〔25〕The conflation or separation are to serve the needs of meaning construction influenced by motivation creation and interests of readers or viewers in a specific social context．

In ancient China，specifically early in Wei，Jin，Northern and Southern Dynasties，language and images are both used as significant artistic signs to convey aesthetic connotation and achieve aesthetic appeal．The readers or viewers are influenced by both verbal reading and audio－visual effects，bringing them to the age of image－language co－reading．Ancient Chinese poetry，for example，have a distinct preference for visual effects，the typical demonstrations of which are the blending of poems and paintings such as painting poetry and poetic paintings since Song Dynasty and Yuan Dynasty．Illustrations in novels and drama in Ming Dynasty and Qing Dynasty are also the blend product of language and visual signs．

Literature，the avant－grade of the“high culture”arts，use an increasingly variety of materials and come entirely towards“multimodal Gesamtkunstwerke，multimedia events”．〔26〕It is more popular and more easily accepted．To be exact，this new literature form needs no long－time reading，even needs no in－depth understanding and careful thinking or reflection necessary in traditional reading process．Image serves as strong force in literature popularization．

Ⅴ．Conclusions

A simple fascination with producing the visual literature lies in the desire to achieve transparency and immediacy in the mediated communication．It is“author’s rethink of literature in image era rather than an attempt of literature self－rescue．”〔27〕Boundaries between modes become increasingly blurred，spurred by the relentless pace of developments in information technology．Images and layout are in-creasingly meshed in the construction of content．Superficially，verbal and written text is giving way to a greater and greater proportion of visual images and combined visual designs by means of layout．But this is not a valuation of images over language．In fact，“it is now no longer possible to understand language and its uses without understanding the effect of all modes of communication that are co－present in any text．”〔28〕Taking a fresh look at the problem from the perspective of multimodal semiotics，we can see that the basic changes in literature are not about the role language plays but about mode－choosing orientation and potentials that language and image offer for making meaning．The emergence of new tendency of literature is neither a threat nor a decline of language arts，instead it extends potentials of literature communication in terms of mode of thinking，means of representation and layout．As Kress states，“traditional pages will continue to exist，… The elites will continue to use writing as their preferred mode，…—because the two modes are apt means for doing so”．〔29〕

〔1〕Kress，G．，What Is Mode?In Jewitt，C．(Ed．)， The Routledge Handbook of Multimodal Analysis，London and New York:Routledge，2009，p．54．

〔2〕〔5〕〔11〕〔12〕〔21〕Jewitt，C．， The Routledge Handbook of Multimodal Analysis，London and New York:Routledge，2009，pp．3，14，15．

〔3〕〔7〕〔23〕Martinec，R．a(chǎn)nd Salway，A．，A System for Image－Text Relations in New(and Old)Media，Visual Communication，2005，4，pp．338，339．

〔4〕〔6〕〔29〕Kress，G．，Gains and losses:New forms of texts，knowledge，and learning， Computers and Composition，2005，22，pp．5－22．

〔8〕劉若愚:《中國(guó)詩(shī)學(xué)》，鄭州，河南人民出版社，1990年，第121頁(yè)。

〔9〕陸濤:《從語(yǔ)象到圖像—論文學(xué)圖像化的審美邏輯》，《江西社會(huì)科學(xué)》2013年第2期，第73－78頁(yè)。

〔10〕〔13〕〔18〕Kress，G．a(chǎn)nd Van Leeuwen，T．， Reading Images，Victoria:Deakin University Press，1990，pp．21，119，120．

〔14〕Ventola，E．，Charles，C．a(chǎn)nd Kaltenbacher，M．， Perspectives on Multimodality，Amsterdam:John Benjamins，2004，p．1．

〔15〕〔16〕Kress，G and Van Leeuwen，T．， Reading Images:The Grammar of Visual Design(2nd Edition)，London:Routledge，2006，pp．19－20，18．

〔17〕Chen Yumin， Interpersonal Meaning in Textbooks for Teaching English as a Foreign Language in China:A Multimodal Approach(PhD thesis)，Sun Yat－sen University，2009，pp．1－2．

〔19〕趙憲章:《語(yǔ)圖傳播的可名與可悅—文學(xué)與圖像關(guān)系新論》，《文藝研究》2012年第11期，第24－34頁(yè)。

〔20〕〔22〕Bateman，J．A．， Multimodality and Genre:A Foundation for the Systematic Analysis of Multimodal Documents，Hampshire:Palgrave Macmillan，2008，pp．2，1．

〔24〕Barthes，R．， Image－Music－Text，New York:Hill＆ Wang，1977，pp．37－39．

〔25〕趙憲章:《文學(xué)和圖像關(guān)系研究中的若干問(wèn)題》，《江海學(xué)刊》2010年第1期，第183－191頁(yè)。

〔26〕Kress，G and Van Leeuwen，T．， Multimodal Discourse:The Modes and Media of Contemporary Communication，London:Arnold，2001，pp．1－2．

〔27〕劉巍:《圖像時(shí)代的文學(xué)取向》，《當(dāng)代作家評(píng)論》2011年第6期，第173－180頁(yè)。

〔28〕Kress，G．，Multimodality:Challenges to Thinking About Language， TESOL Quarterly，2000，Vol．34，2，pp．337－340．