Intersemiosis

Intersemiosis in the classroom

The study of the teacher’s multimodal classroom orchestration entails the examination of the ways in which the various embodied semiotic modes, such as language, gestures, positioning, and movement, as well as semiotic technologies, combine in their interaction and integration, that is, in their intersemiosis. O’Halloran (2005: 159) proposes the term ‘intersemiosis’ to describe ‘the meaning arising across semiotic choices’. Intersemiosis has also been observed by Royce (1998, 2006) as ‘intersemiotic complementarity’ where ‘visual and verbal modes semantically complement each other to produce a single textual phenomenon’ (Royce, 1998: 26). Lemke (1998) notably describes the meanings arising from the combination of semiotic resources in scientific discourse in textbooks as a ‘multiplying of meanings’. Hence, the ‘emergent meaning’ (Lim, 2004) arising from the combined use of the semiotic resources is more profound than the meaning made by an individual semiotic resource. While this is generally recognised, what is of interest in intersemiosis is the ways, or mechanisms, in which the semiotic resources work in combination to produce the emergent meaning.

As described in Chapter 1, the approach to the study of multimodal pedagogic discourse in this book is informed by the work in linguistics, in particular, of Michael Halliday’s Systemic Functional Theory. Halliday (1978,1994(1985]) conceives language as a social semiotic system and has developed a robust theoretical apparatus, the systemic functional grammar, to systematise the meaning potential of language and foreground the notion of choice in meaning-making.

The extension of Systemic Functional Theory as a productive approach to investigate other semiotic resources has been shown through its extension through the Systemic Functional Multimodal Discourse Analysis (SFMDA) approach (O’Halloran & Lim, 2014; Jewitt, Bezemer, & O’Halloran, 2016) as well as the social semiotics lens on teaching and learning (Kress, 2009; Kress & Bezemer, 2015) adopted in this book. The SFMDA approach focuses on the ‘grammatics’ of semiotic resources in fulfilling specific functions and how the semiotic choices work in combination to make meaning over space and time. A key aspect in the SFMDA approach is its study of'multimodal grammatics, where interacting systems of meaning are a key motif’ (Jewitt et al., 2016: 39). As such, how the semiotic resources in intersemiosis interact and integrate are of particular interest and are central to the SFMDA approach.

In this section, we will discuss the ways in which intersemiosis across various semiotic resources happen. In particular, we will consider the basis of the combination, that is the contextualising relations across the resources. The purpose of this is to provide a language for educational researchers to describe the patterns of meanings made, as well as understand the interaction and integration of the repertoire of semiotic resources, used by the teacher in her multimodal classroom orchestration.

Thibault (2004) explains that different semiotic resources are organised according to different principles. Thibault (2004: 26) observes that for example ‘language is predominantly typological-categorical; it is based on discrete categorical contrast or difference. Gesture, on the other hand, is topological-continuous; it is based on continuous variation of visual and spatial relations’. As such, language and gesture ‘do not simply express the same meanings by alternative means of expression. Instead, they make different meanings on the basis of their different principles of organization’. Baldry and Thibault (2006: 4) also explain that ‘different modalities adopt different organisational principles for creating meaning’. Hence, it is necessary to examine the specific semiotic resources in focus within the multimodal text and explore the unique ways in which they combine and interact in their joint co-deployment.

Matthiessen (2009) describes the orchestration of multimodal semiotic resources as‘semiotic harmony’. Matthiessen (2009: 11) observes that‘one interesting — and critical — aspect of the division of semiotic labour among the denotative semiotic systems is the extent to which they operate in semiotic harmony with one another’. Nonetheless, like Baldry and Thibault (2006), Matthiessen (2009) acknowledges that how this is accomplished is unique to different resources. Specifically of interest is ‘how the different resources for creating meaning complement one another and how the semiotic labour (the work of creating meaning in context) is divided among them’ (Matthiessen, 2009: 23).

The intersemiosis between the semiotic modes of language and images has been most theorised to date. Systems and theories have been proposed by many researchers to investigate language—image relations (see, for example Lim, 2004; Royce, 2006; Lim, 2007; O’Halloran & Lim. 2009; Daly & Unsworth. 2011; Painter et al., 2011; Lim & O’Halloran, 2012; Bateman, 2014). In particular, Unsworth and Cleirigh (2009) argue that ‘advancing understanding of how images and language interact to construct meaning seems crucial in seeking to reconceptualise literacy and literacy pedagogy from a multimodal perspective’. In Lim (2011; 2019), the intersemiosis between language and gestures is examined and discussed with reference to the intersemiosis between language and images. Jiang and Lim (submitted for publication) extend the study beyond the intersemiosis of two semiotic modes to investigate the interaction and integration across three semiotic modes, examining the meaning patterns across language, images, and gestures.

Contextualising relations

Intersemiosis between semiotic resources arises from the contextualising relationships, which are the meanings made from the spatial co-occurrences or temporal co-occurrences of the semiotic resources. As Thibault (2000: 362) explains, it is ‘on the basis of co-contextualizing relations that meaning is created’. Extending this idea, Lim (2004: 239) distinguishes two types of contextualising relations:

In cases where the meaning of one modality seems to ‘reflect’ the meaning of the other through some type of convergence, the two resources share cocontextualizing relations. On the other hand, in cases where the meaning of one modality seems to be at odds with or unrelated to the other, their semantic relationship is one that creates divergence or dissonance. In the latter case, the resources share re-contextualising relations.

The intersemiosis between language and images has been studied from the perspective of contextualising relationships (O’Halloran & Lim, 2009; O’Halloran et al., 2019). In examining the contextualising relationships in mathematical discourse, O’Halloran (2007: 95) explains that ‘[ijntersemiosis creates new semantic layers where the meaning of the re-contextualised ideational relations extend beyond that possible with language... Co-contextualising textual and interpersonal relations enable the new ideational content to be foregrounded’ (O’Halloran, 2007: 95). Co-contextualising relations bring about semantic convergence, where through the similar meanings made across the various semiotic resources, the emergent meaning is that of reinforcement, through repetition and redundancy. For example the teacher may signal gesturally a thumbs-up sign when praising a child for the good work done.

Re-contextualising relations, on the other hand, create new semantic layers through the reconciliation of the divergent meanings made.This emergent meaning usually carries new semantic layers such as irony, sarcasm, dilemma, and ambivalence. Thibault (2000: 321) explains that ‘ideological disjunction’ results from ‘the complex, often intricate, relations of inter-functional solidarity among the various semiotic resource systems that are co-deployed’. Re-contextualising relations lead to semantic divergence. However, the semantic divergence can be reconciled when the additional layer of meaning is recognised as sarcasm. For example the teacher may shake her head in mock disapproval, when she says ‘yes’ to a child seeking permission to be excused from the class to visit the restroom.

For communication to be successful, both semantic convergence and divergence through the combination of semiotic selections must be reconciled in the emergent meaning to ultimately serve the intent of the message. Re-contextualising relations leading to a semantic divergence that cannot be reconciled result in ambiguity, confusion, and a consequent breakdown in communication.

The notion of semiotic cohesion can be helpful to describe the combination of semiotic resources in their co-deployment in multimodal classroom orchestration. Semiotic cohesion was first proposed by O’Halloran (2008) to describe language—image relations. Semiotic cohesion is where ‘system choices function to make the text cohesive’ (O’Halloran, 2008: 453). Liu and O’Halloran (2009) argue that it is important to recognise ‘a coherent multimodal message, rather than a cooccurrence of language and images’ (Liu & O’Halloran, 2009: 367). They explain that‘from the perspective of logogenesis, semiotic cohesion can be regarded as the ongoing process of contextualization, in which meanings are made across different semiotic resources in multimodal discourse’ (Liu & O’Halloran, 2009: 385).