Discourse Aspects

Discourse or textual knowledge refers to “how sentences are organized to form texts” and includes knowledge of cohesion, coherence, and text and paragraph organization (Bachman & Palmer, 2010, p. 45; Connor & Mbaye, 2002). As explained above, we also include argumentation under discourse (cf. Coffin, 2004; Cumming et al. 2005; Fulkerson, 1996; Hillocks, 2010).

Coherence and Cohesion. Following Halliday and Hasan (1976), Crossley and McNamara (2016) defined cohesion as “the presence or absence of explicit cues in the text that allow the reader to make connections between the ideas in the text” (p. 354). Crossley and colleagues (2016, p. 2) explained that generally these cues are local in nature (i.e., cohesion at the sentence level), but they can also be based on global or text cohesion, as explained below. While cohesion is “specific to the text,” coherence is “specific to the reader and refers to the understanding that the reader derives from the text (i.e., the coherence of the text is in the mind of the reader)” (Crossley & McNamara, 2016, p. 354; cf. Chiang, 1999;Todd et al., 2004). As such, coherence depends on a number of factors, including explicit and implicit cohesion cues as well as “non-linguistic factors such as [the reader’s] prior knowledge and reading skill” (Crossley & McNamara, 2016, p. 354; cf. Chiang, 1999;Todd et al., 2004).

Cohesion and coherence play an important role in text comprehension and ratings of students’ writing in LI and L2 (Crossley et al., 2015; Crossley, Kyle, & McNamara, 2016; Crossley & McNamara, 2016). For example, an increase in text cohesion generally leads to greater comprehension of a text. In addition, several studies have shown that text coherence is an important indicator of essay quality. However, as Crossley and colleagues noted (e.g., Crossley et al., 2015; Crossley, Kyle, & McNamara 2016; Crossley & McNamara, 2016), these studies show that while indices of global cohesion (e.g., semantic similarity between paragraphs) are significantly and positively correlated with judgments of text coherence and overall quality, measures of local cohesion often fail to demonstrate significant correlations with ratings of text coherence or overall quality and those that do are negatively correlated. Finally, the use of cohesion devices seems to change as learners’ proficiency develops (Crossley & McNamara, 2016). Crossley and McNamara (2016) cited research indicating that young writers often use local, explicit cohesion devices (e.g., referential pronouns and connectives) to link sections of text together, but around grade 8, these writers begin to use fewer explicit cohesion cues to organize text. For adolescent and adult writers, the use of explicit local cohesion cues is generally associated with less proficient writing. For instance, less proficient adult writers tend to use a greater repetition of words, have greater word overlap between sentences, and use more connectives than more proficient writers. For adults, it is usually global, rather than local, cohesion that distinguishes the writing of proficient writers.

One widely used tool to examine cohesion and coherence in written texts is Coh-Metrix. Coh-Metrix is a web-based program that provides more than 200 computational linguistic indices of syntactic complexity, text coherence and cohesion, and word diversity and characteristics (see McNamara et al., 2014). As McNamara et al. (2014) explained, Coh-Metrix measures four key dimensions of cohesion, connectives, coreference cohesion, conceptual cohesion and givenness, at three levels: local, global, and text (Crossley, Kyle, & McNamara, 2016; McNamara et al., 2014). Local cohesion is measured by “assessing the overlap between consecutive, adjacent sentences, whereas global cohesion is assessed by measuring the overlap between all of the sentences in a paragraph or text” (McNamara et al., 2014, p. 63). Connectives provide explicit cues to the types of relationships between ideas in a text, thus, providing important information about a text’s cohesion (Crismore et al., 1993; Halliday & Hasan, 1976; McNamara et al., 2014). Five types of connectives are usually examined: causal (e.g., because, therefore), additive (e.g., also, moreover), temporal (e.g., after, first), adversative/contrastive (e.g., however, although), and logical (e.g., and, or) (Crossley et al., 2009, 2011; McNamara et al., 2014).

McNamara et al. (2014) explained that “coreference is a linguistic cue that can aid readers in making connections between propositions, clauses, and sentences in their textbase understanding” (p. 63). Coreference cohesion occurs when a noun, pronoun, or noun phrase refers to another constituent in the text (Crossley et al., 2007; McNamara et al., 2010). There are four main types of coreferentiality that vary in terms of the explicitness of the overlap: noun overlap, argument overlap, stem-overlap, and content word overlap. Noun overlap measures the proportion of sentences in a text for which there are overlapping nouns, that is, how often a common noun of the exact same form is shared in two sentences (e.g., study/ study but not study/studies). Argument overlap measures how often two sentences share common arguments (i.e., nouns, pronouns and noun phrases) (e.g., study/ study, study/studies, it/it). Stem-overlap measures how often a noun in one sentence shares a common morphological stem with other content words (i.e., nouns, verbs, adjectives, adverbs; e.g., study, studies, and studying, or studied), but not pronouns, in another sentence.

Conceptual cohesion concerns the extent to which the content of sentences or paragraphs is similar semantically or conceptually. The main measures of conceptual cohesion are based on Latent Semantic Analysis (LSA). LSA is a statistical, corpus-based technique that provides an index of local and global conceptual cohesion and coherence between parts of a text by considering similarity in meaning, or conceptual relatedness, between and within parts of a text (i.e., sentences,paragraphs) (Crossley et al.,2008;Crossley et al.,2009,201 l;McNamara et al., 2007; McNamara et al., 2014). LSA, utilizing factor analysis, works by creating matrices in which texts (e.g., paragraphs) are represented in columns and words in rows. Each cell represents the frequency of each word in each text. Then, through transformations, LSA calculates the frequency/weight of each word relative to each text and the conceptual clustering of words in texts (Landauer et al., 1998). Unlike lexical markers of coreferentiality (i.e., noun and argument overlap), LSA provides for the tracking of words that are semantically similar, but may not be related morphologically. Text cohesion (and sometimes coherence) is assumed to increase as a function of higher conceptual similarity between text constituents (Crossley et al., 2007; Landauer et al., 2007).

Finally, “givenness reflects the amount of information that is recoverable from the preceding discourse” (Crossley et al., 2015, p. 5).To assess givenness, an LSA given/new index is computed for each text.This index is “a proxy for how much given versus new information exists in each sentence in a text, compared with the content of prior text information” (McNamara et al., 2014, p. 66). When there is more given information in a text and less new information, the index approaches 1. When there is less given information, then the index approaches 0 to indicate that there is lower cohesion (McNamara et al., 2014, p. 67). Givenness indices, calculated at the text level, have demonstrated positive relations with measures of text coherence in previous studies (Crossley & McNamara, 2011; Crossley et al., 2015).

Crossley and colleagues (e.g., Crossley et al., 2015; Crossley, Kyle, & McNamara 2016; Crossley & McNamara, 2016) have used a computerized, corpus-based approach to analyze cohesion and coherence in written texts. However, Schiftner (2013) has warned that such an approach is limited and called instead for the use of “a multiple-perspective approach to the analysis of coherence in learner writing” (p. 275). Specifically, she cautioned that “because the exact relationship between cohesion and coherence is not clear, ... coherence of learner writing cannot be judged by measures of cohesion alone” (p. 268). Instead, researchers should combine human holistic ratings of coherence and the analysis of cohesive elements on the textual surface level in order to “allow for the consideration of aspects of both form and meaning that contribute to coherence” (p. 266). Consequently, it is advisable to combine computer analyses and human rating when examining the cohesion and coherence of learners’ texts.

Chiang (1999, 2003), for example, developed rating scales for coherence and cohesion based on models of communicative competence (e.g., Bachman, 1990; Canale, 1983) to assess the “effectiveness and appropriateness in the organization of ideas above the sentence level and among different parts of the text” (p. 222). The cohesion subscale consists of statements pertaining to the degree of compactness or efficiency with which ideas within individual sentences are related to each other. The coherence subscale developed by Chiang (1999, 2003) pertains to discourse-level features that are responsible for giving prose its overall organization. The focus in this scale is on the relationships among ideas represented in complexes of propositions or paragraphs.

Text Organization is the basic structure of a written text (Plakans, 2014) or how ideas and support are ordered (Di Gennaro, 2009) and contributes significantly to text coherence (Hirose, 2003; Kubota, 1998a).Text organization is usually evaluated in terms of its quality using human rating (e.g.. Di Gennaro, 2009; Kubota, 1998a). Another approach to evaluate text organization is whether a text includes specific components such an introduction, a thesis statement, and a conclusion (e.g., Hirose, 2003; Kubota, 1998a; Storch & Tapper, 2009). This can be done manually or using a computer program.

Argumentation Quality. The most common approach to examining argument quality in writing builds onToulmin’s (1958/2003) work.To evaluate the quality of arguments. To ulmin, Rieke, and Janik (1984, p. 81) proposed examining the structural soundness and substantive strength of the argument. Structural soundness has to do with the presence and quality of three elements: claims, data, and warrants, and how coherently they are connected. As such, assessing soundness entails identifying each discrete argument element and assessing its quality. Substantive strength concerns acknowledgment of opposition to the claims one makes (also called rebuttals) and responding to this opposition, by rejecting or refuting the opposition, or changing, modifying, or qualifying (hedging) one’s claims (Toulmin et al., 1984, pp. 81—2). Assessing substantive strength, thus, involves identifying rebuttals, responses to rebuttals, and qualifying words as discrete linguistic units and assessing their quality. Similar approaches have been used in several studies on L2 learners’ texts (e.g., Cumming et ah, 2005; Connor, 1991; Connor & Lauer, 1988; Hadi di, 2017).