Discourse Aspects

To examine discourse, each essay was analyzed in terms of coherence, cohesion, text organization, and argument quality.

Coherence and Cohesion: We used a combination of human rating and computer analysis to measure coherence and cohesion. First, Coh-Metrix was used to examine four dimensions of cohesion (connectives, coreference cohesion, conceptual cohesion, and givenness) at three levels: local, global, and text (Crossley et al., 2016; McNamara et al., 2014).

• • Local Cohesion (i.e., cohesion between adjacent sentences):
• • Connectives: Incidence score for all connectives.

• • Coreference Cohesion: Argument-, noun- and stem-overlap for adjacent sentences were highly correlated in the data (r > .70). Consequently, only argument overlap for adjacent sentences is reported below.
• • Conceptual Cohesion: LSA (Latent Semantic Analysis) overlap for adjacent sentences, which measures how similar each sentence is to adjacent sentences, was computed for each essay.
• • Global Cohesion (i.e., cohesion between paragraphs):
• • Coreference Cohesion: Argument-, noun-, and stem-overlap for all sentences were highly correlated (r > .70). Consequently, only argument overlap for all sentences was included. Note that argument overlap for all sentences and argument overlap for adjacent sentences were highly correlated (r= .86).
• • Conceptual Cohesion: We examined LSA overlap for all sentences in a paragraph.
• • Text Cohesion (i.e., cohesion throughout an entire text):
• • Conceptual Cohesion: We examined LSA overlap for adjacent paragraphs.
• • Givenness: LSA given/new.

Second, we used the coherence and cohesion rating scales developed by Chiang (1999, 2003) to evaluate the essays in this study (see Appendix B for a copy of the rating scales and instructions for rating coherence and cohesion). The cohesion subscale used in this study included four items from Chiang (2003) that concern (a) the signaling of relationships between and within sentences through the use of tense, aspect, and junction; (b) how important or new content is introduced; (c) the appropriate use (or non-use) of punctuation in grouping and ordering propositions; and (d) the appropriate use of pronouns. Each item was rated on a four-point scale (strongly disagree to strongly agree). Scores on the four cohesion items were averaged to obtain one overall cohesion score for each essay.

The coherence subscale included eight items from Chiang’s (1999) coherence rating scale. These items concern (a) the effectiveness of the first section of the essay in helping the reader anticipate forthcoming messages and facilitating his/ her understanding of the text; (b) the relevance of the essay content to the topic; (c) the appropriate and effective grouping of all relevant propositions in the middle section of the essay to foster an orderly sense within and across idea groups; and (c) the effectiveness of the final section of the essay in announcing to the reader that the discourse is coming to a close (see Appendix B). Each item was rated on a four-point scale (strongly disagree to strongly agree). Scores on the eight coherence items were averaged to obtain one overall coherence score for each essay.

Text Organization: Text organization was measured using two methods: computer analysis and human rating. Specifically, the web-based program Criterion (Ramineni et al., 2012;Weigle, 2011) was used to measure the organization of each essay. Criterion automatically identifies sentences in each essay that correspond to each of five discourse elements: Introductory material (background).

thesis statement, main idea, supporting ideas, and conclusion (Ramineni et al., 2012;Weigle, 2011). Criterion identifies whether each essay includes each of the five discourse elements. For organization, we looked at the presence of three discourse elements: Introductory Material,Thesis Statement, and Conclusion.

In addition, an analytic rating scale, based on the holistic rubric for scoring text organization developed by Kubota (1998a), was used to rate the essays in terms of the quality of three aspects of text organization: Main idea, reader orientation, and organization. The main idea scale concerns whether a main idea is stated clearly and effectively in the essay. Reader orientation refers to whether the writer attends to the readability of what he/she writes for the audience, such as whether the writer provides relevant background information for the reader to understand the main point of the essay. Organization refers to the way in which an essay is structured and addresses the question of whether the essay follows a discernible structure/plan. Each of the three criteria was rated on a four-point scale (see Appendix C for a copy of the rating scales and instructions for rating text organization). Scores on the three criteria were averaged to obtain one overall text organization score for each essay.

Argumentation Quality for the Independent Task: The quality of argument structure in each independent essay was rated in terms of six features using a four-point rating scale from Cumming et al. (2005,2006; cf. Connor & Lauer, 1988; McCann, 1989; Toulmin, 1958/2003): thesis, claims, data, warrants, recognition of opposition, and response to opposition (see Appendix D). Thesis refers to the thesis of the essay or what is being said about the question and is evaluated in terms of whether it is clear, complete, and relevant to the question in the writing task. A claim is an assertion that the argument backs up and is evaluated in terms of whether it is clear, complete, and relevant to the thesis. Data or evidence is evaluated in terms of whether the writer gives supporting details that are clear, complete, and relevant to the claims. A warrant holds the argument together, linking the data to the claim. Warrants are evaluated in terms of their explicitness, soundness, clarity, and relevance. Warrants may not be required if the connection between the data and claim is self-evident. Recognition of opposition is evaluated in terms of whether the writer recognizes opposition that is clear, complete, and relevant to the claims. Finally, responses to opposition are evaluated in terms of whether the writer offers a response to opposition that is clear, complete, and relevant to the opposition (see Appendix D for a copy of the rating scales and instructions for rating argument quality). Scores on the six measures of argument quality were averaged to obtain one overall argument quality score for each essay.