Content and Source Use

Content for the Independent Task: We examined content only in relation to the independent task in terms of topic adherence. We used the Constructed Response Analysis Tool (CRAT; Crossley et al., 2016) to estimate two similarity indices for each independent essay: (a) synonym overlap between the entire essay and the prompt (for nouns and verbs) and (b) LSA similarity (LSA cosine score) between the entire essay and the prompt. LSA cosine scores are calculated using the relevant vectors in the factor analysis space (Landauer, Foltz, & Laham, 1998, p. 16). Essays that more specifically address the prompt are expected to result in higher values on both indices (Crossley & McNamara, 2013; Crossley et al., 2016).

Source Use for the Integrated Task: Source use was examined in relation to essays on the integrated writing task only. The indices and measures used in this study to examine source use were determined based on a consideration of

(a) the requirements of the TOEFL-iBT® integrated writing task, (b) the criteria included in the TOEFL-iBT¹' rating scale for the integrated task, and (c) a review of relevant literature on the assessment of source use (see Chapter 2). The TOEFL-iBT® integrated writing task asks test-takers to (a) summarize key points from the lecture and (b) relate points from the lecture to the reading (limitations of theories).The TOEFL-iBT®¹ rubric for the integrated task includes the following criteria: (a) selecting important information from the lecture and the reading;
(b) relating information from the reading to the lecture; and (c) accurate reporting of information from the lecture and reading (i.e., no misunderstanding of ideas in the lecture or reading; no omissions or inaccuracies). Based on recommendations in the literature (see Chapter 2), the requirements of the TOEFL-iBT integrated writing task, and the criteria included in the TOEFL-iBT® rating scale for the integrated task, we combined computer analysis, manual coding, and human rating to examine source use for the integrated essays as follows.

First, the Constructed Response Analysis Tool (CRAT; Crossley et al., 2016; Kyle, 2017) was used to compute lexical and semantic overlap between source texts and students’ essays. For lexical overlap, three indices were examined for each essay in relation to both source texts, based on the COCA Academic Reference Corpus: percentage of unigrams, bigrams, and trigrams in the essay that are keywords. In addition, CRAT provided two indices of semantic overlap for each essay on the integrated task: (a) synonym overlap between the entire essay and both source texts (for nouns and verbs) and (b) LSA similarity (LSA cosine score) between the entire essay and both source texts. We also counted references to source texts. Specifically, we used CLA to compute the incidence (per 1,000 words) of explicit references to the source texts (e.g., the professor/lecture/listening said; according to the reading) in each essay. References to source texts indicate to what extent the students were able to acknowledge the reading and listening input texts as the source of information in their essays (cf. Ohkubo, 2009).

We also coded the essays manually in terms of the importance of ideas from sources. The TOEFL-iBT^e rating scale for the integrated task emphasizes the importance of selecting important and relevant information from the source texts (e.g., score 5). As a result, this study examined this aspect of source use following procedures similar to those used by Hammann and Stevens (2003). The coding of source use was done using NVivo and involved four main steps as follows (see Appendix G for more details about the coding scheme and procedures used in this study):

1. Identifying and segmenting content units in each source text: Both the reading passage and the transcript of the lecture were segmented into content units using guidelines adapted from Larsen (1984) (see Appendix G).The reading text was segmented into 65 content units, while the lecture was segmented into 53 content units.
2. Rating the importance of each content unit in the source texts: Each content unit in the lecture and reading text was rated in terms of importance using a three-point rating scale from Hammann and Stevens (2003, p. 742): (1) Not very important, omit if summarizing text; (2) Of medium importance, could be included or not included in a text summary; (3) Important idea, definitely include in text summary.
3. Segmenting each essay into T-units: Each essay was segmented into T-units following guidelines from Polio (1997; see Appendix G).
4. Coding T-units in each essay: Each T-unit in each essay was coded in terms of the following four aspects using NVivo (see Appendix G for more details):

a. Source of content unit(s) in each T-unit: Whether the content unit(s) in the T-unit is (are) from (i) the lecture, (ii) the reading text, (iii) both, or (iv) not from the source texts (other).

b. Importance of content unit(s) in each T-unit: What is the level of importance of each content unit included in the essay? Based on the coding of the importance of the content units identified in the essay (i.e., step 2 above), three scores were computed for each essay by multiplying the number of content units at each level (1, 2 and 3, see step 2 above) by 1 (for level-1 content units), 2 (for level-2 content units), or 3 (for level-3 content units).

c. Integration style: Each content unit in each essay was coded in terms of how it was integrated in the essay (cf. Cumming et ah, 2006; Plakans & Gebril, 2013;Weigle & Parker, 2012). Each unit was coded as: (i) quotation, (ii) copy, (iii) paraphrase (with minor or major modification), or (iv) summary of two or more content units from the source text(s) (see Appendix G). No instances of quotations were identified in the data.

Finally, we used a modified version of Chan et al.’s (2015) rubric for evaluating reading-into-writing skills to evaluate the quality of source use in each integrated essay. The original rubric includes four criteria to evaluate source use: (a) understanding of source materials, (b) selection of relevant content from source texts, (c) ability to identify common themes and links within and across the multiple texts, and (d) use of paraphrasing and/or summarizing. Because the last criterion (use of paraphrasing and/or summarizing) was examined through manual coding of the essays (see above), only the first three criteria were included in the rating scale used in this study (see Appendix H for a copy of the rating scales and instructions for rating source use). Scores on the three criteria were averaged to obtain one overall source use score for each essay.