Method

This study aimed to examine changes in the linguistic and discourse characteristics of essays written by a group of L2 learners in response to TOEFL-iBT® writing tasks after a period of English language instruction. Data were obtained from the study by Ling et al. (2014) and consisted of the scores and essays of a sample of 85 Chinese learners of English as a foreign language (EFL), who each responded to TOEFL-iBT® writing tasks twice, before and after a period of English language instruction.This study addressed the following research questions:

• 1. What aspects of the learners’ writing, if any, change after a period of English language instruction?
• 2. To what extent and how do the linguistic characteristics of learners’ written responses vary across tasks?
• 3. To what extent and how do the linguistic characteristics of learners’ written responses vary depending on learner initial English language proficiency (ELP)?
• 4. To what extent and how do changes in the linguistic characteristics of learners’ essays across test occasions vary depending on task type and learner initial ELP?

Dataset

Data for this study were obtained from the study by Ling et al. (2014). Ling et al. collected data from two samples in two programs, an international high school in China (N = 480) and an intensive English program in the USA (N = 127). A subsample (TV = 111, 90 in China and 21 in the USA) took the test twice, before andafter 6—9 months of English language study. Only data from the China sample who took the test twice (i.e., TV = 90) are included in this study.

According to Ling et al. (2014, pp. 3—4), all the students in the China sample were native Chinese speakers, and half of them were males. They were all enrolled in general English courses as part of K-12 English education required by the government. Some participants were finishing their first-year studies at the time of the second test, while others were finishing their second-year studies at that time.

Ling et al. (2014, p. 4) described the data collection procedures for the China sample as follows. Students took the first TOEFL-iBT® practice test at the beginning of the school year and the second one at the end of the school year (i.e., after 9 months). All the students took the two practice tests voluntarily but were not aware of each test until one or two weeks before the test date. Between the two tests, the participants took the high school English classes required by the general educational guidelines in China, together with classes on other subjects, such as math, Chinese language arts, physics, biology, and chemistry. In addition, they took TOEFL-iBT® preparation courses and exercises during the second half of the nine months. The English-related course work was fewer than 15 hours a week on average.

Each participant responded to the same independent and integrated writing tasks before and again after instruction. The independent writing task consisted in writing an essay about a general topic (30 minutes), while the integrated task consisted in listening to a two-minute, 150—250-word lecture and reading a 200— 300-word text about the same topic and then writing a summary of both the lecture and the reading (20 minutes). The integrated task asks the students to “summarize the points made in the lecture you just heard, being sure to specifically explain how they cast doubt on points made in the reading’’ (ETS, 2007, p. 284). The reading, listening, and writing components of the integrated task focus on the same topic. The reading text consists of one main idea and three supporting ideas, while the lecture includes ideas that either support or oppose the ideas in the reading. The two writing tasks used in the study are representative of operational TOEFL-iBT®¹ writing tasks. The e-ratet® scoring engine was used to score all the essays (Ling et al., 2014).

Data for this study consisted of copies of the written responses of the 90 Chinese high school students to TOEFL-iBT® independent and integrated writing tasks at the beginning and end of the school year, for a total of 307 essays. The original plan was to collect 360 essays (i.e., 90 students x 2 writing tasks x 2 test occasions), but some participants did not complete all tasks. In addition, some participants submitted a blank response, copied the topic, or wrote a few words, often irrelevant to the task. All these responses were excluded from the sample for this study. At the end only 276 essays, written by 85 participants (41 female and 44 male), were included in this study. Table 4.1 provides a description of the dataset included in this study. As Table 4.2 shows, not all 85 participants completed both tasks on both test occasions. Fifty-three (53) students completed the independent task

TABLE 4.1 Number of Participants Who Completed Each Writing Task at Each Test Occasion

TABLE 4.2 Distribution of Participants by ELP Level for the Full and Longitudinal Samples

twice and 58 completed the integrated task twice; only 48 participants completed both tasks on both test occasions.The result is an unbalanced design with incomplete data for 37 participants. Such unbalanced design makes it difficult to compare results across tasks and test occasions. In order to address this problem, all statistical analyses (e.g., descriptive statistics, analysis of variance; see below) were conducted twice, first for the full sample of 85 students and again for the subset of 48 students who completed all tasks on both test occasions. Below we refer to the first sample as the “full sample” and the second as “the longitudinal sample.” Results are then compared across the two samples and conclusions are drawn based on patterns supported by both sets of analyses.

In order to measure the initial overall English language proficiency (ELP) of each participant, a composite score was computed for each student by summing their scores on the reading, listening, and speaking sections of the TOEFL-iBT^e at test occasion 1 and then dividing the total by 3 (i.e., maximum possible score = 30). The composite scores at test occasion 1 ranged between 2.33 and 27.67 (M = 10.81, SD = 5.51).The participants were then divided into four groups based on their composite scores at test occasion 1. The number of students in each ELP level for each sample is shown in Table 4.2.