Contributions and Implications
Despite its limitations, the study makes several contributions and points to several areas for further research. First, a key contribution of the current study is that it provides an initial description of the patterns of change in TOEFL-iBT- essay characteristics across test occasions and how these changes relate to test-taker initial L2 proficiency and task type. Previous longitudinal studies have examined only changes in overall test or writing scores. The findings of this study indicated that some features (e.g., fluency, source use) do change across test occasions, while other features (e.g., lexical sophistication) do not.These findings suggest that some aspects of L2 writing can develop considerably within nine months of L2 study, while other aspects take longer (and/or require more instruction) to develop. Consequently, test-takers (and their instructors) need to be aware that some writing aspects take longer to develop and that test-takers need to take this into account before attempting the test again.The findings also suggest that some essay characteristics may contribute more to score variance across test occasions than do other aspects. It is also possible that some writing features contributed differently to variability (across tasks and learners) in essay scores at different test occasions. The study also provides detailed, empirically based descriptions of the writing features of TOEFL-iBT® essays and how they vary across tasks and ELP levels.
Second, the study provides empirical evidence in relation to three warrants underlying the explanation inference in the TOEFL-iBT- validity argument. First, the findings indicate that changes in test-takers’ L2 proficiency after a period of English language instruction are reflected in changes in the characteristics of their written texts which are in turn reflected in changes in their writing scores. Second, comparisons across the independent and integrated tasks indicate that the two tasks elicit different types of discourse, which justifies the inclusion of both tasks in the test in order to “broaden the representation of the domain of academic language of the test’’ (Knoch, Macqueen, & O’Hagan, 2014, p. 68). Third, the findings show that essays produced by participants with different levels of initial ELP differed in terms of some aspects of writing as would be expected by theory. Together, these findings provide support for the warrants underlying the explanation inference in the TOEFL-iBT- validity argument. As noted above, however, some aspects of writing examined in the study did not exhibit substantial changes across test occasions or differences across tasks or ELP proficiency groups. In addition, some writing aspects examined in this study are not included in the TOEFL-iBT* rating criteria (e.g., register). These findings suggest that (a) the TOEFL-iBT- writing section does not cover the whole construct of L2 writing ability as defined in the framework used in this study and (b) the test is not sensitive to changes and differences in all aspects of writing, including some aspects that the test aims to assess. Lack of sensitivity to change might also be because these aspects of writing did not exhibit true change in the period under study.
Third, the study adopted a model that is theoretically sound and empirically grounded to analyze the essays. This model combined different methods (manual coding, computer analysis, and human rating), which allowed detecting changes (across occasions) and differences (across tasks and ELP groups) in terms of various writing features at the micro- and macro-levels and both quantitatively (i.e., frequency of particular features such as errors, content units from source texts) and qualitatively (e.g., holistic judgment of the effects of errors on essay comprehensibility or the quality of source use). Computer analyses of individual linguistic features can detect fine-grained changes (and differences) in specific features (e.g., use of passive voice) that human ratings do not detect or reflect, while human ratings can detect changes (and differences) that fine-grained analyses of individual linguistic features cannot detect such as the simultaneous change in multiple linguistic features that, when considered separately, do not show substantive change, but, when considered together, as when reading and evaluating a piece of writing, can considerably enhance the overall quality of the essay as perceived by the reader or rater (e.g., register ratings). The model adopted in this study can be used in future longitudinal and cross-sectional studies to examine changes and differences in L2 writing performance across time, tasks, contexts, and learner groups.
The study points to several implications for future research on TOEFL-iBT® writing tasks. First, future studies need to examine changes in TOEFL-iBT9 essay characteristics in relation to L2 instruction in different contexts and different study periods. Ling et al. (2014), for example, found that ESL students who completed an intensive six-month English course in the USA exhibited more improvement in their TOEFL-iBTe writing scores than did the EFL students included in this study (who studied English for nine months at high school in China). Future studies could compare the nature, magnitude, rate, and direction of changes in writing features (and writing scores) across contexts (e.g., ESL vs. EFL), courses of various intensity and length (e.g., 3, 6, 9, or 12 months), study levels (e.g., high school, undergraduate, graduate), and learners from different backgrounds (e.g., age, L2 proficiency level, LI). One of the questions to be addressed is whether and how the nature and rate of change of writing features and scores vary across contexts and learner groups. Such research needs to be longitudinal, to assess writing at least at three time points (Barkaoui, 2014), and to combine linguistic analyses with qualitative methods (e.g., interviews, observation) in order to find out not only what aspects of L2 writing (e.g., writing scores, linguistic features, writing processes, motivation, and goals) change, but also why and how L2 learners’ writing develops over time and in relation to L2 instruction (cf. Norris & Manchon, 2012; Polio, 2017; Polio & Park, 2016).
While the primary purpose of TOEFL-iBT®1 is to assess test-takers’ English language proficiency at one point in time, some programs may use TOEFL-iBT9 to measure L2 writing development or progress over time and/or in relation to instruction. In order to be able to use the test to make valid claims about L2 writing development, more research needs to be conducted that examines the test’s sensitivity to change and instruction effects. Because sensitivity to change is a function of task requirements, rating scale criteria and levels, and rater training and rating procedures, all these aspects of the test need to be investigated longitudinally. This study examined changes in essay linguistic characteristics, but not changes (if any) in how essays are rated. Consequently, there is a need for more research on whether and how the rating scales and rating procedures of TOEFL-iBTe can detect true changes in L2 writing ability over time and which aspects of the rating scale are sensitive to L2 writing development and which may not be. Future studies need also to examine whether the relative importance of different criteria on the rating scale changes over time and whether such changes are due to true changes in the combined effects of multiple features on the overall quality of an essay or to other (construct-irrelevant) factors such as changes in rater’s interpretations of the rating criteria and/or reactions to writing tasks and test-taker characteristics (e.g., LI). For example, raters may become used to the writing of candidates from a particular L1 background and, as a result, they may become more or less severe when marking essays written by test-takers from that background. In this case, changes in writing scores would not reflect true changes in test-taker writing ability as much as changes in raters’ perceptions. This line of research can inform revisions of the rating scale, rating procedures, and rater training for TOEFL-iBT1' writing tasks in order to improve the test’s sensitivity to both differences across test-takers and changes over time in test-takers’ L2 writing abilities.
Because the current study included e-rater scores, it was not possible to examine the relationships between changes in essay characteristics and changes in essay scores across raters and across occasions for the same rater. Future studies could examine the relationships between changes in essay linguistic characteristics and the ratings assigned by individual raters across test occasions. These studies may also look into whether and how the rating criteria and processes that raters employ vary over time (cf. Barkaoui, 2010a).
Several writing indices were examined in this study. As shown in Tables 9.1 and 9.2, some of these indices detected changes in L2 writing performance over time (Column 1 of Table 9.1) and differences in L2 performance across tasks (Column 2 of Table 9.1) and ELP levels (Column 3 ofTable 9.2). Fluency, for example, was significantly related to the three independent variables examined in this study (i.e., occasion, task, and ELP group). Accuracy ratings were sensitive to differences across ELP groups, but were not associated with occasion or task type, suggesting that overall accuracy, as measured by holistic ratings, does not vary significantly across tasks or change significantly after nine months of English language instruction, although the number of errors declined after instruction. Similarly, several indices of syntactic and lexical complexity tended to vary as a function of task type, but not occasion or ELP level. Thus, it seems that some indices are better at detecting changes in L2 writing performance over time, while others are better at detecting differences across task types and/or ELP levels.
Some indices, however, did not vary significantly across tasks, ELP groups, and/or test occasions.The academic formula list (AFL), for example, was not significantly associated with any of the independent variables in this study. Knoch, Macqueen, and O’Hagan (2014) also found that AFL did not vary significantly across task types or score levels. Future studies need to examine the sensitivity of the writing indices used in this study to changes over time and differences across tasks and ELP levels (assuming there is true change/difference to be detected) with L2 learners from different backgrounds and who are studying English in different contexts over different periods of study to address questions such as: do some indices take longer to show significant changes? Do learners with different levels of initial ELP (e.g., beginner vs. advanced) exhibit different patterns of changes over time in terms of these indices? Do the patterns of change over time in these writing indices vary (assuming there is true change/difference to be detected) across contexts (e.g., ESL vs. EFL)? Such research can identify which indices are most sensitive to change over time and differences across tasks for various contexts and learner groups. Findings from this line of research can inform the design and validation of L2 writing tasks, rating scales, and automated scoring systems that are sensitive to changes in L2 writing ability and can be used to assess L2 writing development over time and in relation to L2 instruction. Currently, there is little to no research on indicators of change or development in L2 writing ability that can inform the development and validation of tests to assess L2 writing development over time. As noted above, most L2 proficiency tests aim to assess L2 proficiency at one point in time. However, there is a growing need to develop and validate tests that can detect change in L2 ability over time in order to assess L2 development and/or the effectiveness of L2 instruction.
Finally, the study has implications for test validation and second language acquisition (SLA) research. First, the study has demonstrated how L2 writing development can be examined using a variety of writing measures and indices. Specifically, the various measures and indices used in this study allowed the detection of differences across tasks, ELP groups, and individual students in terms of specific writing features as well as the examination of changes in these features over time. Theoretically, the findings of the study describe some of the linguistic features that distinguish writing performance at various levels of achievement in L2 learning; identify some of the linguistic features of L2 learners’ texts that change after nine months of English language instruction; and describe how differences in initial L2 ability relate to changes in the linguistic characteristics of L2 learners’ texts. These analyses were supported by Connor and Mbaye’s (2002) framework which provides a theoretically grounded conceptualization of L2 writing ability. This framework was operationalized in this study to examine variability in the linguistic characteristics of test-takers’ essays across tasks, test-takers, and test occasions (cf. Barkaoui & Knouzi, 2012). But the framework could be used in longitudinal studies to examine the relationships between (a) amount and nature of English language instruction, (b) changes in learner English language proficiency, (c) changes in the linguistic characteristics of L2 learners’ texts, and (d) changes in their writing scores over time. Such a program of research can greatly improve our understanding of the nature of, and the individual, contextual, and instructional factors that shape L2 writing development in different contexts.
