Statistical Analyses

Data for this study consisted of essay scores and the indices of linguistic features listed in Table 4.1 for each essay. The scores of the essays were obtained using e-rater. Before conducting any statistical analyses, the data were examined to check if they met two statistical assumptions: normality and homogeneity of variance, using Kolmogorov-Smirnov Tests and Levene’s test of equality of error variance, respectively (Field, 2009). It was found that almost all variables (for each task and test occasion) did not meet one or both of the assumptions. As a result, a decision was made to use non-parametric tests to address the research questions of the study, using the computer program PERMANOVA+ (Anderson et al., 2008), an add-on package for the statistical analysis software PRIMER (Clarke & Gorley, 2015).

PERMANOVA+, originally developed for use in ecology (Anderson, 1999, 2001; Anderson et al., 2008; Legendre & Anderson, 1999; McArdle & Anderson, 2001), can perform various types of univariate and multivariate analysis of variance (i.e., ANOVA, MANOVA) that rely on resemblance (or distance or dissimilarity) measures and can handle data that “do not fulfil the assumptions of traditional statistical approaches because they have too many response variables, or because one or more response variables are not normally distributed” (Anderson et al., 2008, p. 6).These methods can also handle data that do not meet the assumption of homogeneity of covariance and/or include many zeros in the data matrix (Legendre & Anderson, 1999, cited in Cumming et al., 2005). Another advantage of PERMANOVA+ is that it can analyze outcome variables measured on continuous or dichotomous scales.

Like other non-parametric methods, the routines in PERMANOVA+ are “based on measures of distance or dissimilarity between pairs of individual multivariate observations” (Anderson, 2001a, p. 33) but construct a test statistic based on the ranks of dissimilarities similar to the F-ratio in ANOVA. P-values are then obtained using an appropriate permutation procedure for each term (Anderson, 2001b; Anderson & ter Braak, 2003; Anderson et al., 2008). Anderson et al. (2008) explain how tests by permutation work:

The PERMANOVA F ratio should be thought of as a “pseudo” F statistic, because it does not have a known distribution under a true null hypothesis. An appropriate distribution for the pseudo-F statistic under a true null hypothesis is obtained by using a permutation (or randomization) procedure. The idea of a permutation test is this: if there is no effect of the factor, then it is equally likely that any of the individual factor labels could have been associated with any one of the samples. The null hypothesis suggests that we could have obtained the samples in any order by reference to the groups, if groups do not differ. So, another possible value of the test statistic under a true null hypothesis can be obtained by randomly shuffling the group labels onto different sample units.The random shuffling of labels is repeated a large number of times, and each time, a new value of pseudo-F is calculated ... If the null hypothesis were true, then the pseudo-F statistic actually obtained with the real ordering of the data relative to the groups will be similar to the values obtained under permutation. If, however, there is a group effect, then the value of pseudo-F obtained with the real ordering will appear large relative to the distribution of values obtained under permutation ... Power and precision increase with increases in the number of permutations. To draw inferences at a significance level of 0.05, P-values should be calculated using at least 999 permutations, whereas 4,999 permutations should be done to draw inferences at a level of 0.01.

(Anderson et al., 2008, pp. 19—24)

Anderson et al. (2008) explained that one advantage of the permutation p-values is that they “provide an exact test of each individual null hypothesis of interest” (p. 20, emphasis in original). Anderson et al. (2008) explained that

an exact test is one where the probability of rejecting a true null hypothesis is exactly equal to the a priori chosen significance level. Thus, if [alpha] were chosen to be 0.05, then the chance of a type I error for an exact test is indeed 5%.

(P- 20)

PERMANOVA+ also performs a posteriori pair wise comparisons among levels of factors, including within individual levels of other factors (or cells) in the case of significant interaction effects (Anderson et al., 2008). For a detailed discussion of the PERMANOVA+ methods see Anderson (2001a), McArdle and Anderson (2001), and Anderson et al. (2008). For a discussion of permutation techniques, see Anderson (2001b) and Anderson and ter Braak (2003).

The data in this study formed a 2 (occasion) ^x 2 (task type) ^x 4 (proficiency level) factorial design, with student nested in proficiency level as follows:

• 1. Factor 1: Occasion (fixed effects, with two levels: before and after instruction);
• 2. Factor 2: Task type (fixed effects, with two levels: integrated and independent);
• 3. Factor 3: English language proficiency (ELP; composite score) at occasion 1
• (before instruction) (fixed effects, with four levels: VL, L, M, and H);
• 4. Factor 4: Student (random effects, nested in ELP, N = 85 for full sample, N = 48 for longitudinal sample).

Note that factor 2 (task type) was not included in some analyses when some writing features (i.e., argument quality, content, source use) were examined for one task only.

To avoid multicollinearity, we excluded variables that are highly correlated as follows. First, we examined the inter-correlations among indices measuring the same writing aspects. As noted above, if two or more indices were highly correlated (i.e., r > .70), we kept only one of the indices and deleted the others.We also examined the correlations among all the indices and measures in the study. Only nine correlations were above .70; six of them were above .80; and only one (between C/T and MET) was above .90. These high correlations are reported in Chapters 5—8. Additionally, outliers for composite and essay scores and each index were identified using x-scores. A z-score > 3.29 was considered to indicate an outlier (Field, 2009).There were no outliers for the composite and essay scores and several indices. However, some indices had between one (e.g., syntactic similarity, AWL) and five outliers (e.g.,T/S) out of 276 cases for the full sample and out of 192 cases for the longitudinal sample. Given that no outliers were identified for essay scores, that the number of outliers was small for all indices, and that the sample was small, particularly for the longitudinal sample, a decision was made not to exclude any cases from the analyses.

We first standardized all the data to x-scores in order to put all measures on the same scale (with a mean of 0 and standard deviation of 1; cf. Cumming et al., 2005). We then used Euclidean distances to measure the magnitudes of the dependent variables as recommended by Anderson (personal communication, November 23, 2017; Anderson et al., 2008). To address the research questions, we conducted a multivariate analysis of variance (MANOVA) to examine the effects of occasion, task type, ELP level, student (nested within ELP), and their interactions on each set of indices for each writing feature (e.g., syntactic complexity, cohesion, and coherence) separately. To test for significant differences, we conducted F tests based on 9,999 restricted permutations of the standardized raw data. Because the study included a large number of dependent variables (relative to the study sample size), we reduced the threshold for the significance value from the commonly used p <.05top <.01 (cf.Knoch et al., 2014). IfMANOVA indicated a significant effect, a non-parametric univariate ANOVA was conducted, using PERMANOVA+, on each dependent variable involved in the significant effects separately. When a significant main effect for ELP and/or a significant interaction effect was detected, we conducted follow-up pair wise comparisons, using non-parametric t-tests, among levels of ELP or within individual levels of factors (or cells) involved in significant interaction effects.

As noted above, all analyses were conducted twice, first for the full sample (N = 85) and again for the longitudinal sample (N = 48). Below we report three types of results: (a) descriptive statistics (across occasion, tasks, and ELP levels) for the full sample, (b) results from PERMANOVA analyses for the full sample, and (c) line graphs for significant interaction effects for the full sample. Descriptive statistics for the longitudinal sample are reported in Online Appendix 1. For all analyses of variance, we report the exact p- values obtained as well as the effect size for significant effects using partial eta-squared (partial rf). Partial rf > .01 indicates a small effect size; partial rf > .09 indicates a medium effect; and partial rf > .25 indicates a large effect (Field, 2009).

To illustrate how results are reported by PERMANOVA+, we report complete PERMANOVA results for essay scores below (Table 4.8). For the other variables (Chapters 5—8), (M)ANOVA results are reported only for the significant main and interaction effects. Because most of the results were consistent across the full and longitudinal samples, below we include line graphs and discuss descriptive statistics only for the full sample, unless the results are different across samples.

Where the results vary across samples, both sets of results are discussed in the text and line graphs for significant interaction effects for the longitudinal sample are included too. Because the tables reporting descriptive statistics include more than 150 cells each, cells that include means involved in significant main and/or interaction effects are highlighted in gray for the full sample (and for the longitudinal sample in Online Appendix 1) for ease of reference.

Before describing changes in the writing features of the participants’ texts, we describe changes in the participants’ composite and essay scores here. Because each analysis included only one dependent variable (composite score or essay score), non-parametric ANOVA was conducted. Comparison of composite scores across test occasions for the full sample indicated that there was a significant difference (F(l, 275) = 59.78, p = .0001, Ip = .18). Generally, participants obtained significantly higher composite scores after instruction (Af = 14.84, SD = 6.86) than they did before instruction (Af = 10.81, SD = 5.51). For the longitudinal sample, the difference was also significant with a larger effect size: F(l, 184) = 109.04, p = .0001, I]² = .37. The 48 participants in the longitudinal sample obtained significantly higher composite scores after instruction (Af = 18.17, SD = 6.33) than they did before instruction (Af = 12.08, SD = 5.82).The correlation (Spearman’s rho) between composite scores across test occasions for the longitudinal sample was .60 (p < .01).

Table 4.7 reports descriptive statistics for essay scores by task, occasion, and ELP group, while Table 4.8 reports the PERMANOVA results for essay scores for the full sample. Table 4.8 shows significant large effects for ELP group (F(3, 275) = 22.57, p = .0001, rp = .84) and task type (F(l, 275) = 20.61, p = .0001, rp = .39), but not occasion (F(l, 275) = 5.93,p = .019). However, for the longitudinal sample, there were significant effects for occasion (F(l, 191) = 18.64,p = .0003, /)' = .43), task type (F(l, 191) = 10.23,p = .0019, rp = .29) and ELP group (F(3, 191) = 16.88,p = .0001, rp = .84).The 48 participants in the longitudinal sample obtained significantly higher essay scores (Af = 2.91) on test occasion 2 than they did on test occasion 1 (Af = 2.35). For both samples, the participants obtained significantly higher scores on the independent task (Af = 2.62 for the full sample) than they did on the integrated task (Af = 2.21). Overall, participants with higher initial ELP obtained higher essay scores than did participants with lower ELP across test occasions for both samples. All pair wise comparisons for ELP groups were significant at p < .01, except for the difference between the medium and low groups which was not statistically significant.

Table 4.8 also shows that there was a significant main effect for student (nested within ELP group) and a significant student by occasion interaction effect for the full sample. These significant effects indicate that students within ELP groups varied significantly in terms of their writing scores and that individual students exhibited significantly different patterns in terms of score changes across test occasions. Comparison of writing scores across test occasions for those participants who completed the independent task before and after instruction

TABLE 4.7 Descriptive Statistics for Essay Scores for Full Sample

TABLE 4.8 PERMANOVA Results for Essay Scores for Full Sample

(N = 53) indicated that the independent task scores of 32 participants increased (by 1 to 2 points out of 5); the scores of 14 participants did not change, while the scores of the remaining seven participants declined (by 1 to 2 points). For the integrated task, out of 58 participants who responded to the integrated task before and after instruction, the scores of 25 participants increased (by 1 to 3 points); the scores of 19 participants did not change, while the scores of 14 participants declined (by 1 to 3 points).

The next four chapters report the results concerning the writing features examined in the study. Given that statistical analyses were conducted for each set of writing feature separately, the results of these analyses are organized and reported by writing feature, starting with grammatical aspects in Chapter 5. Each of Chapters 5 to 8 starts with a brief discussion of the writing aspect under focus and then reports the results for that aspect. In each chapter, results are reported for the full sample (N = 85). Chapter 9 summarizes and discusses the key findings for all writing features in relation to the research questions of the study.