Coder Training and Coding Reliability
The following procedures were followed to ensure and estimate coder reliability for coding the integrated essays in terms of source use:
Segmentation of source texts into content units:The definitions of content units and guidelines for identifying them were discussed with three RAs. Each RA then independently segmented each source text into content units. The percentage agreement for the segmentation of source texts into content units was 76.66%. The percentage agreement between any two coders ranged between 68% and 95%. Disagreements were resolved through discussion and consensus (e.g., combining two content units into one unit, dividing a content unit into two units). The source texts were segmented into 118 content units (65 for the reading and 53 for the lecture).
Ranking of the importance of content units in the source texts: Each of the three RAs rated each content unit independently on a three-point scale in terms of importance. Cronbach Alpha for the ranking of the content units by the three raters was .81. Percentage agreement was 87.50%. Discrepancies were resolved through discussion. Each unit was assigned one importance score, the final score that the three raters agreed on.
Segmentation of essays into T-units: After discussion and training on T-unit segmentation, three RAs segmented the essays.Two RAs segmented and checked the segmentation of each essay. Out of 1,585 T-units identified in the 141 integrated essays (Al = 11.24T-units per essay, range 2 to 23T-units), the researchers disagreed or were not sure about the coding of 80 units (i.e., 95% agreement). Difficult cases were resolved through discussion among the three RAs.
Coding of T-units from essays: Three RAs coded the integrated essays in terms of source use using NVivo. Each RA read the coding scheme and guidelines as well as relevant references on the topic (e.g., Cumming et al., 2005; Gebril & Plakans, 2013). Next, the RAs participated in several rounds (led by the first author) of discussion, training, coding, and checking using essays from Barkaoui (2014). First, the coding scheme was discussed and tried on three essays by the RAs. Next, each RA coded another four essays independently and discussed their coding with each other and with the first author. This round revealed some inconsistencies in the way the RAs interpreted and used some of the codes. Further group discussions of, and practice with, the coding scheme resolved most of these issues. In a third round, each RA coded three new essays and discussed them with the group. Each RA then coded four new essays independently. Because inter-coder agreement for these four essays was above 80%, a decision was made to proceed to operational coding. By the end of the training sessions, each RA coded 14 training essays.
The operational coding was done in three rounds. In the first round, a randomly selected sample of 15 essays (set A) was assigned to the three RAs to code independently. The results of this coding were then compared and discussed and any discrepancies were resolved through discussion. Next, a second randomly selected sample of 15 essays (set B) was assigned to the three RAs to code independently. The results of this coding were then compared and discussed and any discrepancies were resolved through discussion. Finally, each RA was assigned a unique set of 37 essays to code independently. The overall inter-coder agreement for the 30 essays (i.e., sets A and B; 21% of 141 integrated essays in the study) coded by the three RAs was 84%, but the percentage of agreement varied between 68% (for source of content units to from both sources) and 88% (for integration style). The Pearson r correlations between the numbers of codes assigned by any two coders ranged between .67 (for summary) and .96 (for source of content units).
