Students' Interaction with TE Features

With respect to TE features, students’ use of drag-and-drop, zone selection, audio buttons for listening, recording buttons for speaking, and typing for writing was of particular interest in this study. In the kindergarten sample, 60% of the students were rated as having some difficulties with these TE features at least once across the items (see Figure 7.1). The interviewers’ notes indicated that the standard-sized mouse was too big for students in kindergarten and Grade 1; this was the major obstacle to students performing a drag-and-drop. Students often had difficulties simultaneously pressing and holding the button and moving the mouse for the drag-and-drop, particularly when the object and the dropping zone were relatively long distance from each other (e.g., across the computer screen). One interesting observation was that some students in Grade 3 and above switched to using a track pad instead of a mouse during testing as they were more familiar with and accustomed to using track pads in their school. The use of a track pad made the drag-and-drop a little harder for these students compared to using a mouse.

In addition, the interviewers regularly commented that the younger students (kindergarten and Grade 1) were still developing their motor skills, let alone the ability to use a mouse adequately. For example, interviewers commented that the size of the clickable targets on the computer interface seemed too small for these young students who were not precise in their clicks of the mouse and had difficulties with a relatively small target size.

A similar problem was found in the "Literary Set” item type in reading. This item type included a relatively long passage, requiring the test takers to scroll down to view the entire text. Some interviewers considered this a TE feature issue and assigned a score point of 1 to the students who had difficulty using a mouse and grabbing the scroll bar. All interviewers reported that the scroll bar seemed too narrow and that the color was too discreet for these young students.

As presented in Figure 7.1, a significantly higher percentage of the sample in Grades 2—3 was rated “no” difficulties for the use of TE features (73%) than in Grades K and 1. The students in Grades 2-3 also demonstrated easy use of the audio buttons to replay the given listening stimuli in the listening items. The difficulties using the TE features in this grade band mainly concerned the recording buttons, which was commonly observed in other grades as well.

As described earlier, speaking to the computer and recording responses is, in general, a new type of assessment task for K—12 students in the United States. All students in the study needed to learn how to use the three recording buttons (i.e., record, stop, and playback) in the speaking items. Students’ initial difficulty with the recording buttons seemed mainly due to a lack of practice or explicit directions on the use of the three buttons. The interviewers’ notes were consistent in reporting that all students were able to use recording buttons independently after trying a couple of items. Some students were observed to replay their recorded responses to evaluate their own responses and then re-record their answers if desired.

As far as typing is concerned for extended writing items, many students in Grades 2—3 were noted to have substantial difficulties typing responses to the items. It was clear that the students in the sample were not accustomed to typing. The students often asked the interviewer for help locating letters on the keyboard. Most students were also noted to type using one or two fingers at a slow pace. Some students’ response samples provided evidence that students were not able to fully demonstrate their writing skills when required to type their responses. For example, one advanced-level EL student wanted to type, “I want to be a police officer” for the "Write an Opinion” item type. However, her typed response on the computer was, “i wot to be a pies ofi sr. (sic)” It appeared that the problem was not only her writing skills (e.g., correct spelling being part of the construct) but also her difficulty in finding the letters on the keyboard.

To better disentangle the impact of typing a response versus writing a response, some students were offered the opportunity to write their responses on paper after spending some time typing on the computer. Figure 7.3 includes two examples comparing students’ responses on the computer and paper after a similar length of time. It was apparent that these students’ typing skills might have had an impact on students’ ability to demonstrate their writing skills.

In contrast, the sample of Grades 6—8 students in this study had very little difficulty typing on the computer. Although only 13 students were able to complete the writing items due to the limited testing time, all students except one typed their responses to the “Storyboard” item without any typing issues. One

FIGURE 7.3 Examples of students’ responses in Grades 2—3 on the computer vs. on paper.

student commented that she preferred to write on paper, but other students commented that they preferred to type on the computer due to the neatness of their written responses. In fact, the interviewers generally commented that they found a noticeable difference between the Grades K-3 and the Grades 6—8 samples in terms of the use of TE features. None of the students in Grades 6-8 sample had a problem with the drag-and-drop, zone selection, or drop-down menu features.

In summary, while the study sample students generally exhibited high engagement in the computer-based ELP assessment and TE items, a common problem emerged when students encountered TE features for the first time at the beginning of the assessment. In spite of the video tutorial, students’ attention span and lack of hands-on experience seemed to have limited the impact of the tutorial. Although all students were able to learn how to handle TE features quickly during the testing, the need for explicit directions and opportunities for hands-on tutorial/practice was paramount.

It is important to note that this study was conducted with a small sample using a preliminary test platform when the directions and final design of the test platform were yet to be developed. While the study findings should be carefully interpreted, they provided useful information to enhance the quality of the ELPA21 assessment. In the next section, we describe how the study findings were utilized for further assessment development.