SPECIAL CONSIDERATIONS IN MONITORING AND EVALUATING MEANINGFUL TASK INVOLVEMENT
Special Considerations With the Monitoring Process
One consideration with the monitoring process pertains to analyzing situations in which there are frequent recordings of nonmeaningful task involvement. Earlier it was emphasized that for a task in which a participant is engaged to be meaningful, both the activity and materials associated with the task must meet the guidelines for meaningfulness. Hence, sometimes it can be helpful for the monitor, when marking a task as nonmeaningful, to make notes on the form regarding whether it was the activity and/or the materials that were nonmeaningful and furthermore, what aspects of the activity and/or materials were problematic. Alternatively, the form can be altered to include specific codes and notes about nonmeaningful activities and/or materials. Such information can be useful when proceeding with the actual change process described in Chapter 6, A Staff Training and Supervision Plan to Increase Meaningful Activities (i.e., identifying what specific aspects of respective tasks need to be altered).
Another consideration is more of a procedural technicality. Sometimes, when a participant is observed he/she will be walking or otherwise transitioning from one task to another. In this situation, the observer should continue observing the participant until it becomes clear what new task the participant will be initiating, and then score the meaningfulness of that task. This is another reason that 10 seconds are allowed for the monitor to make a recording regarding what each participant is doing. If, after 10 seconds the new task still has not been initiated, then that recording space on the form should be marked as off-task. It should also be noted that if it becomes apparent that the participant is simply walking around the room with no purpose in terms of initiating another task, then his/her behavior should be scored as off-task.
A third consideration pertains to using the monitoring form to evaluate task involvement of a single participant. The monitoring form is designed to evaluate an overall program’s involvement of its groups of participants in meaningful tasks. Consequently, insufficient data usually result for evaluating the involvement of a single participant because the participant’s task involvement is only observed on one or a few occasions. If a concern exists regarding one particular participant’s involvement in meaningful tasks, use of the form can be modified to conduct an in-depth evaluation with that participant. The modification involves repeating the 10-second observations with that one individual only (i.e., only the individual’s name is listed on the left-hand side of the form in contrast to multiple participants).
Still another consideration when monitoring pertains to obtaining interobserver agreement with use of the monitoring form. As with any systematic behavioral observation, establishing the reliability of the monitoring process should be considered standard practice when observing meaningful versus nonmeaningful task involvement. It is strongly recommended that when initiating the monitoring process, plans be put into place to have two people monitor simultaneously and independently on a regular basis. It is further recommended that these types of reliability checks be conducted for at least 20% of all monitoring sessions over time.
The goal with the interobserver agreement checks is for the monitors to agree on at least 80% of their recordings for each monitoring session. If the monitors agree on less than 80% of their recordings for a given session, then the guidelines for meaningful tasks and the behavior codes should be reviewed to determine the source of the disagreements. Additional monitoring and reviewing should then occur until the monitors consistently agree on at least 80% of the recordings.
Ensuring the reliability of the monitoring process through periodic interobserver agreement checks is also important during the maintenance phase of the overall improvement process (see chapter: Maintaining Meaningful Activity Participation). Specifically, after the initial change procedures have been implemented and meaningful task involvement has improved to desired goals, interobserver agreement checks should continue during at least 20% of the maintenance observations. Such checks are necessary to prevent potential drift among the observers away from the guidelines and behavior code definitions
(Green, Rollyson, Passante, & Reid, 2002). Sometimes observers start making their own judgments over time about the meaningfulness of different types of task involvement independent of the definitions. Conducting periodic reliability checks with two independent observers can help prevent such occurrences and ensure that the monitoring is performed consistently based on the definitions.
Conducting interobserver agreement checks is also valuable because it provides a regular opportunity for two or more monitors to have in-depth conversations about the data and the whole monitoring process. That is, the need for taking data that is in high agreement across monitors serves as a prompt for them to very carefully evaluate and discuss the entire process, which helps to enhance their buy-in and participation.
