Goal Attainment Scaling in action research: enhancing a systems thinking orientation

Eileen Piggot-Irvine, Lesley Perkins, and Phil Cady

Introduction

The relationship between systems thinking and action research (AR) has been well established by researchers grounded in both domains (Barton, Stephens, and Haslett 2009; Burns 2014; Flood 2010; Ison 2013). Further, the advantages for action researchers and systems-oriented researchers in cross-fertilisation of the two traditions have also been identified, albeit not fully explored (Burns 2014; Ison 2013).This chapter further develops the linkages between systems ideas in research and AR by exploring the employment of a Goal Attainment Scaling (GAS) tool with six AR case study projects in the Evaluative Study of Action Research (ESAR).

Having a clear purpose is important for any project, including an AR project. Purpose is usually articulated in some form of goal(s), or attainment statements, early on in a project. As a team of international researchers, we decided to employ the little-known GAS tool to summarise goal achievement as part of our repertoire of methods for evaluating process and impacts of AR projects from around the globe. The GAS was used as a triangulation tool within six case studies in ESAR.

We argue that the GAS power is in its deployment within a complex system whereby outcomes are emergent properties of interaction among system agents (Kurtz and Snowden 2003; Snowden and Boone 2007). In the context discussed in this chapter, the system agents are both the team members of the AR case study projects and the researchers evaluating those projects in the ESAR. We discuss that such interaction offered both the project team members and the researchers a way to engage with and learn more about the complexity of evaluating individual and multiple AR projects in diverse contexts. As Flood (2010) argued, systemic thinking is not so much an approach to undertaking AR, but is a grounding for AR that offers the potential to broaden action and deepen the research endeavour.

We adopt the latter stance offered by Flood (2010) as we outline the background to the ESAR within which the GAS tool was used. As we establish this context, we draw the connections between the literature associated with AR, systems thinking, goal attainment, and assessing goal achievement generally with the GAS tool. We show how the customised use of GAS for the purposes of the ESAR project offered the opportunity for the GAS tool to be completed individually by AR project team members, followed by compilation by the ESAR researchers who returned the compilation to the project team members with encouragement for them to discuss the findings and then share their response back with the ESAR researchers.The exchange between project team members and the ESAR researchers was designed to offer the opportunity to engage more deeply with AR project team members in learning about the complexity of AR projects in varied contexts at the same time as generating data from the GAS compilation that could be triangulated with interview, survey, and documentary analysis data for case study AR project evaluation. We discuss the advantages and disadvantages of the GAS tool and provide ideas for further implementation of this tool within AR systems. The key contribution of this chapter overall is demonstration of how systemic thinking provided a grounding/erAR through the employment of GAS to both broaden action and deepen the research endeavour.

Background to ESAR context

The ESAR is a meta-evaluative study of multiple AR projects. It was conceived because of an identified need to better understand the complex systems that sit behind AR studies (Piggot-Irvine, Rowe, and Ferkins 2015). AR (in all variations) is typically described as a developmental research methodology with rigorous design, data collection, and theoretical integration elements (‘research’), as well as a change (‘action’) agenda (Piggot-Irvine et al. 2015). In deploying the approach, action researchers frequently espouse claims of personal, team, organisational, and community improvement/transformation (Somekh and Zeichner 2009). Indeed, AR is widely promoted as an effective framework of empowerment and emancipation to improve a social situation condition (Reason and Bradbury 2008; Stringer 2007), which is also consistent with Jacksons (2003) categorisation of emancipatory methods inherent in pluralistic thinking within complex systems. However, the validity of such espousals has been substantially unexplored, and where evaluations have occurred, they have been focused more on the process than impact (Piggot-Irvine and Bartlett 2008; Piggot-Irvine et al. 2015).

Indeed, to date, no research has been comprehensively conducted that explores the intricacies of this methodology in order to evaluate if, how, and why AR meets its ideals.The ESAR therefore aims to examine processes, outcomes, and impacts of AR on a global scale. As part of this overarching aim, we are motivated to advance knowledge and understanding of some of the elements which enhance processes, outcomes and the impacts of AR (Piggot- Irvine et al. 2015). Such understanding is consistent with Cabrera and Cabrera’s (2015) assertion that systems thinking involves, in part, the ability to make distinctions between various elements of a system (what is in, what is out) and, in this case, the various components of an AR undertaking. More specifically, our views on systems thinking are intentionally informed by Cabrera and Cabrera s contemporary approach to the field in what they term systems thinking 2.0 (Cabrera and Cabrera 2015). Cabrera and Cabrera argue that systems thinking 2.0 is a way of thinking that balances both ‘the systems and the thinking part of systems thinking ... take[ing] into account not just what is perceived about the real world, but also the perceiver’s predilection to misinterpret’ (24-25).

Critical to the ESAR has been the composition of a seven-strong research team from New Zealand, Australia, and Canada, with combined skills (investigative evaluation, both quantitative and qualitative analysis, AR, project management, systems analysis) to conduct rigorous mixed methods research while also holding and applying deep experience with AR principles and values. In previous chapters (Piggot-Irvine et al. 2015; Zornes, Ferkins, and Piggot- Irvine 2015), we have described our epistemological position as being rooted in an emergent, participatory worldview (Reason and Bradbury 2008), while also noting that such a worldview draws from the interpretivist—constructivist research paradigm (Denzin and Lincoln 2013). Our positioning reflects an important systems thinking consideration in that an epistemological perspective includes both a point and a view (Cabrera and Cabrera 2015;Williams 2008).

We have deployed a mixed methods approach as part of our evaluative set of tools and deemed this as an appropriate means to gather data on process, outcomes, and impact (Piggot-Irvine et al. 2015).The tools have included documentary analysis, online survey, interviews, focus groups, and GAS - the focus of the present chapter. In the first instance, the tools have been adopted within ESAR as part of a set of six case studies which were purposefully selected (Adams, Khan, Raeside, and White 2007) because we drew upon projects able to be accessed with relative ease and in reasonable proximity to the research team.

In case study methodology, a single unit analysis is based upon depth that is both holistic and exhaustive (Bassey 2007), but which also retains the meaningful characteristics of realistic events. In the ESAR, a previously designed indicator matrix (Piggot-Irvine et al. 2015) guided our use of the case study methodology. The methodology allowed us to explore participant experiences and understandings as an experiential whole rather than solely the component parts (Hannabuss 2000) - a key systems thinking tenet (Cabrera and Cabrera 2015; Williams 2008). GAS was used, as noted earlier, as a complementary tool alongside in-depth interviews, focus groups, a survey, and documentary analysis. The following section of the chapter explores theory which informed our use of GAS in the case studies.

Reviewing and connecting related literature

In this section, we traverse several bodies of literature to rationalise our aim of employing GAS as a tool for dealing with the complexity of evaluating multiple AR projects in multiple contexts. First, we take on the daunting task of summarising the systems thinking field, in a succinct way, to demonstrate some of the pertinent connections to GAS and ESAR. Next, we discuss literature related to goals and AR, and assessing goal achievement generally with the GAS tool.

Systems thinking as a grounding for AR

The field of systems thinking has evolved over the years to form a large body of literature (Francois 2004; Midgley 2003) replete with a myriad of models, methodologies, and frequently conflated and confused concepts and terminology (Richmond 1994).The term‘systems thinking’, first coined by Richmond (1994), is defined as the art and science of making reliable inferences about behaviour by developing an increasingly deep understanding of underlying structure. Authors such as Jackson (2003) offer coherence to the burgeoning field of systems thinking and its associated tools and models by creating taxonomies or frameworks such as the System of System Methodologies (SoSM). Bringing coherence, in and of itself, is a considerable task due to a lack of agreement in the field about what the term systems thinking actually refers to (Arnold and Wade 2015; Cabrera and Cabrera 2015). Mostly, systems thinking as an intellectual and cognitive endeavour is seen to hold great promise in addressing messy, and at times, intractable problems and issues (Trochim, Cabrera, Milstein, Gallagher, and Leischow 2006).

Over time, systems thinking has been aggregated into concepts or models with an aim to providing clarity amid the myriad of terms endemic to the discipline (Senge 2006). For example, AckofFs (1971) work on a system of systems concepts has afforded researchers and practitioners practical utility generally, as have the works of Flood and Carson (1993); Lawson (2010); and Checkland and Scholes (1990), to name but a few. Amid the confusion and often conflated terms and terminology, some promising, uncomplicated, and helpful concepts appear to be emerging. For example, Williams (2008) cited interrelationships, boundaries, and perspectives as central to systems thinking. Similarly, in their DSRP model, Cabrera, Cabrera, and Powers (2015) and Cabrera and Cabrera (2015) noted distinctions (what is in and what is out), systems (of parts and wholes), relationships (of actions and reactions) and perspective (consisting of both a point and a view) as a set of meta-cognitive processes which give rise to systems thinking.

Arguably, applying the concepts inherent to systems thinking to the process of AR is critical to effective exploration of whether AR lives up to the promises it espouses. We would suggest that even the GAS tool itself is undoubtedly a product of application of Cabrera and Cabreras (2015) meta-cognitive, modular, iterative, and fractal processes. Finally, the mere process of engaging in the ESAR project, and specifically the evaluation of goal achievement as part of the study, would not be possible without considering the central tenets of systems thinking as identified by authors such as Williams (2008) and Cabrera and Cabrera (2015). It is this area of goal achievement that we focus on next.

Goals and AR

There is considerable support for the importance of goal setting for enhanced achievement, alignment of strategies, and focus (Asplund and Blacksmith 2013; Ferguson and Porter 2010; Gordon 2006; Kouzes and Posner 2007; Latham and Locke 2006; Leithwood, Aitken, and Jantzi 2006; Locke and Latham 2013; Piggot-Irvine 2015; Snyder and Lopez 2005; Uhl-Bien and Marion 2009).

Piggot-Irvine (2003) has also long explored a preference for the establishment of goals for AR-aligned projects to be articulated in a format that indicates ‘deep’ as opposed to ‘surface’ plans.‘Deep’ refers to plans outlining inquiry learning type projects (Piggot-Irvine and Bartlett 2008) where clear actions and expectations are indicated, where goal pursuers seek to be well-informed about their goal topic and make evidence-informed decisions, where collaboration with key personnel who are impacted by goals is prioritised to enhance ownership of outcomes, and where continual reflection on process and outcomes is valued. Deep planning is particularly important in projects where cause and effect in the system is not apparent and where analysis is required to determine a course of action (Kurtz and Snowden 2003; Snowden and Boone 2007). Such depth is important overall because, as Latham and Locke (2006) have suggested, ‘setting high goals means setting the bar higher . . . the higher the goal, the higher the performance’ (333). Little attention, however, has been placed on how to assess goal achievement regardless of whether the goal is written in depth or not.

Assessing goal achievement using GAS

Self-review against measurable outcomes is a common component of evaluation of goals, but in our experience, this is often conducted in quite informal ways, especially in AR. GAS, however, provides an approach for more objectively quantifying the achievement of goals (Molyneux et al. 2011). GAS had its origins in mental health with authors Kiresuk and Sherman (1968). Since then, GAS has mostly been associated with the health sector (Lannin 2003), and less so in coaching (Spencer 2007;Whiston 1996) and education (Latham and Locke 2006; Roach and Elliott 2005).

Generally, GAS has mostly been utilised as a way of individually self-reviewing/ evaluating goal achievement with little mention of aggregating or utilising selfreview for dialogue about process or outcomes in collaborative endeavours. Apart from a recommendation that GAS might be implemented and evaluated to identify if it is a useful method for consistently evaluating progress in the study conducted by Molyneux et al. (2011), and a more general advisory for use by Whiston (1996), we found no studies showing implementation of GAS as an evaluative activity in AR projects either as an individual or collective pursuit. We also found no evidence of employment at a meta-evaluative level across multiple AR projects for aggregation and comparison of results.

In the ESAR our intent in employing GAS was to use such aggregation and comparison to look across projects - an approach which aligns with the views of Cabrera and Cabrera (2015) and Williams (2008) on the concept of perspectives. Further, such cross-project comparison is entirely consistent with the inherently fractal and modular nature of Cabrera and Cabrera’s (2015) DSRP systems thinking rules mentioned earlier, where the D, S, R, and P are seen to work together.

GAS in the ESAR

GAS initially involves establishing criteria against which progress can be rated as ‘expected’ (neutral, 0, scoring),‘below expected’ (—1 or —2 scoring), or‘above expected’ (+1 or +2 scoring), as noted in the example GAS table constructed for the ESAR (see Table 6.1). As Table 6.1 shows, we used the outcomes for each phase of AR to form horizontal categories. Such categories (or distinctions as identified in Cabrera and Cabrera’s 2015, DSRP framework) included preparatory, reconnaissance (current situation analysis), implementation, evaluation, and reporting phases. Ratings formed the vertical categories using the —2 to +2 categorisation described earlier. In the ESAR, the measurable outcomes transferred directly from the AR phases, so little work was involved in relocating them to the GAS table. The ratings statements themselves are reasonably standard and logical across GAS tables and again, therefore, minimal adjustment was needed in constructing those ratings. For example, the neutral, 0, expected level standard wording is close to many ‘outcome’ statements expressed in AR project phases.The —1 level is at the ‘limited’ achievement level, or some similar synonym.The —2 level indicates no achievement, and so forth.

First impressions of our research team suggested that the table was very complicated, and we originally felt that both construction of the categorisation and participant completion would require a lot of work. Such concerns were quickly allayed, however, when categorisation proved to be simple, and when we piloted and then employed GAS with participants in AR project teams, we quickly saw that the table was easy to complete. This realisation reinforces the value of perspective in considering solutions to problems as they are socially constructed through dialogue (Kurtz and Snowden 2003).

Participants

As noted earlier, six case studies (Cl— C6) were purposefully selected (Adams et al. 2007) for the ESAR. Although we drew upon AR projects readily accessible as cases (described in Table 6.2), allocation of a researcher to each case was based on the researcher having no prior relationship with the case participants. The intent of the latter approach was to offer some impartiality in dealing with the evaluative nature of GAS data collection and analysis.

Invitations to complete the GAS were issued first to the six case project leads, followed by formal consent documents and a request that the lead provided

Criteria

Outcome attainment levels Rating

Preparatory Outcome 1 Focus for project

Preparatory Outcome 2 Collaborative approach

Reconnaissance Outcome 1 Project informed

Reconnaissance Outcome 2 Reconnaissance data gathering

Implementation Outcome 1 Plan for action

Implementation Outcome 2 Reflections on progress by project team and/

or stakeholders

Evaluation Outcome 1 Data

gathering for determining improvements

Evaluation Outcome 2 Planning for future improvements

Reporting Reporting out to others

Most unfavourable outcomes (-2)

No focus

Project carried out solely by research team

No literature, previous materials, used to inform project

No data collected, analysed, or summarised

No plan developed

No reflections on progress recorded

No data shown or summarised

No further planning conducted

No reporting out

Less than expected success with

outcomes

(-1)

Vague focus only determined

Research

team

involves

other

stakeholders

with

limited

collaborative

approach

brief, simplistic, or unsynthesised review of material to inform project

Limited data collected

A simplistic and/or brief plan developed

Brief and superficial reflections on progress recorded

Limited data collected

Limited

further

planning

for

improvement

Brief &/or simplistic reporting out

(Continued)

Criteria

Outcome attainment levels Rating

Preparatory Outcome 1 Focus for project

Preparatory Outcome 2 Collaborative approach

Reconnaissance Outcome 1 Project informed

Reconnaissance Outcome 2 Reconnaissance data gathering

Implementation Outcome I Plan for action

Implementation Outcome 2 Reflections on progress by project team and/

or stakeholders

Evaluation Outcome 1 Data

gathering for determining improvements

Evaluation Outcome 2 Planning for future improvements

Reporting Reporting out to others

Expected

level

outcomes

(0)

Focus

determined

Collaborative approach established between researchers and other stakeholders

A few literature, etc., sources used to inform project

Data gathered on the existing situation associated with project goals

Plan for action developed based on reconnaissance findings

Reflections on progress recorded throughout change implementation

Data

gathering conducted to show evidence of

improvement

Further planning for future improvements conducted

Reporting out to stakeholders

More than expected success with

outcomes

(+1)

Strong focus set for project, goals articulated

Collaborative approach established through active participation of key stakeholders in key activities

Range of references, materials drawn upon to inform the project

More than two data sources provided as evidence collected, analysed, and

summarised

Enhanced, detailed action plan developed and documented briefly

Reflections on progress recorded in detail

More than two data

sources

provided

as

evidence collected, analysed, and summarised

Plan for future improvement documented and shared

Enhanced reporting out to boundary partners

Best

anticipated

success

with

outcomes

(+2)

Strong

focus,

well-

defined

goals

with

detailed

objectives

articulated

Collaborative

approach

established

and

actively maintained through all activities of the project

Comprehensive and extensive range of references, materials drawn upon to inform the project

Extensive data sources provided as evidence collected, analysed, and

summarised

Extensive, clearly detailed action plan developed and documented; adjusted as project evolves

Extensive, in-depth, reflections on progress recorded

Extensive

data

sources

provided

as

evidence

collected,

analysed,

and

summarised

Extensive plan for future improvement documented and

discussed with others

Extensive reporting out to boundary partners

Table 6.2 Cases

Case number

Sector and focus

AR team respondents

Cl

Sport: developing governance capability in national sport organisations

2 leaders (L), 2 AR teams

C2

Education: reviewing and improving goalsetting process within a not-for-profit music therapy organisation

1 L, 2 AR teams

C3

Health: leadership capacity development in a health organisation

1 L, 1 AR team

C4

Community: community programme with Indigenous peoples to assist in addressing social issues

C5

Community and mental health: to understand and improve housing support needs and practices for homeless Indigenous peoples

1 L, 1 AR team

C(>

Health: developing best practice guidelines for engagement and assessment of Indigenous persons with acquired brain injury and their communities

1 L, 1 AR team

us with contact details for other project team members and key stakeholders. Our intent was to use the tool with a wide range of individuals either involved in the case study AR project or impacted by it in order to obtain an array of unique perspectives. With all participants, the completed GAS tool was returned directly to the allocated researcher.

Where limited or no response was gained from case project leads, the allocated ESAR researcher made multiple follow-up requests where required. Unfortunately, in C4, the AR project lead whom we had previously successfully interviewed could not be contacted despite multiple attempts. Five cases are therefore reported upon in our findings. The five AR cases varied considerably in relation to sector, focus, and AR team respondents, as summarised in Table 6.2.

Findings and discussion

Meta-evaluation results in the ESAR

The results of GAS in the ESAR were analysed at a meta-evaluation level in order to aggregate and compare the multiple individual GAS results across the five projects. As noted earlier, such aggregation and comparison is entirely consistent with the nature of Cabrera and Cabreras (2015) systems rule in their DSRP model. The results of this meta-evaluative employment are described in the following section. The summarised findings from the 13 respondents are presented in Table 6.3 where ‘L’, subsequent to a case number, signifies the project lead response.

Outcome attainment levels

Preparatory Outcome 1 Focus for project

Preparatory Outcome 2 Collaborative approach

Reconnaissance Outcome 1 Project informed

Reconnaissance Outcome 2 Reconnaissance data gathering

Implementation Outcome 1 Plan for action

Implementation Outcome 2 Reflections on progress by project team and/ or stakeholders

Evaluation Outcome 1 Data gathering for determining improvement

Evaluation Outcome 2 Planning for future improvements

Reporting Reporting out to others

Most unfavourable outcomes (-2)

3 (L)

Less than expected success with outcomes (-1)

3 (L)

3 (L)

Expected level outcomes (0)

1,5

2,5

1,6

1

3 (L), 5

1 (L), 3 (L), 5,6

1 (L), 1

1 (L), 1

1 (L), 1

More than expected success with outcomes (+1)

  • 1 (L). 1 (L).
  • 1,2 (L),3 (L), 3, 5 (L)
  • 1 (L), 1 (L), 1,2 (L),
  • 3 (L), 5 (L), 6

1 (L), 1 (L), 1,3 (L), 5

  • 1,1 (L),2, 3, 3 (L),
  • 5 (L), 5,6
  • 1 (L), 1 (L).
  • 1,5 (L)

1 (L), 1,2 (L), 5 (L)

1 (L), 1,2 (L), 2, 3,5

2,3,5,6 (L), 6

1 (L),2 (L), 2,6

Best

anticipated success with outcomes (+2)

2,2, 6 (L), 6

1,2, 2,3,6 (L)

2 (L), 2, 2, 3,5 (L),6 (L)

1 (L), 2 (L), 2, 6 (L)

1,2 (L), 2,2, 3,6 (L), 6

2,2,3, 6 (L)

2,5 (L), 6 (L),6

1,2 (L), 2, 5 (L)

  • 1,2,3,
  • 5 (L), 5, 6 (L)

100 Eileen Piggot-Irvine et al.

For Cl and C2, both the project leads (L) and the AR team members scored all phases at the expected outcomes (0) or higher (+1, +2) level. Little difference existed between the responses from leads and team members. C3 showed greater discrepancy between the lead and team member responses. In the preparatory and reconnaissance phase outcomes, the scoring was close. However, in implementation phase (Outcomes 1—2), the project lead scored lower (0) than the team member (+2), and even less in the evaluation phase outcomes. The most extreme difference was in the reporting outcome with a —2 score from the lead and +2 from the team member.

C5 and C6 respondents all scored positively between 0 and +2. Of interest with these two cases was the slightly higher scoring by the lead. In C5, the team member scored lower in seven out of the nine outcomes; in C6, five out of the nine were scored lower.

Discussion

In summary, the great majority of case participants scored positive outcomes (+1, +2) for their AR project phases. Further, where there was an instance of negative response for the evaluation and reporting phase outcomes from the C3 lead, there was also contradiction with a positive response by the team member in this specific case. There is no evidence that case project leads consistently scored higher or lower than other team members.

As a way of discussing the findings and employment of the GAS tool, we offer a perspective on the advantages and disadvantages of GAS linked to the ESAR by drawing upon the work of Roach and Elliott (2005).We have adapted and summarised those advantages and disadvantages in Table 6.4 followed by our perspective on the relevance to the ESAR.

Table 6.4 Advantages and disadvantages of GAS ratings

Advantages of CAS ratings

Disadvantages oj GAS ratings

  • • Time efficient
  • • Personalised/individual review that can then create comparison, dialogue with others
  • • Requires minimal skills to collect data
  • • Non-intrusive review method
  • • Can be used as a self and peer review
  • • Can be used by multiple informants across settings (e.g., team, organisation, community)
  • • Can be used repeatedly to monitor perceptions of intervention progress
  • • Can be used to document perceptions of intervention outcomes
  • • Inexpensive
  • • Requires minimal skills to interpret data
  • • Limited published, empirical research on the use of the method beyond health
  • • Subjective summary of observations collected over time
  • • Not norm-referenced
  • • Guidelines for interpretation are often determined by parties involved with the intervention, thus subject to bias
  • • Has global (i.e. less discrete) accounting of progress and effectiveness

Source: Adapted from Roach and Elliott (2005,15)

Our use of GAS in the ESAR showed disadvantages which align well with those reported by Roach and Elliot (2005), but we believe we have attempted to overcome several. We consider that our use of the tool beyond the health sector and particularly within the AR context is important. In this regard, the ESAR research team discussion of context was an important lens for framing system dialogue (Cady 2016) related to the application of GAS to ESAR.

We also believe that our employment of the tool as a summarising approach to enable triangulation against multiple other data, provided us with a way of balancing varying points of view within our analysis. We did not attempt to norm-reference our results, nor was it feasible given the small number of respondents. We have, however, reduced bias in both creating guidelines for interpretation of levels (as shown in Table 6.1) and ensuring the allocated researchers from our ESAR team had no involvement in the AR case study.

In terms of advantages, the employment of the GAS tool in the ESAR to summarise respondent perceptions was certainly efficient. It took less than ten minutes for respondents to highlight the most relevant descriptor for each of the column criteria and, as noted later, it was our intent that the collation of the individual responses/review could allow respondents to create comparisons, dialogue and peer review with this non-intrusive, inexpensive method. Minimum training time was also required for all ESAR team members to issue the tool to case study respondents and to analyse results.

Our overall intent in using the tool was to document perceptions of AR outcomes, but we also extended this advantage by adding the dimension of perception of process ofAR. In this sense, we applied Cabrera and Cabreras (2015) and Williams’ (2008) notions to our systems thinking in that every perspective is comprised of both a point and a view — a notion which is critical to avoid possibly latent assumptions that project participants experienced their projects similarly. For example, one project participant in C6 identified that they did not have the same opinion regarding project goal attainment as the lead of the project since they were brought on to the AR team at a later stage in the AR cycle.

We consider one of the greatest advantages of GAS employment within AR to be associated with the personalised/individual review which can then create comparison, and dialogue, with others. Because AR case project teams had disbanded before we evaluated them in the ESAR, it has not been possible for us to realise this advantage; however, we discuss its potential in the following section.

GAS for dialogue and further improvement

Though the individually collated results are affirming, we believe that such meta-evaluative aggregation only partially realises the potential of the GAS tool in a collaborative context such as AR.We see that this realisation could occur in two key ways. First, the overview provided by the meta-level analysis was principally seen as a summary of how well each phase was enacted as perceived by the participants in the case study projects, and the findings will be triangulated alongside further and deeper data collected in surveys, interviews, and documentary analysis. The triangulation will be included in a further comparative case study paper.

Second, we intend a more powerful extension of the meta-evaluative results by encouraging project leads to share their team GAS results to stimulate dialogue on outcomes. Employment beyond just a quantifying summary of goal achievement is in keeping with the thinking of Bovend’Eerdt, Botell, and Wade (2009) and Spencer (2007), who are clear that GAS should be used to encourage and stimulate ongoing consultation between all people connected with a goal. Such exploration of the nature of relationships and perspectives also fits with Williams’ (2008) and Cabrera and Cabreras (2015) systems concepts within a project, where a rich opportunity for deep dialogue and reflection about goal achievement is created. We use the word dialogue here in the sense that non-defensive, open, and trust-generating honesty occurs within authentic collaboration (Piggot-Irvine 2015; Senge 2006). Dialogue, in turn, should lead to extended learning and creation of ideas for further development as an emergent property of socially constructed interactions (Kurtz and Snowden, 2003; Plowman et al. 2007; Uhl-Bien, Marion, and McKelvey 2007). Hopefully, such learning and ideas would lead to further implementation of improvements linked to an AR project.

Conclusion

In our experience, GAS has utility at both individual and collective evaluation levels. We have found it to be a meaningful way to evaluate process, improvement, and gains as part of the ESAR. Employment of GAS has therefore contributed data to achieve our following aim in the ESAR:

in this Evaluative Study of Action Research (ESAR) we not only wish to establish the ways in which espoused intents articulated in projects are realised and why certain approaches are adopted and seen to be effective, we also want to add to increased understanding of outcomes and impact ofAR.

(Piggot-Irvine et al. 2015, 545)

We have also contributed to literature associated with systems thinking, goal attainment, AR, and assessing goal achievement through the GAS tool. In this, we have demonstrated the way that we customised the use of GAS for the purposes of our meta-evaluation in the ESAR project. Further, our discussion of the findings of our meta-evaluation against the advantages and disadvantages of GAS, as well as provision of ideas for further implementation of this tool within AR systems, offers action researchers more detail and insight needed to implement the tool within their own AR projects. We have particularly noted our intent for GAS to provide a unique opportunity for enhancement of AR project lead and team member communication and dialogue; to enhance a systems thinking orientation. In such an orientation, as defined by Richmond (1994), the art and science of making reliable inferences about behaviour are encouraged to develop a deeper understanding of underlying structure.

Our research is not without its limitations, and we offer these as a starting point for considering future research opportunities. Our conclusions, established from GAS data, drew from five case studies situated within three Commonwealth countries (New Zealand, Australia, and Canada). While there was a degree of diversity offered between the case studies, broader national and cultural reach may throw up further insights. It may also be interesting to deploy GAS across AR projects in contexts beyond community organisations (e.g., commercial entities, individual-level projects etc). Further, we are mindful that we have used GAS as part of the ESAR project in tandem with interviews, survey, and documentary analysis. We would not promote GAS as a standalone evaluative tool across AR projects.

While we have explored its potential, we have not collected data on the use of GAS as a dialogue tool within AR projects. Future research opportunities could also encompass such further study to determine GAS potential as a tool for dialogue. Our insights about GAS situated within systems thinking indicate that it could be a useful tool to utilise alongside other data collection methods within each of the AR phases. Overall, we offer GAS as a tool with the potential to broaden action and deepen the research endeavour by enhancing the systemic thinking that is not so much an approach to undertaking AR, but is a grounding for AR (Flood 2010).

References

Ackoff, R. 1971. “Towards a System of Systems Concepts.” Journal of Management Science 17: 661-671. https://doi.org/10.1287/mnsc. 17.11.661 Adams,J., H.T.A. Khan, R. Raeside, and D. I.White. 2007. Research Methods for Graduate Business and Social Science Students. New Delhi: Sage Publications.

Arnold, R. D.,and J. P. Wade. 2015. “A Definition of Systems Thinking: A Systems Approach.” Procedia Computer Science 44:669—678.

Asplund,J.,and N. Blacksmith. 2013. “Strength-based Goal Setting.”Journal of Gallup Business. Retrieved October 1, 2013 from http://businessjournal.gallup.com/content/152981/ strengths-based-goal-setting.aspx

Barton, J., J. Stephens, and T. Haslett. 2009. “Action Research: Its Foundations in Open Systems Thinking and Relationship to the Scientific Method.” Systemic Practice and Action Research 22: 475-488. doi:10.1007/sl 1213-009-9148-6 Bassey, M. 2007. “Case Study Research.” In Research Methods in Educational Leadership and Management, edited by M. Coleman and A. R. J. Briggs, 108—121. London: Sage Publications.

Bovend’Eerdt.T., R. Botell, and D. Wade. 2009.“Writing SMART Rehabilitation Goals and Achieving Goal Attainment Scaling: A Practical Guide.” Clinical Rehabilitation 23: 352- 361. https://doi.org/10.1177/0269215508101741 Burns, D. 2014. “Systemic Action Research: Changing System Dynamics to Support Sustainable Change.” Action Research 12 (1): 3-18. https://doi.org/10.1177/1476 750313513910

Cabrera, D., and L. Cabrera. 2015. Systems Thinking Made Simple: New Hope for Solving Wicked Problems. New York: Odyssean Press.

Cabrera, D., L. Cabrera, and E. Powers. 2015. “A Unifying Theory of Systems Thinking with Psychosocial Applications.” Systems Research and Behavioral Science 32: 534—545. doi: 10.1002/sres.2351

Cady, P. 2016. “A System of System Lenses for Leadership Decision-making.” Healthcare Management Fomin 29 (1): 8-11. doi:10.1177/0840470415614412

Checkland, R, and J. Scholes. 1990. Soft Systems Methodology in Action. New York: John Wiley & Sons Inc.

Denzin, N., and Y. Lincoln. 2013. Strategies of Qualitative Inquiry. 4th ed. London: Sage Publications.

Ferguson, M., and S. Porter. 2010. “What Is Implicit About Goal Pursuit?” In Handbook of Implicit Social Cognition: Measurement, Theory, and Applications, edited by B. Gawronski and K. Payne, 311-334. New York: Guilford Press.

Flood, R. 2010.“The Relationship of‘Systems Thinking’ to Action Research.” Systemic Practice and Action Research 23:269-284. doi:10.1007/sl 1213-010-9169-1

Flood, R., and E. Carson.1993. Dealing With Complexity: An Introduction to the Theory and Application of Systems Science. New York: Plenum Press.

Francois, C., ed. 2004. International Encyclopedia of Systems and Cybernetics. Munchen: K. G. Saur.

Gordon, G. 2006. Building Engaged Schools: Getting the Most Out of American Classrooms. New York: Gallup.

Hannabuss, S. 2000. “Telling Tales at Work: Narrative Insight into Managers’Actions.” Library Review 49 (5): 218-229. https://doi.org/10.1108/00242530010331745

Ison, R. 2013. “Systems Thinking and Practice for Action Research.” In The Sage Handbook of Action Research: Participative Inquiry and Practice (2nd ed.), edited by P. Reason and H. Bradbury, 139-158.Thousand Oaks, CA: Sage Publications.

Jackson, M. 2003. Systems Thinking: Creative Holism for Managers. San Francisco, CA: John Wiley and Sons.

Kircsuk.T., and R. Sherman. 1968. “Goal Attainment Scaling: A General Method for Evaluating Comprehensive Community Mental Health Programmes.” Community Mental Health Journal 4: 443-453. https://doi.org/10.1007/BF01530764

Kouzes, J., and B. Posner. 2007. The Leadership Challenge. San Francisco, CA:Jossey-Bass.

Kurtz, C., and D. Snowden. 2003. “The New Dynamics of Strategy: Sense-making in a Complex and Complicated World.” IBM Systems Journal 42 (3): 462—483. doi:10.1147/ sj.423.0462

Lannin, N. 2003. “Goal Attainment Scaling Allows Program Evaluation of a Home-based Occupational Therapy Program.” Occupational Therapy in Health Care 17: 43-54. https:// doi.org/10.1080/J003v 17n01_04

Latham, G., and E. Locke. 2006. “Enhancing the Benefits and Overcoming the Pitfalls of Goal Setting.” Organisational Dynamics 35 (4): 332-340. doi:10.1016/j.orgdyn.2006.08.008

Lawson, H. 2010. A Journey Through the Systems Landscape. London: College Publications.

Leithwood, K., R. Aitken, and D. Jantzi. 2006. Making Schools Smarter: Leading with Evidence. Thousand Oaks, CA: Corwin Press.

Locke, E., and G. Latham. 2013. New Developments in Coal Setting and Ttsk Performance. New York: Routledge.

Midgley, G. 2003. Systems Thinking.Thousand Oaks, CA: Sage Publications.

Molyneux, С., M.Willis, A.Talmage, R. Scoones, R.Travaglia, N. Gang, and E. Piggot-Irvine. 2011. Collaborative Coal Setting and Reviewing in Music Therapy for Children with Special

Needs: An Action Research Project to Improve Practice and Measure Efficacy. Report to fulfil requirements for New Zealand Lotteries Commission Grant.

Piggot-Irvine, E. 2003.“Key Features of Effective Appraisal.” The International Journal of Educational Management 17 (4): 170-178. https://doi.Org/10.l 108/09513540310474392

Piggot-Irvine, E. 2015. Goal Pursuit in Education Using Focused Action Research. New York: Palgrave McMillan.

Piggot-Irvine, E., and B. Bartlett. 2008. Evaluating Action Research. Wellington: New Zealand Council for Educational Research Press.

Piggot-Irvine, E.,W. Rowe, and L. Ferkins. 2015. “Conceptualising Indicator Domains for Evaluating Action Research.” Educational Action Research 23 (4): 545—566. doi:10.1080/09 650792.2015.1042984

Plowman, D. A., S. Solansk,T. Beck, L. Baker, M. Kulkarni, and D. Travis. 2007. “The Role of Leadership in Emergent, Self-organisation.” The Leadership Quarterly 18 (4): 341-356. doi: 10.1016/j.leaqua.2007.04.004

Reason, P, and H. Bradbury. 2008. The Handbook of Action Research: Participative Inquiry and Practice. 2nd ed. Thousand Oaks, CA: Sage Publications.

Richmond, B. 1994. “Systems Thinking/System Dynamics: Let’s Just Get on With It.” Systems Dynamics Review 10: 135-157. doi: 10.1002/sdr.4260100204

Roach, A., and S. Elliott. 2005.“Goal Attainment Scaling: An Efficient and Effective Approach to Monitoring Student Progress.” Teaching Exceptional Children 37 (4): 8-17. https://doi. org/10.1177/004005990503700401

Senge, P. 2006. The Fifth Discipline:TheArt and Practice of the Learning Organization. New York: Doubleday.

Snowden, D., and M. Boone. 2007. “A Leaders Framework for Decision Making.” Harvard Business Review 85 (11): 68-76. https://doi.Org/10.l 177/004005990503700401

Snyder, C., and S. Lopez. 2005. Handbook of Positive Psychology. New York: Oxford University Press.

Somekh, B., and K. Zeichner. 2009.“Action Research for Educational Reform: Remodeling Action Research Theories and Practices in Local Contexts.” Educational Action Research 17 (1): 5-21. https://doi.org/10.1080/09650790802667402

Spencer, G. 2007.“GAS Powered Coaching: Goal Attainment Scaling and Its Use in Coaching Research andPractice.’TnternationalCoachingPsychologyReview2(2):55-167.Retrieved from www.researchgate.net/profile/Jonathan_Passmore/publication/47528957_The_state_of_ executive_coaching_research_What_does_the_current_literature_tell_us_and_what’s_ next_for_coaching_research/links/5730dlel08ae08415e6a7fa5/The-state-of-executive- coaching-research-What-does-the-current-literature-tell-us-and-whats-next-for- coaching-research.pdf#page=43

Stringer, E.T. 2007. Action Research.Thousand Oaks, CA: Sage Publications.

Trochim, W., D. Cabrera, B. Milstein, R. S. Gallagher, and S. |. Leischow. 2006. “Practical Challenges of Systems Thinking and Modeling in Public Health.” American Journal of Public Health 96 (3): 538-546. doi:10.2105/AJPH.2005.066001

Uhl-Bien, M., and R. Marion. 2009. “Complexity Leadership in Bureaucratic Forms of Organising: A Meso Mode.” The Leadership Quarterly 20: 631-650. https://doi. org/10.1016/j.leaqua.2007.04.002

Uhl-Bien, M., R. Marion, and B. McKelvey. 2007. “Complexity Leadership Theory: Shifting Leadership from the Industrial Age to the Knowledge Era.” The Leadership Quarterly 18 (4): 298-318. doi: 10.1016/j.leaqua.2007.04.002

Whiston, S. 1996. “Accountability Through Action Research: Research Methods for Practitioners.” Journal of Counselling & Development 74 (6): 616—623. https://doi. org/10.1002/j. 1556-6676.1996.tb02301.x

Williams, B. 2008. “Bucking the System: Systems Concepts and Development.” The Broker.

Retrieved from www.thebrokeronline.eu/Articles/Bucking-the-system Zornes, D., L. Ferkins, and E. Piggot-Irvine. 2015. “Action Research Networks: Role and Purpose in the Evaluation of Research Outcomes and Impacts.” Educational Action Research 24 (1): 97-114. doi: 10.1080/09650792.2015.1045538

 
Source
< Prev   CONTENTS   Source   Next >