Indicators of research credibility

The question of how to meaningfully evaluate the credibility of qualitative research is a complex and contested area (Cutcliffe & McKenna, 2004; Flick, 2009). A commonly adopted approach has been to utilise indicators of research credibility drawn from the quantitative field and reconfigure them to suit the unique characteristics of qualitative exploration (Corbin & Strauss, 2015; Lincoln & Guba, 1985).The most common indicators used to assess the credibility of research outcomes in quantitative research are reliability and validity. Reliability is about the consistency of the research instruments used and is related to replicability (Rentier &Van Ryzin, 2011).Therefore in quantitative research, studies are seen to have reliability when the measures, methods and procedures are repeated and generate the same results (Leung, 2015; Walliman, 2006).This measurement of reliability is problematic in qualitative research where emphasis is placed on capturing the nuances of individual experiences through less standardised and more flexible approaches to data collection. Qualitative researchers may digress from their interview schedules and probe study participants about something they have said or observed (see Rubin & Rubin, 2005 for further information on the technique of probing). The implication of differences between quantitative and qualitative exploration is that an indicator of reliability, based on replication of results, does not facilitate meaningful assessment of research quality within the qualitative sphere. While proposals differ, there appears to be some consensus that dependability is a more accurate indicator to measure the credibility of research outcomes in qualitative research (Lincoln & Guba, 1985; O’Leary, 2017). In this context, dependability refers to consistency of approach both in the execution and documentation of the research process and related outcomes (Leung, 2015; O’Leary, 2017).

Similar concerns exist about the appropriateness of validity as an indicator to evaluate the credibility of research outcomes in qualitative studies. In quantitative research validity broadly refers to whether the construct of interest is actually being measured (Rentier &Van Ryzin, 2011). Therefore validity involves establishing the correctness of a “singular truth” and the soundness of the procedures to create this “truth” (O’Leary, 2017, p. 60). In contrast, the pursuit of truth in qualitative research is focussed on conveying multiple truths and communicating the detail and nuances of individual experience or phenomena. For this reason, authenticity is proposed as a more accurate indicator to establish if research findings have been accurately captured. O’Leary (2017, p. 67) defines authenticity as “concerned with truth value while recognising that multiple truths may exist [and] . . . with describing the deep structure of experience/ phenomena in a manner that is ‘true’ to the experience”.

Practical strategies for developing research credibility

Drawing on examples from the research study outlined, much of the remainder of this chapter focusses on the strategies adopted during the data collection and analysis phases to support the development of credible and robust research findings.

Embedding credibility in the data collection process

Preparation of the interview and focus group schedules involved extensive review of the literature on compliance, consultation with practitioners and piloting of the schedules. It was identified very early in the data collection process that some questions in the young people’s interview schedule did not fully elicit the depth of information required. For example, initial questions asked were:‘Do you ever miss supervision appointments?’‘Ifyes, what would be the most common reasons for missing them?’ Following reflection, it became apparent that the difficulty arose due to crafting questions based on literature from other jurisdictions where expectations surrounding compliance with supervision requirements differed and young people were afforded less flexibility than in the Irish system (Bateman,2011). Given the differences in practice, it was recognised that there was much to be learned from exploring how reasons for non-attendance were presented by young people to their supervisors, what their understanding was of acceptable and unacceptable reasons and how they managed the process. By expanding the nature of questioning, a more nuanced account emerged of young people’s decision-making about compliance. This is demonstrated in the following when Claire describes the notion of acceptability' of excuses for non-attendance at appointments:

Like you couldn’t text her [probation officer] an’ say ‘ah listen I’m dyin’ [ill from excessive consumption alcohol], I’m not goin’ in’ [for a supervision appointment]. If I was sick or if there was someone else ill or if I just literally wasn’t able to make it or if I had to mind me brother or somethin’, they’d be the only reasons I do make anyway. [Claire]

Failure to critically reflect on the nature of the questions asked after the research commenced would have resulted in much less detailed and consequently less accurate insights into how young people navigated their way through the requirements of community supervision. This highlights the importance of researchers continually reflecting critically on their data and their research practice to ensure that participants’ accounts are fully captured and contribute to the development of authentic findings.

In addition to reflecting on what was said during an interview, detailed listening of interview and focus group recordings shortly after they had taken place was beneficial. This process allowed the researcher to reflect on whether interactions with participants were conducted in a way that encouraged their participation and facilitated their authentic accounts. Overall, reflection on interview and focus group recordings helps to identify potential threats to the quality of the data, such as the interviewer unnecessarily interrupting participants and/or failing to adequately listen to their accounts, thereby missing cues to prompt the interviewee or focus group participant for further information.

Tuesner (2016, p. 89) provides a practical strategy for exploring validity in her qualitative investigation as an insider researcher in the field of occupational health and safety. She posed a series of questions to facilitate reflection on potential threats to validity including querying if there was potential for the researchers relationship with the researched to negatively impact on the quality' of the data and if the researchers existing knowledge could lead them to make incorrect assumptions about their data.2 In the current study, many of the young people had experience of the care, education, health and/or social services systems and had become disillusioned in recounting the stories of their lives over and over within different contexts.This was highlighted by' one supervisor when she described her first meeting with a young person who expressed frustration at being asked questions stating‘read my file, it’s all there’. In light of this information, it was important to consider participants’ perceptions of the researcher as a potential threat to the authenticity of the study'. It was decided to overcome any' potential threat to capturing young people’s full accounts by' explaining to them that the researcher was not a criminal justice employee and therefore did not have familiarity with their cases. Time was taken to explain why they were selected for the study and their reflection and critique of the system without judgement was encouraged. Care was also taken to seek clarification on unfamiliar terms used by the young people to avoid misrepresentation of their accounts. The following illustrates the approach where Robert is asked to explain his assertion that prison ‘is not worth it’. Had the researcher not sought guidance from Robert on what he meant, she could have imposed an entirely different and wholly' inaccurate interpretation on the phrase ‘it’s not worth it’:

Robert: I don’t want to go back to prison - it’s not worth it.

MS: You’ve said that quite a bit - it’s not worth it - what do you mean

by' that?

Robert: It s just not worth goin’ down that track ‘cause when you get locked

up, yer just goin’ to keep goin’ back an’ back an’ back, d’ye know like that? Because, they say in prison, wherever ye write your name, you’re always goin’ to come back to that. An’ one of me mates were in there, has his name all over one of them prison walls, an’ ever since he put that there he’s always been back to that cell, since he writ his name he’s been goin’ back to them an’ all, so they always say, wherever ye write yer name ye’ll always come back ... I’ve scribbled out all me names, I scribbled all me names out, just in case.

The final strategy used during the data collection process to support the development of credible research findings was to record fieldwork annotations and observations after each interview and focus group. The purpose was to provide a consistent mechanism for reflecting on the data and a baseline record to cross-check, later in the study, that the core features of young people’s accounts were captured through the data analysis process. In Figure 16.1, an example is provided of an annotation from an interview with a young female participant (aged 18 years) who had moved away from offending behaviour.

Systematic analysis of data: developing credibility through the analysis process

Given the level of data and multiple perspectives gathered in this study, the issue of how it would be managed and analysed was to the forefront from the beginning. The analysis involved a rigorous interrogation, which involved deconstructing the interview transcripts and a systematic rebuilding of the data into key categories and themes (see Figure 16.2).

A number of phases were involved. In Phase 1 each interview and focus group transcript was examined and segments of meaning were coded. This stage of coding, known as open coding, involved the researcher attaching labels to segments of the data in a way that summarised what was occurring in the data. The example that follows, from a young person’s interview transcript, describes his views on the challenges of being under the supervision of the Probation Service in the community. The segment of text was coded as ‘challenges of probation supervision’:

It’s partly being locked away now, but you’re not locked away. It’s kind of like being locked away, but you are not locked away.... It’s hard like, because you can’t really go and do your own thing, you are not free like, you are still under like, you’re still, like you are not, you are still under the law like you know what I mean? You can’t go away and do what you want like.

Bazeley (2013, p. 151) points out that while a number of factors impact the way researchers approach data coding and interpretation including their disciplinary

Name f Sources

References

0 kQ Theme 1 Community

0

»1

ffi O Community Relatio

24

196

ffi O Community Structur

26

124

® Q Family

43

507

O Otood'og

25

236

a O References p

12

42

® O Sch°°|

12

31

O System

2

5

Q Q Theme 2 Supervision

0

0

ffi o Change Over Super

19

207

® O Probation Officer Ex

13

51

ffi O Profile and Backgro

19

111

ffi Q Promoting Complia

13

22

® O Reindin9 to Non

6

46

® O The Report and the

15

25

ffi O Young People and

28

393

® o Young People and T

23

1218

® o Young People's Exp

20

253

i Q Youth Justice Courts

23

304

S) 0 Youth Justice North

8

425

Well before it «is annoying Before it «is 1 dee, oh God 1 doo t waa to g o os-er to her today. I'm not able to listen to her. «tat is she going to find out today* What charges is she gomg to say I have”1 You know' And then it «is like you're actually King to her, and she s looking down at her papers and she's going ah «eD I just tang tp the Gerda station and you do hast charges, aid I'm hke no I don't, and she’s like ah w do. do yvukno-.v' Thtyhave everything, they can check anytang Anythmg they want to know, they find out And I learned that dowtyas I «ent along. And it's better, it's a helping hand »ally. Like I could

rmg my Probation Officer and I could sayl'm really basing a bad day 1 don't know «hat to do, I just don't know «hat when I'm tn a bad mood I blank, you know, I kind of go «eird, I blank and I'd go into town and I'd rob and tt’s nothmg tome butFdnng my Probation Officer and she'd calm me do«n She'd talk to me for an hour like. Before, «hen I «is pregnant my mam and I «is killing each other and my Probation Officer just met me and « e «ent to the cafe and Do you know, she « as ahvays there for me. dial's «hat it «is I never really had anyone there for me. and it's like I can look upto my Probation Officer if ±e| day of my probation ending «ill be die day that I don't know «here I'm going to go in hfe. you know' It's going to be hke I'm neser gomg to have my Probation Officer, but I know I «ill always have her no matter if I'm not on probation, I know 1'11 al«iys have her to turn back to She's vety good to me like Oh she'san angel like She's buBxant hke She practically-put me on the nano«’ road like, she sat me tn a room. I don t know how she put up «ithtnelike. she sat me m a room so mans'ernes hke and drilled it mtomybram about everything aid I just laughed at her really at the start And then she just stuck «i th me like. She just stuck with me.

Annotations integrate hard data recorded in transcripts with soft data that captures contextual factors such as field notes and observations, coding assumptionsand even researchers thoughts and ideas as illustrated in this example.

Annotated text in transcript Researchers linked note

Qualitative data is time and context bound, so capturing context is a core value that philosophically underpins the entire paradigm

Content

The issue that strikes me most from Sarah's interview is the consistent support provided by her probation officer. Her account of how her probation officer stuck with her' despite multiple attempts to reject her was noteworthy. At the time of interview, Sarah said that she had not offended in 12 months following a history of persistent petty offending since childhood. She attributed her probation officer's consistent support, combined with the birth of her baby, as the reasons for stopping offending. However, her voice lowered and her positive demeanour subsided dunng interview when she referred to a time in the future when supervision would end. She expressed concern saying the day of my probation ending will be the day I don't know where I'm going to go in life'.

Figure 16.1 Example of an annotation background, epistemology, methodology and personal influences, a reasonable expectation is that consistency exists across the data coding in a project and data should be interpreted and coded into categories that “make sense” to an external evaluator or study informant. To ensure consistency in the coding process, each code in this study had a clear label or rule for inclusion attached.The rule for inclusion in the aforementioned code ‘challenge of probation supervision’ was that it referred to young people’s descriptions of the challenges of probation supervision. The practice of systematically labelling each code meant that an external validator could understand why segments of text (data) were coded in particular ways.

Phase 1 resulted in over 166 individual codes. In Phase 2 these codes were merged, renamed and reordered and related codes were clustered into categories and themes (as illustrated in Figure 16.2).The coding stage known as axial coding sought to construct linkages between the data and to build a framework that brought focus to the enquiry. The process of writing descriptive memos also commenced in Phase 2 (see Figure 16.3 for an example of a descriptive memo) and was the starting point in translating raw data into research findings.

Theme

Sub-theme

Categories of Codes

Codes

Acceptable reasons

Informal responses to non-compliance

Unacceptable

Management of

Theme 2 Supervision

Responding to non compliance

Formal responses to non-compliance

Managerial & team support

Custody not a deterrent

Being consistent

Absence of unmediate consequence Barners to

proceedings Breach is an opportunity to re-engage

proceedings

Example of workflow from codes to categories of codes to themes

Figure 16.2 Flow chart of themes, categories and codes

Axial coding

4- Name ’ Sources

References

ffi 0 Theme 1 Community

0

0

ffi O Theme 2 Supervision

0

0

ffi 0 Change Over Supervision

18

50

ffi O Probation Officer Expectations of YP

13

51

ffi O Profile arid Background of YP

19

111

ffi O Promoting Compliance Probation Officers

13

22

ffi 0 Responding to Non-Compliance Probation Office

6

46

ffi O Report and the PO Court Relationship

15

25

ffi 0 Young People and Negotiating Compliance

28

393

ffi O Youn9 People and The Decision to Comply

23

1218

ffi 0 Barriers to Stop Offending

16

41

0 Benefits of Compliance

17

37

ffi 0 Challenges to Probation Supervision Complia

16

95

ffi O Conversation with Self when Contemplating

15

25

ffi 0 How 1 Used to be wtih Probation

19

274

® 0 Motivation to Comply

22

324

0 O Motivation to Stop Offending

20

81

0 Avoid Court and or Custody

18

52

0 Employment or Training

0

0

ffi 0 Family or Partner

15

104

0 Offender Fatigue

11

28

0 Probation Officer

2

2

0 Self Motivation

22

ffi 0 New me

123

0 Pending Charges and Suspended Sentences

9

17

ffi 0 Perception of Being Caught

0

0

ffi 0 Planned or Unplanned Noncompliance

15

22

ffi 0 Reaction of PO for Negative Compliance

13

^^5

ffi 0 Reaction of PO for Positive Compliance

17

1^

ffi 0 Reasons for Non-Compliance

0

0

Drag selection here to code to a new node

In Nodes

Example of Descriptive Memo x 0 Motivation to Stop Offending

Descriptive memo -1

ition to stop offending

Click to edit

motivations to stop offending. Four broad issues and custody (18 sources / 52 references); Family I references); Fatigue with offending lifestyle (11

All 20 participantsVsources) talked about the! emerged: Wishing t avoid court appearance^ reasons (including partner) (15 sources / 10-sources / 28 reference» Self-motivation (10 soirees / 22 references). In only two cases, the young people directly attributeltheir motivation to stc® offending to their probation officers.

Avoiding repeated courtppearances and alcustodial sentence was identified as a strong motivator to stop offendingVarticipants were a» would no longer benefit fro mV he more lenient ra that they were now at greateiVisk of receiving J system. Family reasons wereyub-divided in« references), other family or doseYuardian (6 sour references). The importance of chikren in the live sources and references. Children coiVs1 nephews. Overwhelming participants X not experienced themselves as children and child relatives' childhood. In relati

ted of their! inted to be I I Reasons all o o other there fc

centred on three main issues: wanting to recognising that they had 'too much to lose'Yy reoffel had partners - they did not want to risk losinVthem I half of participants described fatigue with offending « to stop offending. Many who referred to offerer f justice system and with the physical, mental, ®d Self-motivation was the final motivation; participât wanting to go back to offending, wanting to get Y aspirations such as training and employment and sp happier and more content with their lives when not of

d 18 and 19 years at the time of the research so jime of the youth justice system. They identified 1 custodial sentence in the adult criminal justice 1 children/younger relatives (9 sources / 25 les / 14 references) and partner (5 sources / 14 1 of participants was reflected in the number of wn children, younger siblings, and nieces and lere for their children in a way that many had o included not wanting to miss their children [adult) family, motivation to avoid offending 1 their family; not causing any further trouble; Iding. The motivation was clear for those who reoffending and risking incarceration. Over aid withan offending lifestyle as a motivation faigue described being 'sick of the criminal emotional impact of an offending lifestyle. 3 leferred to moving on with their lives, not bole from life, pursuing present and future airing pursuits. Participants described being felling which of itself provided motivation.

Descriptive memos linked to the sub-category "Motivation to Stop Offending" under the main category "Young People and the Decision to Comply" allowed the researcher reduce the data from 20 interviews containing 81 individual quotes in this category to a document describing and summarising coded content reducing the

sub-category to manageable proportions.

Figure 16.3 Example of descriptive memo

Phase 3 involved breaking down the now reorganised data so as to better understand the meanings embedded within it. For example,‘motivation to stop offending’ (illustrated on the left-hand panel of Figure 16.3) was broken down into the components parts: ‘avoid court and/or custody’; ‘family or partner’; ‘offender fatigue’;‘probation officer’; and ‘self-motivation’.

In Phase 4, theoretical coding, the task involved writing analytical memos to develop deeper and more systematic thinking about the emerging findings (see Figure 16.4 for an example of an analytical memo) and to consider their importance in relation to existing literature. Memos varied but broadly included some or all of the following: content of the code; the processes included within the code; its importance in sequencing events and outcomes that influenced the overall findings; the relevance of background factors; how the code related to other codes and how it differed; and consideration of the broader literature in locating the study findings in context.

The final phase of analysis involved validating the analytical memos through critical reflection such as exploring potential alternative explanations for the emerging findings and exploring how the findings related to each other and to the wider literature in the field of offender compliance.

Audit trail

The production of an audit trail for the data analysis is an important criterion on which the credibility of the research can be established. The audit trail was originally proposed by Lincoln and Guba (1985) as a mechanism to ensure that findings were grounded in the data and not unduly influenced by the researcher’s bias or judgement. An audit trail lists the processes undertaken by the researcher with each process supported by evidence and documented in the main body or appendices of any resulting report. One of the key pieces of evidence in an audit trail is the production of a codebook showing each of the cycles of coding undertaken during the encoding process. The codebook lists the codes produced during each phase of coding, their purpose (rule for inclusion) and the levels of data they attracted (expressed as number of sources and references as illustrated in the left-hand panels of Figures 16.3 and 16.4).The codebook allows the researcher to make two statements:

The work was carried out and not just claimed to have been done;

The coding was conducted in a manner consistent with the guidelines in literature from the chosen method of analysis (for example thematic analysis; grounded theory).

Other evidence documented as part of an audit trail includes comprehensive records of decisions made throughout the fieldwork as well as fieldwork annotations and observations and descriptive and analytical memos as previously discussed. The audit trail brought transparency to the study in demonstrating consistency of approach in the research processes and the strategies utilised

Axial coding

M Example cf Analytical Memo x

Name ' Sources

References 0

0

B O Th*™* 2 Supervision

8 O Change Over Supervision __

“ 207

© O Change in Compliance over Tim<^^

14

73

(2) Change in Supervisory Relationship over Time

18

50

8 Q The Turning Point and the Future

19

84

Q The Turning Point

10

43

O Five Year Plans

19

41

© Q Probation Officer Expectations of VP

13

51

IS O Profile and Background of YP

19

111

© O Promoting Compliance Probation Officers

13

22

© O R**P°nding to Non-Compliance Probation Office

6

46

® O and the PO Court Relationship

15

25

$ O Y°vng People and Negotiating Compliance

28

393

8 O Young People and The Decision to Comply

23

1218

© Q Barriers to Stop Offending

16

41

0 Benefits of Compliance

17

37

© O Challenges to Probation Supervision Complia

16

95

ffi 0 Conversation with Self when Contemplating N

15

25

© 0 How 1 Used to be wtih Probation

19

274

© O Mot,v4t'°n to Comply

22

324

© O Motivation to Stop Offending

20

81

© 0 New me

14

123

0 Pending Charges and Suspended Sentences

9

17

© 0 Perception of Being Caught

0

0

© O P*4nne<l <* Unplanned Noncompliance

15

22

© 0 Reaction of PO for Negative Compliance

13

15

© 0 Reaction of PO for Positive Compliance

17

18

© 0 Reasons for Non-Compliance

0

0

______________________________________________________________________________________Okie to edit________ Changing perceptions of supervision over time

In this study, change occurred in two-thirds of young people's perceptions of probation supervision over time supporting Robinson and McNeilTs (2010) hypothesis that attitudes towards compliance and compliant behaviour are not fixed or static for offenders on community disposals. While probation officers described having concerns about some young people's motivation to comply in the early stages of supervision, they reported that the level of willingness to comply improved considerably as young people progressed through the supervision process. Probation officers' views were supported in young people's accounts. Dylan reflects the views of other young people in describing the change in his perception of supervision over time:

I

... she (probation officer] was tellin' me what to do. And especially a woman tellin' me what to do. I'd issues around that big time. And I resisted against it for a year or more, but then I was seein' to know (starting to see] that she was trying to help me. She wasn't against me like, d'you know? ... My mam went to jail... and that's when me and (probation officer's] friendship blossomed really'cause she had me in every day, 'cause she was worried about me, 'cause I'd have mental health problems too, suicide and all that kind of shit. And she helped me through it like.

Improved attendance, an openness to participate at supervision appointments and readiness to take advice and direction from their probation officers characterised the kinds of behavioural that young people described as arising from their changed perceptions. Their descriptions suggest that they had moved beyond undertaking the minimum requirements of supervision (formal compliance) to more active and meaningful participation (substantive compliance) (Bottoms 2001).

Analytical memos allowed the researcher conduct a systematic review of her thematic framework in relation to extant literature.

Je At Enter node поте (CTRL *Q)

Figure 16.4 Example of an analytical memo to support the authenticity of the findings. Silverman (2017, p. 385) advises researchers of the need to overcome the “problem of anecdotalism” and “convince themselves (and their audience) that their ‘findings’ are genuinely based on critical investigation of all their data and do not depend on a few well-chosen ‘examples’”.

 
Source
< Prev   CONTENTS   Source   Next >