Unit 7 - Quality in qualitative research

7.1 - Welcome to the Unit

Welcome to Unit 7. In this unit we will return to some of the methods explored in previous units and explore how these contribute specifically toward quality in qualitative research. The quality of qualitative research is important, and we must continually strive to improve our methods and approach to rigorous qualitative research. However, quality is not a simple concept to evaluate, and this unit will explore some of the leading debates in the field rather than prescribe a one-size-fits-all approach to quality assessment.

7.2 - What is quality

Qualitative researchers must resist the urge to describe their studies in ways more familiar to quantitative researchers. Using terms more associated with positivist paradigms limits commitment to qualitative values and restrains qualitative insight through a lens of bias reduction. Qualitative researchers should, instead, be confident, and committed, to reporting their study in a way more commensurate with qualitative inquiry and describe their means and methods to achieve what is known as trustworthiness.

Trustworthiness is a judgement of how confident we can be in the findings derived from a qualitative study. Typically, four domains are associated with trustworthiness: credibility, dependability, confirmability and transferability,¹ and reflexivity is key to them all. In qualitative research, the researcher is an active participant in data creation and data analysis; any findings are subsequently created in a way they feel is appropriate. None of this is a passive process; therefore, researchers must be self-aware of these influences and decisions and continuously reflect on how these may be shaping their study.²

Domain	Definition in brief	Definition in full (taken from Korstjens and Moser2(p.121) based on Lincoln and Guba1
Credibility	Representativeness of the findings to the sample	The confidence that can be placed in the truth of the research findings. Credibility establishes whether the research findings represent plausible information drawn from the participants’ original data and is a correct interpretation of the participants’ original views.
Dependability	Consistency of the inquiry process including data collection and analysis	The stability of findings over time. Dependability involves participants’ evaluation of the findings, interpretation and recommendations of the study such that all are supported by the data as received from participants of the study.
Confirmability	Findings are clearly derived from the data	The degree to which the findings of the research study could be confirmed by other researchers. Confirmability is concerned with establishing that data and interpretations of the findings are not figments of the inquirer’s imagination, but clearly derived from the data.
Transferability	Findings have resonance outside the study sample	The degree to which the results of qualitative research can be transferred to other contexts or settings with other respondents. The researcher facilitates the transferability judgement by a potential user through thick description.
Reflexivity	Critical reflection on how the ‘self’ influences the research process	The process of critical self-reflection about oneself as researcher (own biases, preferences, preconceptions), and the research relationship (relationship to the respondent, and how the relationship affects participant’s answers to questions).

7.3 - Ways to increase quality

It is important to state that there are many ways to increase quality in qualitative studies. Not all of these can, or should, be applied to all qualitative studies. In comparison to quantitative research, quality in qualitative research is not an easy concept to define. While the parameters of validity, reliability, bias and generalisablity are somewhat universally accepted in quantitative research, the heterogeneity within the field of qualitative methodology means that any similar parameters applied to all qualitative research quickly become restrictive or redundant.

To inform decisions about quality markers, researchers must understand their research philosophy, their methodology and the implications of adopting any methods to improve the quality of their study. Several methods are presented in Table 7.2 to achieve trustworthiness. However, researchers do not have to use all of these in the same study and some of these may actually be inappropriate for their methodology. Therefore, the researcher needs to make an informed decision about which to adopt and how this will advance the rigour of their study.

“Quality concepts and practices are not necessarily cast in stone – you may be able to re-work them to better suit your research”³ (p.278)

For example, the process of member checking (i.e. returning data to participants to confirm the accuracy), and respondent validation (i.e. where findings are returned to participants to check meaningful interpretation) has been encouraged to ‘validate’ researcher findings.^4,5 In doing so researchers can avoid misinterpretation or misrepresentation of the data and increase the credibility of their final results.⁶

However, the value of member checking in more interpretive inquiry has been questioned as these interpretations may not always be recognisable to participants.³ Similarly in narrative methodology researchers are encouraged to stand by their analytical work and take responsibility for the findings “in the final analysis, the work is ours. We have to take responsibility for its truths”.⁷ (p.67 )

Where respondent validation is not used, researchers may want to consider other methods such as longitudinal designs and prolonged engagement in the field, so they have a deeper understanding of participant experiences and can be more confident in representing this. Longitudinal designs also offer the opportunity to explore emerging analytical ideas with participants in follow-up interviews to aid later interpretation.

Another example of methods that do not suit all qualitative designs is the use of additional researchers to check analysis. In the past this process was used to check if the analysis was ‘correct’. Some post-positivist, small q, studies may even use ‘inter-rater reliability’ to check coding accuracy. However, more attention is now paid to the competence and reflexivity of the person analysing the data and that a meaningful interpretation, rather than a ‘correct’ interpretation, has been achieved.⁸ Having additional researchers to work through the data is then used to advance interpretation rather than confirm interpretation is correct.³ See Unit 6 for an extended discussion of analytical techniques.

This discussion demonstrates that the appropriateness of the strategies described in Table 7.1 should be evaluated during study design and conduct. The choice of which to use and why should be informed by the research question, philosophical position and methodology.

Domain	Strategy	Methods
Credibility	Prolonged engagement	Long interviews and/or observations Longitudinal designs Multiple data collection points
	Triangulation of data and methods	Multiple sources of data, multiple researchers, multiple methods
	Member checking	Returning data to participants to check accuracy
	Respondent validation	Checking interpretation of data with participants and building responses into the analytical process
	Expertise	Consultation/supervision from a qualitative expert Evidence of qualitative training
	Peer debriefing	Sharing of analysis with peers to sense-check meaningful interpretation
Dependability and confirmability	Audit trail	Clarity of methods and procedural rigour reported in a study Transparency of process involved to analyse data from codes to findings
	Authenticity	Present raw data in the form of direct quotes Ensure participants are fairly represented – i.e. do not overly rely on quotes from one person Findings are well grounded and supportable
Transferability	Thick description	Contextual / demographic information for participants Provision of rich detailed quotes
Reflexivity	Diary	Maintain a reflexive diary, or/and, use memoing and use in the analytical process

---

Table 6.2: Strategies to increase quality in qualitative research.

7.4 - Qualitative checklists

There are several different checklists that can support researchers to conduct qualitative research. These checklists can be used to evaluate the methods used in a qualitative study (see Critical appraisal checklists like CASP⁹) or to report qualitative research (see reporting checklists like COREQ¹⁰ and SQRS¹¹). While these checklists are not without their critics, they can be a useful tool, especially for novice researchers, to understand the methods of qualitative research while designing and reporting their studies. Concern regarding checklists stems from their blanket use in qualitative studies. To apply such checklists requires uniformity of qualitative work; however, as discussed previously, qualitative research is an extremely heterogeneous methodology, which makes standardised assessment of rigour particularly challenging.

The quote below by Dixon-Woods et al.¹² illustrates the division within the field of qualitative research: at one end are those calling for clear criteria to determine quality, and at the other those who advocate against any such criteria. Those who are sympathetic to the latter position would say criteria are restrictive and restrain the interpretive discipline through the application of a positivist lens to quality assessment.

“It is clearly important that policymakers and practitioners can have confidence in the quality of such research.^[A] There is, however, disagreement not only about the characteristics that define good quality qualitative research, but also on whether criteria for quality in qualitative research should exist at all. Many argue that a set of criteria distinct from those applied to natural scientific quantitative approaches and specifically designed for qualitative research is required.^[A-C] However, others have called for an end to ‘‘criteriology’’,⁷ arguing that this leads to privileging of method as a ‘‘sacred prescription’’ rooted in positivist philosophical traditions, and the stifling of the interpretive and creative aspects of qualitative research. Still others argue that criteria are best regarded as guides to good practice^[E] rather than as rigid requirements in appraising papers.”^14-16 p.223 A Lincoln YS, Guba EG. Naturalistic inquiry. Newbury Park, CA: Sage, 1985. 5 B Seale C. Quality issues in qualitative inquiry. Qualitative Social Work 2002;1:97–110. C Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res 1998;8:341–51. D Schwandt TA. Farewell to criteriology. Qualitative Inquiry 1996;2:58–72. E Henwood K, Pidgeon N. Qualitative research and psychological theorising. Br J Psychol 1992;83:97–111.

However, in the context of evidence-based practice, a fundamental stage is the appraisal of evidence and it would be unjustifiable simply to omit this for qualitative research.¹³ While there are some important concerns raised by those who reject criteria-based appraisal, the methods used by qualitative researchers must be laid bare and properly scrutinised. Otherwise, we run the risk of reinforcing a view that qualitative research is not rigorous, thereby further reducing the potential impact of this important methodology. This led Williams et al.¹³ to state:

“Given its scope and its place within health research, the robust and systematic appraisal of qualitative research to assess its trustworthiness is as paramount to its implementation in clinical practice as any other type of research.” ¹³

7.4.1 - Critical appraisal checklists

There are a large number of formal checklists to appraise qualitative research. The most well known and well used are probably the Critical Appraisal Skills Programme (CASP)⁹ and the Joanna Briggs Institute (JBI)¹⁴ tools for qualitative research. Criticism of these tools stems from the fact that they tend to mimic quantitative appraisal tools for their emphasis on methods that are present or not, rather than provide an in-depth evaluation of the impact these may or may not have had on the findings of the study. Therefore, many researchers prefer to evaluate a study against the domains presented in Table 7.2 (credibility, dependability, confirmability, and transferability). In order to use these domains the appraiser must be familiar with these concepts which may be easier for those with more experience in qualitative methods.¹³ In contrast, Greenhalgh⁸ posed nine questions to frame quality appraisal in qualitative research:

1. Did the paper describe an important clinical problem addressed via a clearly formulated question?
2. Was a qualitative approach appropriate?
3. How were (a) the setting and (b) the subjects selected?
4. What was the researcher’s perspective, and has this been taken into account?
5. What methods did the researcher use for collecting data – and are these described in enough detail?
6. What methods did the researcher use to analyse the data – and what quality control measures were implemented?
7. Are the results credible, and if so, are they clinically important?
8. What conclusions were drawn, and are they justified by the results?
9. Are the findings of the study transferable to other settings?

Another challenge in appraising quality in qualitative research is that the checklists do not usually differentiate between different methodological approaches, such as phenomenology, narrative inquiry or grounded theory. Instead, one qualitative checklist is often used for all these different approaches. As explained in Unit Two, there are distinct approaches to qualitative research which will often have their own expected reporting characteristics e.g. theoretical sampling in grounded theory, unstructured interviews in narrative inquiry, observations in ethnography. Any deviation from these methods needs to be considered very carefully because of the potential impact on the findings of the study. Therefore, appraisal of qualitative research must be done by someone with knowledge and understanding of the qualitative field.

7.4.2 - Reporting criteria

Unfortunately, there are many examples in the literature of qualitative research that is not well reported. This has significant consequences for the state of qualitative research in the evidence base and reduces its impact in clinical practice. Reporting criteria are a useful way to help researchers to comprehensively describe their methods in journal publications. The most commonly used reporting guidelines¹⁵ are the COREQ (COnsolidated criteria for REporting Qualitative research)¹ and the SRQR (Standard for Reporting Qualitative Research).¹¹

COREQ is a 32-item checklist for study reporting qualitative interviews and focus groups; full-text is available from: here.
SRQR is a 21 Items checklist that the authors consider esential for complete, transparent reporting of qualitative research available from here.

Journals will often ask for this checklist to be completed and uploaded as a specific file during manuscript submission. In this submission researchers will identify where in the manuscript the reporting item is located and an account of how this was enacted in the study. Researchers should be mindful that not all the criteria are appropriate for all qualitative studies so they should not use the checklist in a mechanistic way. Think about each of the criteria and provide a defence if there are methods that were not appropriate for the way the study was conducted.

Most reporting criteria can be sourced on the EQUATOR Network (https://www.equator-network.org/) which is an online resource to support researchers to write and publish high-impact health research. However, if using reflexive Thematic Analysis by Braun and Clark (REF) you may want to consider using their own guide for editors and reviewers before submitting for publication (https://www.thematicanalysis.net/editor-checklist). Another, checklist relevant to qualitative research is ENTREQ (Enhancing transparency in reporting the synthesis of qualitative research).¹⁶This checklist is for secondary studies i.e. qualitative evidence synthesis.

7.5 - Unit summary

In this unit, we explored quality in qualitative inquiry and the transparent reporting of qualitative studies. While there is no one-size-fits-all approach to quality in qualitative research, there are strategies that can be adopted to improve quality in these studies. Judging quality in qualitative studies requires knowledge and understanding of this field of inquiry so that novel and complex studies are not dismissed in favour of more simplistic, or post-positivist, designs.

References
1. Lincoln YS, Guba EG. Naturalistic Inquiry. Sage; 1985.
2. Korstjens I, Moser A. Series: Practical guidance to qualitative research. Part 4: Trustworthiness and publishing. Eur J Gen Pract. 2018;24(1):120-124. doi:10.1080/13814788.2017.1375092
3. Braun V, Clarke V. Thematic Analysis, a Practical Guide. SAGE; 2022.
4. Varpio L, Ajjawi R, Monrouxe LV, O’Brien BC, Rees CE. Shedding the cobra effect: problematising thematic emergence, triangulation, saturation and member checking. Med Educ. 2017;51(1):40-50. doi:10.1111/medu.13124
5. Morse JM, Field PA. Nursing Research: The Application of Qualitative Approaches. 2nd ed. Stanley Thornes; 1996.
6. Holloway I, Wheeler S. Qualitative Research in Nursing. 2nd ed. Blackwell; 2002.
7. Riessman C. Narrative Analysis. SAGE Publications; 1993.
8. Greenhalgh T. How to Read a Paper : The Basics of Evidence-Based Medicine. Fifth edition. John Wiley & Sons; 2014.
9. Critical Appraisal Skills Programme. CASP Checklist: 10 questionsto help you make sense of a Qualitative research. Published 2018. https://casp-uk.net/
10. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349-357. doi:10.1093/intqhc/mzm042
11. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research: A Synthesis of Recommendations. Academic Medicine. 2014;89(9). https://journals.lww.com/academicmedicine/Fulltext/2014/09000/Standards_for_Reporting_Qualitative_Research__A.21.aspx
12. Dixon-Woods M. The problem of appraising qualitative research. Quality and Safety in Health Care. 2004;13(3):223-225. doi:10.1136/qshc.2003.008714
13. Williams V, Boylan AM, Nunan D. Critical appraisal of qualitative research: necessity, partialities and the issue of bias. BMJ Evid Based Med. 2020;25(1):9-11. doi:10.1136/bmjebm-2018-111132
14. Joanna Briggs. Checklist for Qualitative Research. Published 2017. https://jbi.global/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Qualitative_Research2017_0.pdf
15. Walsh S, Jones M, Bressington D, et al. Adherence to COREQ Reporting Guidelines for Qualitative Research: A Scientometric Study in Nursing Social Science. International Journal of Qualitative Methods. 2020;19:160940692098214. doi:10.1177/1609406920982145
16. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Medical Research Methodology. 2012;12(181). doi:10.1186/1471-2288-12-181