Skip to content

Unit 6: Quality in Qualitative Research

NEURO_QUAL Podcast Quality in Qualitative Research

COMING SOON!

6.1 Unit introduction

Welcome to the final unit in this training resource. In this unit we will return to some of the methods explored in previous units and explore how these contribute specifically toward quality in qualitative research. The quality of qualitative research is important, and we must continually strive to improve our methods and approach to qualitative research. However, quality is not a simple concept to evaluate, and this unit will explore some of the leading debates in the field rather than prescribe a one-size-fits-all approach to quality assessment.

Reflection Point

Would you recognise a ‘good’ qualitative study? What factors inform this evaluation?

6.2 What is quality?

In comparison to quantitative research, quality in qualitative research is not an easy concept to define. While the parameters of validity, reliability, bias and generalisablity are somewhat universally accepted in quantitative research, the heterogeneity within the field of qualitative methodology means that any such parameters applied to qualitative research quickly become restrictive or redundant. Therefore, before discussing attempts to define criteria for quality, we first turn attention to language and how quality is described in qualitative studies.

First, qualitative researchers must resist the urge to describe their studies in ways more familiar to quantitative researchers, as these are usually inappropriate. Instead, researchers should describe their means and methods to achieve what is known as trustworthiness. Trustworthiness is a judgement of how confident we can be in the findings derived from a qualitative study. Typically, four domains are associated with trustworthiness: [credibility], [dependability], [confirmability] and [transferability],1 and [reflexivity] is key to them all. In qualitative research, the researcher is an active participant in data creation and data analysis; any findings are subsequently created in a way they feel is appropriate. None of this is a passive process; therefore, researchers must be self-aware of these influences and decisions and continuously reflect on how these may be shaping their study.2

Table 1: Domains of trustwothiness

Domain Definition in brief Definition in full*
Credibility Representativeness of the findings to the sample The confidence that can be placed in the truth of the research findings. Credibility establishes whether the research findings represent plausible information drawn from the participants’ original data and is a correct interpretation of the participants’ original views.
Dependability Consistency of the inquiry process including data collection and analysis The stability of findings over time. Dependability involves participants’ evaluation of the findings, interpretation and recommendations of the study such that all are supported by the data as received from participants of the study.
Confirmability Findings are clearly derived from the data The degree to which the findings of the research study could be confirmed by other researchers. Confirmability is concerned with establishing that data and interpretations of the findings are not figments of the inquirer’s imagination, but clearly derived from the data.
Transferability Findings have resonance outside the study sample The degree to which the results of qualitative research can be transferred to other contexts or settings with other respondents. The researcher facilitates the transferability judgement by a potential user through thick description.
Reflexivity Critical reflection on how the ‘self’ influences the research process The process of critical self-reflection about oneself as researcher (own biases, preferences, preconceptions), and the research relationship (relationship to the respondent, and how the relationship affects participant’s answers to questions).

* taken from Korstjens and Moser2 (p.121) based on Lincoln and Guba1

Key Points: What is research philosophy?

  • Do not revert to quantitative terms and methods when evaluating qualitative studies.
  • Report strategies that increase the trustworthiness of the study.
  • Trustworthiness includes: credibility, dependability, confirmability and transferability, and reflexivity is key to them all.

6.3 Strategies to increase quality

It is important to state that there are many ways to increase quality in a qualitative study – and not all are relevant to all types of qualitative study, while some would be considered inappropriate for particular studies. This is where a balanced approach to quality is required: researchers might not have employed one particular method, but they have done something else to counteract the omission. For example, emphasis was traditionally placed on member checking (i.e. returning data to participants to confirm the accuracy), and respondent validation (i.e. where findings are returned to participants to check meaningful interpretation). However, more recently this reliance on respondent validation and member checking has been questioned. Instead, researchers may use longitudinal designs and prolonged engagement in the field, so they may not feel it necessary to use such strategies because they have had an opportunity (through follow-up interviews) to explore their early interpretation with participants. Another example is the use of additional researchers to check analysis. In the past this process was used to check if the analysis was ‘correct’. But discussion has again moved on, and more attention is now paid to the competence and reflexivity of the person analysing the data and that a meaningful interpretation, rather than a ‘correct’ interpretation, has been achieved.3

Therefore, the appropriateness of common strategies used to improve trustworthiness described in Table 2 should be explored while designing the qualitative protocol. Some studies will use very few of these, others will use many – and, unfortunately, some will report none. All qualitative studies should employ some strategies to improve quality. However, choice of which to use and why should be informed by your research question, methodology and philosophical position.

Table 2: Strategies to increase quality in qualitative research

Domain Strategy Methods
Credibility Prolonged engagement Long interviews and/or observations
Longitudinal designs
Multiple data collection points
Triangulation of data and methods Multiple sources of data, multiple researchers, multiple methods
Member checking Returning data to participants to check accuracy
Respondent validation Checking interpretation of data with participants and building responses into the analytical process
Expertise Consultation/supervision from a qualitative expert
Evidence of qualitative training
Peer debriefing Sharing of analysis with peers to sense-check meaningful interpretation
Dependability and confirmability Audit trail Clarity of methods and procedural rigour reported in a study
Transparency of process involved to analyse data from codes to findings
Authenticity Present raw data in the form of direct quotes
Ensure participants are fairly represented - i.e. do not overly rely on quotes from one person
Transferability Thick description Contextual / demographic information for participants
Provision of rich detailed quotes
Reflexivity Diary Maintain a reflexive diary and use in the analytical process

Key Points: Strategies to improve quality in qualitative research

  • There is no universal method to increase quality that can be applied to all qualitative studies.
  • Methods that can be used to increase quality include: prolonged engagement; member checking; respondent validation; expertise; peer debriefing; thick description; authenticity; audit trail; reflexivity; triangulation of data and methods.
  • Choice of which to use and why should be informed by the research question, methodology, and philosophical position.

6.4 Qualitative checklists

There are a number of different checklists that can support researchers to do qualitative research better. Critical appraisal checklists (see 6.4.1) are useful to evaluate the methods used in the study, while reporting checklists (see 6.4.2) are an expectation of manuscript submission. Insight and understanding of both types of checklists prior to embarking on qualitative research can help researchers to design a more robust protocol and ensure they do not miss an important reporting item when they conduct the study.

As stated in Unit Two, many researchers and health professionals, treat qualitative research as one unified approach; however, as discussed, qualitative research is extremely heterogeneous, which makes assessment of rigour particularly challenging. The quote below by Dixon-Woods et al. 4 illustrates the division within the field of qualitative research: at one end are those calling for clear criteria to determine quality, and at the other those who advocate against any such criteria. Those who are sympathetic to the latter position would say criteria are restrictive and restrain the interpretive discipline through the application of a positivist lens to quality assessment.

"It is clearly important that policymakers and practitioners can have confidence in the quality of such research4. There is, however, disagreement not only about the characteristics that define good quality qualitative research, but also on whether criteria for quality in qualitative research should exist at all. Many argue that a set of criteria distinct from those applied to natural scientific quantitative approaches and specifically designed for qualitative research is required.4-6 However, others have called for an end to ‘‘criteriology’’,7 arguing that this leads to privileging of method as a ‘‘sacred prescription’’ rooted in positivist philosophical traditions, and the stifling of the interpretive and creative aspects of qualitative research. Still others argue that criteria are best regarded as guides to good practice8 rather than as rigid requirements in appraising papers.” 4 (p.223)

4 Lincoln YS, Guba EG. Naturalistic inquiry. Newbury Park, CA: Sage, 1985.
5 Seale C. Quality issues in qualitative inquiry. Qualitative Social Work 2002;1:97–110.
6 Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res 1998;8:341–51.
7 Schwandt TA. Farewell to criteriology. Qualitative Inquiry 1996;2:58–72.
8 Henwood K, Pidgeon N. Qualitative research and psychological theorising. Br J Psychol 1992;83:97–111.

However, in the context of evidence-based practice, a fundamental stage is the appraisal of evidence and it would be unjustifiable simply to omit this for qualitative research.5 While there are some important concerns raised by those who reject criteria-based appraisal, the methods used by qualitative researchers must be laid bare and scrutinised in an equivalent way to quantitative research. Otherwise, we run the risk of reinforcing a view that qualitative research is not rigorous, thereby further reducing the potential impact of this important methodology. This led Williams et al.5 to state:

“Given its scope and its place within health research, the robust and systematic appraisal of qualitative research to assess its trustworthiness is as paramount to its implementation in clinical practice as any other type of research.” 5

6.4.1 Critical appraisal checklists

There are a large number of formal checklists to appraise qualitative research. The most well known and well used are probably the Critical Appraisal Skills Programme (CASP)6 and the Joanna Briggs Institute (JBI)7 tools for qualitative research. These tools tend to mimic quantitative appraisal tools for their emphasis on methods that are present or not, rather than provide an in-depth evaluation of the impact these may or may not have on the findings of the study. Therefore many researchers prefer to evaluate a study against the domains presented in Table 2, namely: credibility, dependability, confirmability, and transferability. However, in order to use these domains the appraiser must be familiar with these concepts.5 In contrast, Greenhalgh3 poses nine questions to frame quality appraisal:

  1. Did the paper describe an important clinical problem addressed via a clearly formulated question?
  2. Was a qualitative approach appropriate?
  3. How were (a) the setting and (b) the subjects selected?
  4. What was the researcher’s perspective, and has this been taken into account?
  5. What methods did the researcher use for collecting data – and are these described in enough detail?
  6. What methods did the researcher use to analyse the data – and what quality control measures were implemented?
  7. Are the results credible, and if so, are they clinically important?
  8. What conclusions were drawn, and are they justified by the results?
  9. Are the findings of the study transferable to other settings?

Another challenge in appraising quality in qualitative research is that the checklists do not usually differentiate between different methodological approaches, such as phenomenology, narrative inquiry or grounded theory. Instead, one qualitative checklist is used for all these different approaches. As explained in Unit Two, there are distinct approaches to qualitative research which will often have their own expected reporting characteristics e.g. theoretical sampling in grounded theory, unstructured interviews in narrative inquiry, observations in ethnography. Any deviation from these reporting methods needs to be considered very carefully because of the potential impact on the findings of the study. Therefore, appraisal of qualitative research must be done by someone with knowledge and understanding of the qualitative field.

Key Points: Critical appraisal checklists

  • Critical appraisal of qualitative research is required for evidence-based practice and to increase its reach and impact.
  • The most common checklists for qualitative research are by CASP and JBI.
  • Other, less constrained, criteria exist to judge the quality of qualitative research.
  • Few appraisal checklists differentiate between the different types of qualitative research.
  • Appraisal should be conducted by someone with knowledge and understanding of the field.

6.4.2 Reporting criteria

Unfortunately, there are many examples in the literature of qualitative research that is not well reported. This has significant consequences for the state of qualitative research in the evidence base and reduces its impact in clinical practice. Reporting criteria are, then, an essential element of advancing the methodology and increasing its impact.

The most well cited checklist for qualitative research is COREQ (COnsolidated criteria for REporting Qualitative research).8 COREQ is a 32-item checklist for study reporting qualitative interviews and focus groups; the full-text is available by clicking here.

High-impact journals will ask for this checklist to be completed and uploaded as a specific file during manuscript submission. However, in the interests of improving qualitative reporting, authors should still consider the items in the checklist when developing their qualitative manuscript even if a journal does not require this checklist.

Template COREQ Checklist

📝 Click here to download the COREQ Checklist.

All reporting criteria can be sourced on the EQUATOR Network, which is an online resource to support researchers to write and publish high-impact health research. Another, checklist relevant to qualitative research is ENTREQ (Enhancing transparency in reporting the synthesis of qualitative research).9 This checklist is for secondary studies of qualitative evidence synthesis and not for primary research.

Key Points: Reporting checklist

  • COREQ is the most used reporting checklist for primary qualitative research.
  • Be aware of expected reporting criteria when developing a qualitative protocol.
  • Always use a reporting checklist while writing a qualitative manuscript, even if it is not a mandatory requirement of the journal.

Reflection Point

Having completed the final unit in this learning resource, what do you now think about qualitative research? Would you use qualitative methods in the future? What additional learning needs do you have?

6.5 Unit summary

In this unit, we explored quality in qualitative inquiry and the transparent reporting of qualitative studies. While, there is no one-size-fits-all approach to quality, there are strategies that can be adopted to improve quality in these studies. Including items in manuscripts from the list of reporting criteria will also improve the qualitative evidence base. Judging quality in qualitative studies requires knowledge and understanding of this field of inquiry so that novel and complex studies are not dismissed in favour of more simplistic designs.

References

  1. Lincoln YS, Guba EG. Naturalistic inquiry. Beverly Hills, Calif. ; London: Sage 1985. ↩↩

  2. Korstjens I, Moser A. Series: Practical guidance to qualitative research. Part 4: Trustworthiness and publishing. Eur J Gen Pract 2018;24(1):120-24. doi: 10.1080/13814788.2017.1375092 [published Online First: 2017/12/06] ↩↩

  3. Greenhalgh T. How to read a paper : the basics of evidence-based medicine. Fifth edition. ed: John Wiley & Sons 2014. ↩↩

  4. Dixon-Woods M, Shaw RL, Agarwal S, et al. The problem of appraising qualitative research. Qual Saf Health Care 2004;13(3):223-5. doi: 10.1136/qhc.13.3.223 [published Online First: 2004/06/04] ↩

  5. Williams V, Boylan AM, Nunan D. Critical appraisal of qualitative research: necessity, partialities and the issue of bias. BMJ Evid Based Med 2020;25(1):9-11. doi: 10.1136/bmjebm-2018-111132 [published Online First: 2019/03/14] ↩↩↩↩

  6. Critical Appraisal Skills Programme. CASP Checklist: 10 questionsto help you make sense of a Qualitative research 2018 [01/06/2021]. Available from: https://casp-uk.net/. ↩

  7. Joanna Briggs. Checklist for Qualitative Research 2017 [01/06/21]. Available from: https://jbi.global/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Qualitative_Research2017_0.pdf. ↩

  8. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19(6):349-57. doi: 10.1093/intqhc/mzm042 [published Online First: 2007/09/18] ↩

  9. Tong A, Flemming K, McInnes E, et al. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol 2012;12:181. doi: 10.1186/1471-2288-12-181 [published Online First: 2012/11/29] ↩