Validity Matrixx

Validity and Reliability
Although validation of findings occurs throughout the steps in the research process, this discussion focuses on how the researcher writes a passage in a proposal or study on the procedures to be undertaken to validate the proposed study’s findings. Researchers need to convey the steps they will take in their studies to check for the accuracy and credibility of their findings. Validity does not carry the same connotations in qualitative research that it does in quantitative research; nor is it a companion to reliability (examining stability) or generalizability (the external validity of applying results to new settings, people, or samples), topics discussed in Chapter 8. Qualitative validity means that the researcher checks for the accuracy of the findings by employing certain procedures, whereas qualitative reliability indicates that the researcher’s approach is consistent across different researchers and among different projects (Gibbs, 2007).

Defining qualitative validity. Validity is one of the strengths of qualitative research and is based on determining whether the findings are accurate from the standpoint of the researcher, the participant, or the readers of an account (Creswell & Miller, 2000). Terms abound in the qualitative literature that address validity, such as trustworthiness, authenticity, and credibility (Creswell & Miller, 2000), and it is a much-discussed topic (Lincoln, Lynham, & Guba, 2011).
Using multiple validity procedures. A procedural perspective that we recommend for research proposals is to identify and discuss one or more strategies available to check the accuracy of the findings. Researchers should actively incorporate validity strategies into their proposals. We recommend the use of multiple approaches, which should enhance the researcher’s ability to assess the accuracy of findings as well as convince readers of that accuracy. There are eight primary strategies, organized from those used most frequently and easiest to implement to those used occasionally and more difficult to implement:
Triangulate different data sources by examining evidence from the sources and using it to build a coherent justification for themes. If themes are established based on converging several sources of data or perspectives from participants, then this process can be claimed as adding to the validity of the study.
Use member checking to determine the accuracy of the qualitative findings by taking the final report or specific descriptions or themes back to participants and determining whether these participants feel that they are accurate. This does not mean taking back the raw transcripts to check for accuracy; instead, the researcher takes back parts of the polished or semi-polished product, such as the major findings, the themes, the case analysis, the grounded theory, the cultural description, and so forth. This procedure can involve conducting a follow-up interview with participants in the study and providing an opportunity for them to comment on the findings.
Use a rich, thick description to convey the findings. This description may transport readers to the setting and give the discussion an element of shared experiences. When qualitative researchers provide detailed descriptions of the setting, for example, or offer many perspectives about a theme, the results become more realistic and richer. This procedure can add to the validity of the findings.
Clarify the bias the researcher brings to the study. This self-reflection creates an open and honest narrative that will resonate well with readers. Reflexivity has already been mentioned as a core characteristic of qualitative research. Good qualitative research contains comments by the researchers about how their interpretation of the findings is shaped by their background, such as their gender, culture, history, and socioeconomic origin.
Present negative or discrepant information that runs counter to the themes. Because real life is composed of different perspectives that do not always coalesce, discussing contrary information adds to the credibility of an account. A researcher can accomplish this by discussing evidence about a theme. Most evidence will build a case for the theme; researchers can also present information that contradicts the general perspective of the theme. By presenting this contradictory evidence, the account becomes more realistic and more valid.
Spend prolonged time in the field. In this way, the researcher develops an in-depth understanding of the phenomenon under study and can convey detail about the site and the people that lends credibility to the narrative account. The more experience that a researcher has with participants in their settings, the more accurate or valid will be the findings.
Use peer debriefing to enhance the accuracy of the account. This process involves locating a person (a peer debriefer) who reviews and asks questions about the qualitative study so that the account will resonate with people other than the researcher. This strategy—involving an interpretation beyond the researcher and invested in another person—adds validity to an account.
Use an external auditor to review the entire project. As distinct from a peer debriefer, this auditor is not familiar with the researcher or the project and can provide an objective assessment of the project throughout the process of research or at the conclusion of the study. The role is similar to that of a fiscal auditor, and specific questions exist that auditors might ask (Lincoln & Guba, 1985). The procedure of having an independent investigator look over many aspects of the project (e.g., accuracy of transcription, the relationship between the research questions and the data, the level of data analysis from the raw data through interpretation) enhances the overall validity of a qualitative study.
Using qualitative reliability. How do qualitative researchers check to determine if their approaches are reliable (i.e., consistent or stable)? Yin (2009) suggested that qualitative researchers need to document the procedures of their case studies and to document as many of the steps of the procedures as possible. He also recommended setting up a detailed case study protocol and database, so that others can follow the procedures. Gibbs (2007) suggested several qualitative reliability procedures:
Check transcripts to make sure that they do not contain obvious mistakes made during transcription.
Make sure that there is not a drift in the definition of codes, a shift in the meaning of the codes during the process of coding. This can be accomplished by continually comparing data with the codes and by writing memos about the codes and their definitions (see the discussion on a qualitative codebook).
For team research, coordinate the communication among the coders by regular documented meetings and by sharing the analysis.
Cross-check codes developed by different researchers by comparing results that are derived independently. Proposal writers need to include several of these procedures as evidence that they will have consistent results in their proposed study. We recommend that several procedures be mentioned in a proposal and that single researchers find another person who can cross-check their codes for what is called intercoder agreement (or cross-checking) (also see Guest et al., 2012; Creswell, 2016). Such an agreement might be based on whether two or more coders agree on codes used for the same passages in the text. It is not that they code the same passage of text; rather they determine whether another coder would code it with the same or a similar code. Reliability subprograms in qualitative computer software packages can then be used to determine the level of consistency of coding. Miles and Huberman (1994) recommended that the consistency of the coding be in agreement at least 80% of the time for good qualitative reliability.
Qualitative generalization is a term used in a limited way in qualitative research, since the intent of this form of inquiry is not to generalize findings to individuals, sites, or places outside of those under study (see Gibbs, 2007, for his cautionary note about qualitative generalizability). In fact, the value of qualitative research lies in the particular description and themes developed in the context of a specific site. Particularity rather than generalizability (Greene & Caracelli, 1997) is the hallmark of good qualitative research. However, there are a few discussions in the qualitative literature about generalizability, especially as applied to case study research in which the inquirer studies several cases. Yin (2009), for example, felt that qualitative case study results can be generalized to some broader theory. The generalization occurs when qualitative researchers study additional cases and generalize findings to the new cases. It is the same as the replication logic used in experimental research. However, to repeat a case study’s findings in a new case setting requires good documentation of qualitative procedures, such as a protocol for documenting the problem in detail and the development of a thorough case study database.