WHEN TWO PEOPLE DO THE SAME THING, IT’S NOT THE SAME THING: CONSISTENCIES AND DIFFERENCES IN CONTENT ANALYSIS OF THERAPY DIARIES

Abstract

One of the frequent objections to qualitative approaches and the use of qualitative methods for research is the subjectivity of the evaluation – it is entirely up to the analyst to choose which sections to analyse, what codes to use, what themes to generate as significant and pass them on for further analysis or interpretation. This study, which is part of the broader DAQUAo project (psycholinguistic analysis of therapeutic diaries), is concerned with content analysis as a fundamental and usually the first step in most qualitative approaches. The aim of this study is to verify the agreement of assessors by statistically comparing codes obtained by the content analysis method in the open coding phase, performed by two independent analysts onidentical texts. Stimulus material: therapy journals, N=179 unique text entries, total 22,046 words. Results: Analytical K identified 32 codes, analyst L identified 70 codes. Codes that were close in content were referred to assessor agreement analysis (paired t-test). There was an agreement for two codes (movement-travel, emotion-good mood). Most of the differences were due to different levels of generality – there were mostly hyper- and hyponymy relationships between the analyst’s codes K and L (the superordinate category K corresponded to several specific categories of L) or adjacent categories (two specific instances of one superordinate category). Approximately one-third of the codes were unique and could not be easily compared. Conclusion: coding text through content analysis requires careful training, a clear definition of the level of generality required in the first stage of coding, and last but not least, a formulated research question. These three conditions can ensure higher inter-assessor agreement and de facto better reliability of the method as such.