Reliability in Content Analysis
The main reliability concern in content analysis research is intercoder reliability, which is defined by The Content Analysis Guidebook as the amount of agreement or correspondence among two or more coders. Reliability is paramount in content analysis in order to establish the objectivity of the codebook, and allow the confident interpretation of results. Chapter 6 of The Content Analysis Guidebook (2nd ed.) discusses a variety of intercoder reliability coefficients and their formulas, programs that help calculate reliability coefficients, dealing with multiple coders, and the treatment of variables that do not reach acceptable levels of reliability. The purpose of this section of The Content Analysis Guidebook Online is to provide additional reliability resources to content analysis researchers and students.
Programs that Calculate Reliability Coefficients
ReCal- ReCal (Reliability Calculator) is an online utility that computes a variety of intercoder/interrater reliability coefficients for content analysis data. It is compatible with Excel, SPSS, Stata, OpenOffice, Google Docs, and any other database, spreadsheet, or statistical application that can export CSV files. ReCal was developed by Deen Freelon, associate professor at the University of North Carolina at Chapel Hill. .
PRAM - PRAM is a computer program that was developed to simplify the calculation of intercoder reliability coefficients for two or more coders. It has not been updated recently, but it is freely available to those who wish to use it for academic purposes. PRAM requires input data to be formatted specifically in an Excel spreadsheet and the PRAM output is available to be viewed immediately onscreen or saved as an .xls file. PRAM calculates percent agreement, Scott's pi, Cohen's kappa, Spearman rho, Pearson correlation coefficient (r), Lin's concordance correlation coefficient (rc), Fleiss' adaptation of Cohen's kappa for multiple coders, and Krippendorff's alpha.
Links to other Reliability Resources