Reliability in Content Analysis

The main reliability concern in content analysis research is intercoder reliability, which is defined by The Content Analysis Guidebook as the amount of agreement or correspondence among two or more coders. Reliability is paramount in content analysis in order to establish the objectivity of the codebook, and allow the confident interpretation of results. Chapter 7 of The Content Analysis Guidebook discusses a variety of intercoder reliability coefficients and their formulas, programs that help calculate reliability coefficients, dealing with multiple coders, and the treatment of variables that do not reach acceptable levels of reliability. The purpose of this section of The Content Analysis Guidebook Online is to provide additional reliability resources to content analysis researchers and students.

Programs that Calculate Reliability Coefficients

PRAM - PRAM is a computer program that was developed to simplify the calculation of intercoder reliability coefficients for two or more coders. It may be downloaded for free for academic use. PRAM requires input data to be formatted specifically in an Excel spreadsheet and the PRAM output is available to be viewed immediately onscreen or saved as an .xls file. PRAM calculates percent agreement, Scott's pi, Cohen's kappa, Spearman rho, Pearson correlation coefficient (r), and Lin's concordance correlation coefficient (rc); an updated version calculates Fleiss' adaptation of Cohen's kappa for multiple coders and Krippendorff's alpha.

ReCal - ReCal (Reliability Calculator) is an online utility that computes intercoder/interrater reliability coefficients for nominal content analysis data. It processes multiple variables siumlaneously if there are only two coders, and only a single variable at a time if there are three or more variables. It is compatible with Excel, SPSS, Stata, OpenOffice, Google Docs, and any other database, spreadsheet, or statistical application that can export CSV files. ReCal was developed by Deen Freelon, doctoral student in the department of Communication at the University of Washington.

Links to other Reliability Resources

Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects by Matthew Lombard, Jennifer Snyder-Duch, and Cheryl Campanella Bracken.

SPSS macro for computing Krippendorff's alpha by Andrew F. Hayes.

Computing Krippendorff's alpha reliability - Resources provided by Klaus Krippendorff

The Content Analysis Guidebook Online is hosted on a Cleveland State University web server.