Reliability - Introduction to PRAM
PRAM is the Program for Reliability Assessment with
Multiple Coders. PRAM was developed by software engineers at
Skymeg Software, Inc. Dr. Kimberly Neuendorf has served as a consultant
to the programmers, but neither Dr. Neuendorf nor Cleveland State
University claim any degree of ownership in the program. We are grateful
to Skymeg Software management for their generosity in sharing the
"alpha" version of the software with academic content analysis
researchers. We are unaware of other softwares that shortcut the
onerous process of calculating intercoder reliability for
(a) multiple coders, across (b) multiple variables, and providing (c) a variety of reliability coefficients.
PRAM is a Windows-based application for the PC. It was written for Windows XP, and with a basic patch will run on Windows 7. A
free academic version of the program can be obtained on request from Dr. Kim Neuendorf at firstname.lastname@example.org.
PRAM requires an input data file that is formatted for
Excel (with an .xls extension). The SPSS program for PC has the
capability of saving data files as .xls, so data may be exported
from SPSS into Excel and edited for use with PRAM. To structure the
data in Excel for use in Pram: (1) The first column must contain the
coder IDs in numeric form, (2) the second column must contain the unit
(case) IDs in numeric form, and (3) all other columns may contain
numerically coded variables, with variable names on the header line. All
reliability cases may be included in a single file--a main feature of
the program is its ability to organize data lines by coder IDs and unit
IDs, and to calculate reliability coefficients for all pairs of coders,
as well as for all coders taken together as appropriate.
The program provides the following reliability statistics (see Chapter 7 of The Content Analysis Guidebook (or Chapter 6 of the 2nd edition) for explanations and formulas):
|For Coder Pairs
||Assumed level of measurement
|Pearson correlation coefficient (r)
|Lin's Concordance (rc)
|For Multiple Coders
|Fleiss' adaptation of Cohen's kappa for multiple coders
||any (user selects level of measurement)
Graduate students at Cleveland State University, under
the supervision of Dr. Neuendorf, have validated PRAM's calculation of
the following statistics in PRAM: Percent agreement, Scott's Pi, Cohen's Kappa, Spearman Rho,
Pearson Correlation and Lin's concordance coefficient; they have also
validated the calculation of Fleiss' adaptation of Cohen's Kappa for multiple coders.
In a Windows environment, the PRAM user may select the following for each analysis conducted:
- The ID numbers of the coders to be included
- The variable names of the measures to be assessed
- The statistic to be calculated
The output of PRAM is automatically written to an Excel
(.xls) file title "PRAM" and placed in the same folder as the input
file. The statistics can be found on the second sheet of the Excel
file. Each subsequent running of PRAM on a given data set will
overwrite the output file, so it's best to resave the file with a unique
name after it is created. Although PRAM automatically averages
reliability statistics across coder pairs and across variables, use of
these averages for reporting reliability is not recommended.
In conducting analyses for all coder pairs separately,
the program provides a potential diagnostic tool for identifying coders
whose performance may be sub-standard. The program also accommodates
instances in which not all coders analyze exactly all the same units.
For each coder pair, PRAM matches all common units by unit IDs, and
calculates the coder pair reliability coefficients on the basis of that
NOTE: Here is the fix for running PRAM in Windows 7: When you try to open the file in Windows 7 you will get an error that says "comdlg32.ocx is missing or invalid." This is a Microsoft program file that is not used in Windows 7. So, download a copy of the missing file from one of the many free sources online. Save and register it in the same folder as the PRAM software.