Reliability - Introduction to PRAM
PRAM is the Program for Reliability Assessment with
Multiple Coders. PRAM was developed by software engineers at
Skymeg Software, Inc. Dr. Kimberly Neuendorf served as a consultant
to the programmers, but neither Dr. Neuendorf nor Cleveland State
University claim any degree of ownership in the program. We are grateful
to Skymeg Software management for their generosity in sharing the
"alpha" version of the software with academic content analysis
researchers. The goal of PRAM was to provide a software that shortcut the
onerous process of calculating intercoder reliability for
(a) multiple coders, across (b) multiple variables, and providing (c) a variety of reliability coefficients.
PRAM is a Windows-based application for the PC. It was written for Windows XP, and with a basic patch will run on Windows 7; it is not yet clear whether that same patch will work for Windows 10.0. The
free academic version of the program can be downloaded below.
PRAM requires an input data file that is formatted for
Excel (with an .xls extension). The SPSS program for PC has the
capability of saving data files as .xls, so data may be exported
from SPSS into Excel and edited for use with PRAM. To structure the
data in Excel for use in Pram: (1) The first column must contain the
coder IDs in numeric form, (2) the second column must contain the unit
(case) IDs in numeric form, and (3) all other columns may contain
numerically coded variables, with variable names on the header line. All
reliability cases may be included in a single file--a main feature of
the program is its ability to organize data lines by coder IDs and unit
IDs, and to calculate reliability coefficients for all pairs of coders,
as well as for all coders taken together as appropriate.
The program provides the following reliability statistics (see Chapter 6 of The Content Analysis Guidebook (2nd ed.) for explanations and formulas):
|For Coder Pairs
||Assumed level of measurement
|Pearson correlation coefficient (r)
|Lin's Concordance (rc)
|For Multiple Coders
|Fleiss' adaptation of Cohen's kappa for multiple coders
||any (user selects level of measurement)
Graduate students at Cleveland State University, under
the supervision of Dr. Neuendorf, have validated PRAM's calculation of
the following statistics in PRAM: Percent agreement, Scott's Pi, Cohen's Kappa, Spearman Rho,
Pearson Correlation and Lin's concordance coefficient; they have also
validated the calculation of Fleiss' adaptation of Cohen's Kappa for multiple coders.
In a Windows environment, the PRAM user may select the following for each analysis conducted:
- The ID numbers of the coders to be included
- The variable names of the measures to be assessed
- The statistic to be calculated
The output of PRAM is automatically written to an Excel
(.xls) file title "PRAM" and placed in the same folder as the input
file. The statistics can be found on the second sheet of the Excel
file. Each subsequent running of PRAM on a given data set will
overwrite the output file, so it's best to resave the file with a unique
name after it is created. Although PRAM automatically averages
reliability statistics across coder pairs and across variables, use of
these averages for reporting reliability is not recommended.
In conducting analyses for all coder pairs separately,
the program provides a potential diagnostic tool for identifying coders
whose performance may be sub-standard. The program also accommodates
instances in which not all coders analyze exactly all the same units.
For each coder pair, PRAM matches all common units by unit IDs, and
calculates the coder pair reliability coefficients on the basis of that
The version of PRAM that I am empowered to share with academics is PRAM version 0.4.7, which has a few upgrades over its earlier version, but also has a small programming problem. For some unknown reason, PRAM 0.4.7 will on occasion fail to produce Scott's pi. Further, the Krippendorff's alphas have not been validated with statistical checks; as noted above, all other statistics have been checked, including the multicoder kappa.
PRAM will work readily on any PC operating system prior to Windows 7. For Windows 7, see the note below. For Windows 10, the same patch should work, but this has not yet been verified. Unfortunately, PRAM will not run on a Mac.
The file has the extension .ss. Once you have saved the file to your machine, just change the extension to .exe.
NOTE: Here is the fix for running PRAM in Windows 7: When you try to open the file in Windows 7 you will get an error that says "comdlg32.ocx is missing or invalid." This is a Microsoft program file that is not used in Windows 7 and beyond. So, download a copy of the missing file from one of the multiple free sources online, and follow the instructions for registering it.