Reliability - Introduction to PRAM

PRAM is the Program for Reliability Assessment with Multiple Coders. PRAM was developed by software engineers at Skymeg Software, Inc. Dr. Kimberly Neuendorf served as a consultant to the programmers, but neither Dr. Neuendorf nor Cleveland State University claim any degree of ownership in the program. We are grateful to Skymeg Software management for their generosity in sharing the "alpha" version of the software with academic content analysis researchers. The goal of PRAM was to provide a software that shortcut the onerous process of calculating intercoder reliability for (a) multiple coders, across (b) multiple variables, and providing (c) a variety of reliability coefficients.

PRAM is a Windows-based application for the PC. It was written for Windows XP, and with a basic patch will run on Windows 7; it is not yet clear whether that same patch will work for Windows 10.0. The free academic version of the program can be downloaded below.

PRAM requires an input data file that is formatted for Excel (with an .xls extension). The SPSS program for PC has the capability of saving data files as .xls, so data may be exported from SPSS into Excel and edited for use with PRAM. To structure the data in Excel for use in Pram: (1) The first column must contain the coder IDs in numeric form, (2) the second column must contain the unit (case) IDs in numeric form, and (3) all other columns may contain numerically coded variables, with variable names on the header line. All reliability cases may be included in a single file--a main feature of the program is its ability to organize data lines by coder IDs and unit IDs, and to calculate reliability coefficients for all pairs of coders, as well as for all coders taken together as appropriate.

The program provides the following reliability statistics (see Chapter 6 of The Content Analysis Guidebook (2nd ed.) for explanations and formulas):

For Coder Pairs Assumed level of measurement
Percent Agreement nominal
Scott's pi nominal
Cohen's kappa nominal
Spearman rho ordinal (rank-order)
Pearson correlation coefficient (r) interval/ratio
Lin's Concordance (rc) interval/ratio
For Multiple Coders
Fleiss' adaptation of Cohen's kappa for multiple coders nominal
Krippendorff's alpha any (user selects level of measurement)

Graduate students at Cleveland State University, under the supervision of Dr. Neuendorf, have validated PRAM's calculation of the following statistics in PRAM: Percent agreement, Scott's Pi, Cohen's Kappa, Spearman Rho, Pearson Correlation and Lin's concordance coefficient; they have also validated the calculation of Fleiss' adaptation of Cohen's Kappa for multiple coders.

In a Windows environment, the PRAM user may select the following for each analysis conducted:

  • The ID numbers of the coders to be included
  • The variable names of the measures to be assessed
  • The statistic to be calculated

The output of PRAM is automatically written to an Excel (.xls) file title "PRAM" and placed in the same folder as the input file. The statistics can be found on the second sheet of the Excel file. Each subsequent running of PRAM on a given data set will overwrite the output file, so it's best to resave the file with a unique name after it is created. Although PRAM automatically averages reliability statistics across coder pairs and across variables, use of these averages for reporting reliability is not recommended.

In conducting analyses for all coder pairs separately, the program provides a potential diagnostic tool for identifying coders whose performance may be sub-standard. The program also accommodates instances in which not all coders analyze exactly all the same units. For each coder pair, PRAM matches all common units by unit IDs, and calculates the coder pair reliability coefficients on the basis of that common set.

Downloading PRAM

The version of PRAM that I am empowered to share with academics is PRAM version 0.4.7, which has a few upgrades over its earlier version, but also has a small programming problem. For some unknown reason, PRAM 0.4.7 will on occasion fail to produce Scott's pi. Further, the Krippendorff's alphas have not been validated with statistical checks; as noted above, all other statistics have been checked, including the multicoder kappa.

PRAM will work readily on any PC operating system prior to Windows 7. For Windows 7, see the note below. For Windows 10, the same patch should work, but this has not yet been verified. Unfortunately, PRAM will not run on a Mac.

The file has the extension .ss. Once you have saved the file to your machine, just change the extension to .exe.

To download:

PRAM 0.4.7

NOTE: Here is the fix for running PRAM in Windows 7: When you try to open the file in Windows 7 you will get an error that says "comdlg32.ocx is missing or invalid." This is a Microsoft program file that is not used in Windows 7 and beyond. So, download a copy of the missing file from one of the multiple free sources online, and follow the instructions for registering it.




The Content Analysis Guidebook Online is hosted on a Cleveland State University web server.