Reliability – Introduction to PRAM
PRAM is the Program for Reliability Assessment with Multiple Coders. PRAM was developed by software engineers at Skymeg Software, Inc. Dr. Kimberly Neuendorf has served as a consultant to the programmers, but neither Dr. Neuendorf nor Cleveland State University claim any degree of ownership in the program. We are grateful to Skymeg Software management for their generosity in sharing the “alpha” version of the software with academic content analysis researchers. We are unaware of other software programs that shortcut the onerous process of calculating intercoder reliability for (a) multiple coders, across (b) multiple variables, and providing (c) a variety of reliability coefficients.
PRAM is a Windows-based application for the PC. It was written for Windows XP, and with a basic patch will run on Windows 7. A free academic version of the program can be obtained on request from Dr. Kim Neuendorf at k.neuendorf@csuohio.edu.
PRAM requires an input data file that is formatted for Excel (with an .xls extension). The SPSS program for PC has the capability of saving data files as .xls, so data may be exported from SPSS into Excel and edited for use with PRAM. To structure the data in Excel for use in Pram: (1) The first column must contain the coder IDs in numeric form, (2) the second column must contain the unit (case) IDs in numeric form, and (3) all other columns may contain numerically coded variables, with variable names on the header line. All reliability cases may be included in a single file–a main feature of the program is its ability to organize data lines by coder IDs and unit IDs, and to calculate reliability coefficients for all pairs of coders, as well as for all coders taken together as appropriate.
The program provides the following reliability statistics (see Chapter 7 of The Content Analysis Guidebook (or Chapter 6 of the 2nd edition) for explanations and formulas):
| For Coder Pairs | Assumed Level of Measurement |
|---|---|
| Percent Agreement | Nominal |
| Scott’s pi | Nominal |
| Cohen’s kappa | Nominal |
| Spearman rho | Ordinal (rank-order) |
| Pearson correlation coefficient (r) | Interval/Ratio |
| Lin’s Concordance (rc) | Interval/Ratio |
| For Multiple Coders | |
| Fleiss’ adaptation of Cohen’s kappa for multiple coders | Nominal |
| Krippendorff’s alpha | Any (User selects level of measurement) |
Graduate students at Cleveland State University, under the supervision of Dr. Neuendorf, have validated PRAM’s calculation of the following statistics in PRAM: Percent agreement, Scott’s Pi, Cohen’s Kappa, Spearman Rho, Pearson Correlation and Lin’s concordance coefficient; they have also validated the calculation of Fleiss’ adaptation of Cohen’s Kappa for multiple coders.
In a Windows environment, the PRAM user may select the following for each analysis conducted:
- The ID numbers of the coders to be included
- The variable names of the measures to be assessed
- The statistic to be calculated
The output of PRAM is automatically written to an Excel (.xls) file title “PRAM” and placed in the same folder as the input file. The statistics can be found on the second sheet of the Excel file. Each subsequent running of PRAM on a given data set will overwrite the output file, so it’s best to resave the file with a unique name after it is created. Although PRAM automatically averages reliability statistics across coder pairs and across variables, use of these averages for reporting reliability is not recommended.
In conducting analyses for all coder pairs separately, the program provides a potential diagnostic tool for identifying coders whose performance may be sub-standard. The program also accommodates instances in which not all coders analyze exactly all the same units. For each coder pair, PRAM matches all common units by unit IDs, and calculates the coder pair reliability coefficients on the basis of that common set.
NOTE: Here is the fix for running PRAM in Windows 7: When you try to open the file in Windows 7 you will get an error that says “comdlg32.ocx is missing or invalid.” This is a Microsoft program file that is not used in Windows 7. So, download a copy of the missing file from one of the many free sources online. Save and register it in the same folder as the PRAM software.