Reliability (Intro to PRAM)

An Introduction to PRAM--A Program for Reliability Assessment with Multiple Coders
PRAM is under development by software engineers at Skymeg Software, Inc. Dr. Neuendorf has served as a consultant to the programmers, but neither Dr. Neuendorf nor Cleveland State University claim any degree of ownership in the program. We are grateful to Skymeg Software management for their generosity in sharing the "alpha" version of the software with academic content analysis researchers. Never before has there been a software available that shortcuts the onerous process of calculating intercoder reliability coefficients for (a) multiple coders, across (b) multiple variables. PRAM is a Windows-based application for the PC. A trial version of the program may be requested from Dr. Neuendorf at Cleveland State University (k.neuendorf@comcast.net). PRAM requires an input data file that is formatted for Excel (with an .xls extension). (A reminder that SPSS has the capability of saving data files as .xls.) The file must be structured in the following way: (1) The first column must contain the coder IDs in numeric form, (2) the second column must contain the unit (case) IDs in numeric form, and (3) all other columns may contain numerically coded variables, with variable names on the header line. All reliability cases may be included in a single file--the main feature of the program is its ability to organize data lines by coder IDs and unit IDs, and to calculate reliability coefficients for all pairs of coders, as well as for all coders taken together. The program provides the following reliability statistics (see Chapter 7 for explanations and formulas):

For coder pairs:	Assumed level of measurement:
Percent agreement	nominal
Scott’s pi	nominal
Cohen’s kappa	nominal
Spearman rho	ordinal (rank-order)
Pearson correlation coefficient (r)	interval/ratio
Lin’s concordance (r_c)	interval/ratio
For multiple coders:
Cohen’s kappa for multiple coders	nominal
Krippendorff’s alpha	any (user selects)

Graduate students at Cleveland State University, under the supervision of Dr. Kim Neuendorf, have validated PRAM's calculations for the following statistics: Percent agreement, Scott's Pi, Cohen's Kappa, Spearman Rho, Pearson Correlation, and Lin's Concordance. The calculations for Krippendorff's alpha have not been validated. The PRAM user may select the following for each analysis conducted: The ID numbers of the coders to be included The variable names of the measures to be assessed The statistic(s) to be calculated The output of PRAM may be displayed on-screen, and will also be saved automatically as an Excel file named PRAM.XLS in the directory where the data file is located (thus, it is recommended that you change the name of your PRAM output file soon after it is created, so that the next output file may be created in its place). If a "coder pair" coefficient is selected, the output presents (1) the selected reliability coefficient for each and every variable specified for each pair of coders separately, and (2) the average of the selected reliability coefficient across all pairs of coders. If a "multiple coders" coefficient is selected, the output presents the overall statistic for each and every variable specified. It should be noted that while PRAM produces averages for coefficients across coder pairs and across variables, these averages are not recommended for reportage. In conducting analyses for all coder pairs separately, the program provides a type of diagnostic tool for identifying coders whose performance may be sub-standard. The program also accommodates instances in which not all coders analyze exactly all the same units. For each coder pair, PRAM matches all common units by unit IDs, and calculates the coder pair reliability coefficients on the basis of that common set. However, PRAM currently cannot accommodate missing data (i.e., empty cells in the data file).