Andrew Ho

At some point early in our quantitative training, we learn the difference between ordinal scales and interval scales. There is a clear distinction. Ordinal scales, such as rankings, contain only information about order (1st place, 2nd place, 3rd place). Interval scales, such as time and temperature, have equal intervals between successive scale points (1 degree Celsius, 2 degrees Celsius, 3 degrees Celsius). We learn that these differences matter, because the vast majority of statistical analyses, from simple t-tests to value added models, assume that the underlying variables have interval scales.

Educational score scales are not interval scales. Analysts typically address this in one of three ways: (1) use ordinal methods that do not require interval scales; (2) concede the problem and hope that results are robust; or (3) conduct a sensitivity study that uses alternative score scales. In this presentation, I describe how this third approach is an implicit argument that the scale exists on a continuum between the ordinal and the interval. I present a framework that allows researchers to visualize and debate where a given test score scale should be located along this continuum. Finally, I provide methods that quantify the "scale dependence" of analytic results, conditional upon the location of the scale along the continuum. For discussion, I show that this approach allows us to identify which analyses and, more generally, which research questions, are more or less dependent on interval scale assumptions.