Robin Jacob

Increasingly, educational evaluations that are rigorously assessing the impact of school-based programs rely on state assessment data to test impacts on student achievement. Researchers are relying on state assessment data more frequently because the cost of administering assessments is high, because the burden of additional testing is often a disincentive to schools and teachers to participate in research, and because No Child Left Behind has increased the importance of state assessments at all levels of the policy environment and thus as an outcome of interest in educational interventions. While, individual student-level data provide researchers with the greatest flexibility in conducting analyses, privacy concerns and resource constraints have made state officials increasingly reluctant to make these data available to researchers. However, school-level achievement data by grade and subject are widely available and easily downloadable from state department of education Web sites for most states in the U.S. The ease and low cost of access to publicly available aggregate school data, coupled with the increasing difficulty associated with obtaining restricted use student-level data files suggests that researchers are more and more likely to use school aggregate achievement and demographic variables to assess the relationships between school-level interventions and student achievement.

This paper explores the circumstances under which school-level aggregate achievement data are sufficient for addressing questions about the impact of school-level interventions on student achievement. In particular, we examine 1) whether and when publicly available school-level average achievement scores and the average of individual-level student achievement scores are identical, 2) whether the point estimates, standard errors and the proportions of variance explained differ between multi-level analyses of student-level data and single (school) level analyses of aggregate data when the main effects of interest are at the school level, 3) the types of additional analyses that can and cannot be conducted with school-level aggregate data, and 4) the metrics that are currently available in state public-use data sets and the implications these various metrics have for analyses. We conclude that under many circumstances school-level aggregate data are sufficient for addressing a variety of questions regarding the impact of school-based interventions on achievement.