Learning About and From Variation in Treatment Effects

Sponsored by the Institute of Education Sciences
and the Stanford Center for Education Policy Analysis
July 18-21, 2016
CERAS 101, Stanford University
Day 1: Studying Cross-site Distributions of ITT Effects
Instructors: Howard Bloom, Michael Weiss, Steve Raudenbush

Time Topic
8:30 – 9:00 Coffee and Breakfast Available
9:00 – 9:30 Introductions and Workshop Overview (Sean Reardon)
Framing the workshop. Heterogeneity, mediation, moderation (Sean Reardon)
9:30 – 10:30

Estimating Parameters of a Cross-site Distribution of ITT Effects (Howard Bloom). Download presentation

10:30 – 10:45 Break
10:45 – 11:45 Statistical Precision/Power for Estimating Parameters of a Cross-site Distribution of ITT Effects (Howard Bloom). Download presentation
11:45 – 12:30 Small Group Activity (Howard Bloom)
12:30 – 1:45 Lunch
1:45 – 2:45

Empirical Estimates of Parameters of Cross-site Distributions of ITT Effects and a Nascent Theory of Cross-site Impact Variation (Michael Weiss). Download presentation

2:45 – 3:00 Break
3:00 – 4:15

Alternative Estimands and Estimators for Parameters of a Cross-site Distribution of ITT Effects (Stephen Raudenbush). Download presentation

4:15 – 5:00 Small group Activity (Michael Weiss & Stephen Raudenbush)

Background Readings for Day 1:

  • Bloom, H.S., S.W. Raudenbush, M. J. Weiss and K. Porter (conditional acceptance) “Using Multisite Experiments to Study Cross-site Variation in Effects of Program Assignment,” Journal of Research on Educational Effectiveness. Download
  • Bloom, H.S. and J. Spybrook (under review) “Assessing the Precision of Multisite Trials for Estimating Parameters of Cross-site Distributions of Program Effects.” Download
  • Weiss, M.J., H.S. Bloom, N. Verbitsky Savitz, H. Gupta, A. Vigil and D. Cullinan (under review) “How Much Do the Effects of Education and Training Programs Vary Across Sites? Evidence from Existing Multisite Randomized Control Trials.” Download
  • Raudenbush, S.W. and H.S. Bloom (2015) “Learning About and From a Distribution of Program Impacts Using Multisite Trials,” American Journal of Evaluation. Download
  • Weiss M.J, H.S. Bloom and T. Brock (2014) “A Conceptual Framework for Studying the Sources of Variation in Program Effects,” Journal of Policy Analysis and Management. Download
  • Raudenbush, S.W. and D. Schwartz (in progress) “Estimation of Means and Covariance Components in Multi-site Randomized Trials.” Download

Please read bolded readings prior to workshop

Day 2: Studying Mediation of Treatment Effects Using Instrumental Variables Approaches
Instructors: Steve Raudenbush, Sean Reardon

Time Topic
8:30 – 9:00 Coffee and Breakfast Available
9:00 – 9:15 A Framework for Studying Mediation (Sean Reardon)
9:15 – 11:15

Complier Average Effects in Multisite Trials (Stephen Raudenbush). Download presentation

11:15 – 12:00 Small Group Activity (Stephen Raudenbush)
12:00 – 1:30 Lunch
1:30 – 2:30

The MSMM-IV Model: Conceptual Model and Assumptions (Sean Reardon). Download presentation

2:30 – 2:45 Break
2:45 – 4:00

Using the MSMM-IV Model when the Exclusion Restriction is Invalid (Sean Reardon). Download presentation

4:00 – 4:45 Small group Activity (Sean Reardon and Lindsay Fox)

Background Readings for Day 2:

  • Reardon, S.F., and Raudenbush, S.W. (2013). “Under What Assumptions do Site-by-Treatment Instruments Identify Average Causal Effects?” Sociological Methods and Research 42(2): 143-163. Download
  • Reardon, S.F., Unlu, F., Zhu, P., & Bloom, H. (2016). “Using the MSMM-IV Model to Estimate Mediator Effects When the Exclusion Restriction is Invalid.” Working Paper.
  • Reardon, S.F., Unlu, F., Zhu, P., & Bloom, H. (2014). “Bias and Bias Correction in Multi-Site Instrumental Variables Analysis of Heterogeneous Mediator Effects.” Journal of Educational and Behavioral Statistics 39(1): 53-86. Download
  • Raudenbush, S.W., Reardon, S.F., & Nomi, T. (2012). “Statistical Analysis for Multi-site Trials Using Instrumental Variables with Random Coefficients” Journal of Research on Educational Effectiveness 5: 303-332. Download
  • Duncan, G. J., Morris, P., and Rodrigues, C.(2011). Does money really matter? Estimating impacts of family income on young children’s achievement with data from random-assignment Experiments. Developmental Psychology, 47(5): 1263-1279. Download
  • Kling, J. R., Liebman, J. B., & Katz, L. F. (2007). Experimental analysis of neighborhood effects. Econometrica, 75(1), 83-119. Download

Please read bolded readings prior to workshop

Day 3: Principal Stratification
Instructor: Lindsay Page

Time Topic
8:30 – 9:00 Coffee and Breakfast Available
9:00 – 10:15

Conceptual framework and key assumptions (Lindsay Page)

10:15 – 10:30 Break
10:30 – 12:00

Examples: principal stratification set up and defining estimands of interest (Lindsay Page)

  • Small group exercise
  • Discussion
12:00 – 1:30 Lunch
1:30 – 3:30

Bounds Application and Guided Exercise (Lindsay Page)

3:30 – 3:45 Break
3:45 – 4:30 Wrap Up (Nod to Principal Scores / Model-Based Estimation) (Lindsay Page)

Workshop materials:
Principal stratification workshop.pdf
Principal stratification workshop - Covariates to sharpen bounds
Principal stratification small group exercise 1
Stanford workshop - bounds calculation.do

Background Readings for Day 3:

  • Page, L. C., Feller, A., Grindal, T., Miratrix, L. & Somers, M-A. (2015). Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation, 36(4), 514-531. Download
  • Long, D. M., & Hudgens, M. G. (2013). Sharpening bounds on principal effects with covariates. Biometrics, 69(4), 812-819. Download
  • Feller, A., Grindal, T., Miratrix, L. & Page, L. C. (forthcoming). Compared to what? Variation in the impacts of Head Start by alternative child care setting. Annals of Applied Statistics. (Recommended). Download
  • Feller, A., Mealli, F., & Miratrix, L. (2016). Principal Score Methods: Assumptions and Extensions. arXiv preprint arXiv:1606.02682. (Recommended). Download
  • Feller, A., Greif, E., Miratrix, L., & Pillai, N. (2016). Principal stratification in the Twilight Zone: Weakly separated components in finite mixture models. arXiv preprint arXiv:1602.06595. (Recommended). Download

Please read bolded readings prior to workshop

Day 4: Weighting Methods for Causal Mediation Analysis
Instructors: Guanglei Hong, Xu Qin, Lindsay Page, Sean Reardon

Time Topic
8:45 – 9:00 Coffee and Breakfast Available
9:00 – 10:30

Concepts of causal mediation (Guanglei Hong)
Brief review of existing methods for causal mediation analysis (Guanglei Hong)

10:30 – 10:45 Break
10:45 – 12:00

Rationale of the RMPW strategy (Guanglei Hong)

12:00 – 1:00 Lunch
1:00 – 2:30

Parametric and nonparametric analytic procedures and simulation results
(Guanglei Hong)
Hands-on exercise with the RMPW software (Guanglei Hong, Xu Qin)
RMPW extensions: Stata, SAS, and R code (Guanglei Hong, Xu Qin)

2:30 – 2:45 Break
2:45 – 3:30

Multisite causal mediation analysis (Xu Qin)

3:30 – 4:30 Q & A wrt Causal Meditation Analysis and Its Distinctions from and Connection with Other Methods (Guanglei Hong, Xu Qin, Sean Reardon)

Workshop materials:
Stanford Workshop_RMPW slides_GH.pdf
Stanford 2016 Summer Workshop_Xu Qin_rv.pdf
RMPW program manual
NEWWS_Riverside_class use_rv.dta
Riverside.do
Riverside.dta
riverside.sas7bdat
Riverside_R.R
Riverside_SAS.sas
Identification Assumptions

Background Readings for Day 4:

  • Hong, G., Deutsch, J., & Hill, H. D. (2015). Ratio-of-mediator-probability weighting for causal mediation analysis in the presence of treatment-by-mediator interaction. Journal of Educational and Behavioral Statistics, 40(3), 307-340. Download
  • Hong, G., & Nomi, T. (2012). Weighting methods for assessing policy effects mediated by peer change. Journal of Research on Educational Effectiveness special issue on the statistical approaches to studying mediator effects in education research, 5(3), 261-289. Download
  • Hong, G. (2015). Causality in a social world: Moderation, mediation, and spill-over. West Sussex, UK: John Wiley & Sons.
  • Bein, E., Deutsch, J., Porter, K., Qin, X., Yang, C., & Hong, G. (2015). Technical report on two-step estimation in RMPW analysis. MDRC. Download
  • Huber, M. (2014). Identifying causal mechanisms (primarily) based on inverse probability weighting. Journal of Applied Econometrics, 29(6), 920-943. Download
  • Lange, T., Rasmussen, M., & Thygesen, L. (2014). Assessing natural direct and indirect effects through multiple pathways. American journal of epidemiology, 179 (4), 513-518. Download
  • Lange, T., Vansteelandt, S., & Bekaert, M. (2012). A simple unified approach for estimating natural direct and indirect effects. American journal of epidemiology, 176 (3), 190–195. Download
  • Tchetgen Tchetgen, E. J., & Shpitser, I. (2012). Semiparametric theory for causal mediation analysis: Efficiency bounds, multiple robustness and sensitivity analysis. The Annals of Statistics, 40(3), 1816-1845. Download
  • Hong, G., Deutsch, J., & Hill, H. (2011). Parametric and non-parametric weighting methods for estimating mediation effects: An application to the National Evaluation of Welfare-to-Work Strategies. In JSM Proceedings, Social Statistics Section. Alexandria, VA: American Statistical Association, pp.3215-3229. Download
  • Hong, G. (2010). Ratio of mediator probability weighting for estimating natural direct and indirect effects. In JSM Proceedings, Biometrics Section. Alexandria, VA: American Statistical Association, pp.2401-2415. Download

Please read bolded readings prior to workshop