To examine the effects of an early literacy formative assessment program on teacher attitudes and practices, student outcomes, and implementation of the program, CEPA researchers gathered data from multiple sources. Teacher attitudes were assessed with a survey focused on their perspectives, experiences, and professional background. Student outcomes were measured with results from summative assessments of students’ early literacy skills and executive functioning, as well as district administrative data. CEPA researchers analyzed the implementation of the intervention through observations of teacher meetings, school leader interviews, and formative assessment data.
Teacher Survey
The research team gave kindergarten and first grade teachers a survey in the spring of 2012. The survey asked teachers questions about their work background, instructional practices, attitudes toward data, and their views on teaching in the district. The 193 teachers in classrooms that received the summative assessment were offered a gift card in exchange for taking the survey, resulting in a high response rate. The survey was both voluntary and confidential.
2011-2012 SFUSD Teacher Survey (Spring)
Summative Assessment
To study the effects of the early literacy formative assessment in kindergarten and first grade, a summative assessment was administered to students in randomly-selected classrooms within 25 treatment and 25 control school in 2012. The summative assessment consisted of two parts:
- The Stanford Achievement Test, 10th Edition (Stanford-10): 100 kinder and 100 first grade classes were randomly selected to be administered the Stanford 10 assessment in Spanish or English. In kindergarten, the Sound and Letters and the Word Reading subscales were administered. In first grade, the Reading Comprehension and Sentence Reading subscales were used.
- Head-Toes-Knees-Shoulders (HTKS) assessment: 50 kindergarten and 50 first grade classes were administered the HTKS assessment. The HTKS assesses self-regulation through a structured observation that requires children to perform the opposite response to four different oral commands. For example, for the assessment, children are instructed to touch their toes each time they are told to touch their head.
Early literacy formative assessment
At the heart of the intervention was a screening, diagnostic and progress monitoring system designed to measure fundamental components of literacy. It was designed to help identify students in need of additional reading instruction beyond that provided to typically developing readers. System tools include student assessments and an online reporting system. Teachers administered assessments three times during the year in approximately half of the districts’ kindergarten and first grade classrooms, and two times during the school year in close to half of all pre-k classrooms.
pals (Phonological Awareness Literacy Screening)
Grade-Level Data Meeting Observations
As part of the intervention, teachers participated in grade-level meetings focused on discussing and analyzing student formative assessment data, and identifying ways to use assessment results to improve student outcomes. In these meetings, a literacy facilitator led teachers in discussions about the assessment results. CEPA researchers observed these grade-level “data meetings” using an observation protocol. Data meetings were observed in the fall and spring of the 2011-2012 school year. The observation protocol consisted of four parts:
- Questions for the researcher to answer at the beginning of the data meeting.
- An activity log used to record the specific behaviors that took place during the meeting.
- A check list used to tally the number of times specific behaviors took place.
- Questions for the researcher to answer at the end of the meeting.
2011-2012 Interview Protocol (Spring)
2011-2012 Interview Protocol (Fall)
School Leader Interviews
CEPA conducted interviews with school principals and literacy facilitators who worked in the schools that implemented the formative literacy assessment. Interviews took place with principals in the fall and spring, and with facilitators in the fall only. School leaders were asked about the use of literacy instruction and assessments at their school, and specific questions about implementing the assessment at the school site.
2011-2012 Observation Protocol
Administrative Data
The district shared its 2011-12 administrative data on kindergarten, first grade, and pre-kindergarten students, and on staff. Data elements include demographic and assessment information for students, and information on credential(s), education, and work experience for staff.