education northwest REATIN SCHOOLS COMM E5 Neuhaus Education Center: Teachers Make the Difference 2013?201 4 Evaluation Report October 2014 Neuhaus Education Center: Teachers Make the Difference 2013–2014 Evaluation Report Steve Underwood, Ed.D. Angela Roccograndi, M.S.W. Monica Cox, M. P. P. October 2014 About Education Northwest Founded as a nonprofit corporation in 1966, Education Northwest builds capacity in schools, families, and communities through applied research and development. Education Northwest serves as an independent evaluator for Neuhaus Education Center’s prekindergarten literacy project, Teachers Make the Difference (TMTD). TMTD provides indepth professional learning and job-embedded coaching to early childhood teachers in Houston, Texas, in order to improve kindergarten reading readiness skills for all students. Contact Education Northwest 101 SW Main Street, Suite 500 Portland, OR 97204 educationnorthwest.org 503.275.9500 Authors Steve Underwood, Ed.D. Angela Roccograndi, M.S.W. Monica Cox, M.P.P Suggested Citation Underwood, S., Roccograndi, A., & Cox, M. (2014). Neuhaus Education Center Teachers Make the Difference: 2013–2014 Evaluation Report. Portland, OR: Education Northwest. Executive Summary Does evidence-based professional development and coaching improve the knowledge and classroom practice of early childhood educators, and subsequently improve student outcomes? That is what the Neuhaus Education Center (Neuhaus) sought to answer in the evaluation of the Teachers Make the Difference (TMTD) program for prekindergarten (PreK) teachers working with disadvantaged students in traditionally lower performing schools in the Houston Independent School District (HISD). Neuhaus engaged 68 PreK teachers (TMTD teachers) in a yearlong professional development program that sought to improve educator’s skills in teaching four areas of early literacy—oral language, phonological awareness, letter recognition, and concepts of print. The professional development consisted of six workshops and coaching delivered throughout the 2013–2014 school year. Neuhaus used one coaching model, but delivered coaching with different frequency to two groups of workshop participants—teachers in the High Intensity coaching group received weekly coaching, while those in the Low Intensity coaching group received monthly coaching. For comparison purposes, a third group of teachers who did not participate in the workshops or coaching were included in the evaluation. Did TMTD teachers receive the evidence-based program? All TMTD teachers participated in the TMTD professional development workshops. Both High and Low Intensity teachers participated in an average of six workshops. Overall, both the High and Low Intensity teachers were pleased with the workshops and felt the time spent on a variety of early childhood literacy topics was “just right.” All TMTD teachers received coaching. High Intensity teachers received more coaching than Low Intensity teachers (an average of 1.8 and 0.7 sessions per month, respectively), but less coaching than anticipated (59% of the expected sessions). Coaches most commonly observed and demonstrated, and did so similarly across the High and Low Intensity coaching groups. Teachers most commonly received coaching in teaching oral language, letter recognition, and phonological awareness, and assessing student understanding. Other topics covered by coaching included concepts of print, emergent writing, grapheme-phoneme correspondence, and teacher use of questioning strategies. Coaches addressed all topics with teachers in the two coaching groups with equal frequency, with the exception of oral language. Coaches addressed oral language with greater frequency with Low Intensity teachers than with High Intensity teachers. Neuhaus Education Center: Teachers Make the Difference i Did participating in workshops and coaching make a difference in teacher knowledge and classroom practice? The knowledge of TMTD teachers, as measured by the Teacher Knowledge Survey, increased significantly from fall to spring (18 items answered correctly in fall versus 21 items answered correctly in spring). TMTD teachers used the materials weekly, and usually with their most struggling students (Tier 3), during small- and whole-group instruction. Teachers were slightly more likely to use the lessons than the extension activities, and least likely to use the book recommendations. There were no differences in TMTD material use between High and Low Intensity teachers. TMTD teachers reported that participating in coaching activities impacted their instruction, with the greatest impact from demonstration lessons and side-by-side and shadow coaching. TMTD teachers were significantly more likely than comparison teachers to report having more skills to implement instructional practices to improve Tier 1 students’ phonological awareness skills; however, TMTD and comparison teachers’ reported similar skill levels for using instructional practices for building skills in oral language, letter recognition, and concepts of print. Furthermore, there was no difference between TMTD and comparison teachers in their reported skill level for using instructional practices to build Tier 2/3 students’ skills related to phonological awareness, oral language, letter recognition, and concepts of print. TMTD teachers, however, reported significantly lower skills when using instructional practices with their Tier 2/3 students than with Tier 1 students. The reported skill use for instructional practices with Tier 1 and with Tier 2/3 students by the comparison teachers was not significantly different. Almost all teachers believed that parents are capable of improving the early language and literacy skills of their children. Teachers reported having repertoires of strategies to share with parents and spending time communicating those strategies. Most teachers communicated with parents at least once a month. There were no differences in these reports between TMTD and comparison teachers. ii Education Northwest Did students of teachers who received TMTD achieve at higher levels compared to students whose teachers did not receive TMTD? Receptive Letter Vocabulary Sounds Lower Case Letters Upper Case Letters Rhyming Picture Naming Print Concepts Figure 1 Student Outcomes Fall to Spring, by Subtest and Teacher Group TMD Comparison 3.1 2.6 2.1 3.7 2.1 TMD 18.5 Comparison TMD Comparison 17.4 3.1 2.6 4.1 1.3 11.1 Comparison 11.7 9.9 9.9 7.1 6.4 Comparison 6.8 3.1 11.7 3.6 TMD 3.4 9.2 6.7 Comparison Fall 2013 9.2 14.2 TMD Comparison 3.4 2.6 TMD TMD 3.4 5.0 9.9 1.1 0.5 1.1 0.0 5.0 10.0 Comparison Student Growth, Spring 2014 15.0 20.0 25.0 Additional TMD Growth, Spring 2014 In fall 2013 (blue bars), the average ELQA scores of students of TMTD teachers were significantly lower than those of students of comparison teachers on every subtest, with the exception of Picture Naming (Figure 1). By spring 2014 (tan and green bars), this pattern was reversed and average scores of students of TMTD teachers were higher than those of students of comparison teachers on every subtest, with the exception of Lowercase Letters. From fall to spring, the average rate of change in ELQA scores among TMTD teacher’s students was significantly higher than that of comparison teachers’ students on all subtests except Picture Naming. Did implementation of a differentiated coaching model make a difference in student outcomes? Students of High and Low Intensity teachers achieved at similar levels. However, students of High Intensity teachers had statistically lower rates of change than expected in two areas—Rhyming and Letter Sounds—compared to students of Low Intensity teachers. Neuhaus Education Center: Teachers Make the Difference iii iv Education Northwest Contents Introduction ................................................................................................................................................ 1 Background ............................................................................................................................................. 1 Study Design ........................................................................................................................................... 2 Data Collection........................................................................................................................................ 3 Participant Demographics ..................................................................................................................... 4 Teachers Make the Difference Intervention ........................................................................................... 9 Workshops............................................................................................................................................... 9 Coaching ................................................................................................................................................ 11 Teachers’ Outcomes ................................................................................................................................. 19 Increased Early Literacy Knowledge ................................................................................................. 19 Use of TMTD Materials ....................................................................................................................... 20 Changes in Practice .............................................................................................................................. 21 Student Outcomes .................................................................................................................................... 27 Differences Based on Teacher Participation in TMTD .................................................................... 27 Differences Based on TMTD Coaching Intensity ............................................................................. 39 Discussion and Recommendations ........................................................................................................ 41 Discussion .............................................................................................................................................. 41 Recommendations: ............................................................................................................................... 43 Technical Appendix ................................................................................................................................. 45 School and Teacher Assignment ........................................................................................................ 45 Student Outcomes ................................................................................................................................ 47 Difference-In-Difference Regression Tables ..................................................................................... 49 Appendix A: Teachers Make the Difference Spring 2014 Survey .................................................... 53 Appendix B: Neuhaus Education Center Teachers Make the Difference 2013–2014 Coaching Log ............................................................................................................................... 67 References ................................................................................................................................................. 69 Neuhaus Education Center: Teachers Make the Difference v List of Figures Figure 1 Student Outcomes Fall to Spring, by Subtest and Teacher Group ...................................iii Figure 2 Frequency With Which Coaches Used Coaching Techniques ......................................... 13 Figure 3 Frequency With Which Coaches Addressed Early Literacy Topics ................................ 15 Figure 4 Percent of Items Answered Correctly in Fall 2013 ............................................................. 28 Figure 5 Percent of Items Answered Correctly in Spring 2014........................................................ 29 Figure 6 Comparison of Average ELQA Scores in Percent Correct................................................ 30 Figure 7 Change in ELQA Print Concepts Scores ............................................................................. 31 Figure 8 Change in ELQA Picture Naming Scores............................................................................ 32 Figure 9 Change in ELQA Rhyming Scores ....................................................................................... 33 Figure 10 Change in ELQA Uppercase Letters Scores ...................................................................... 34 Figure 11 Change in ELQA Lowercase Letters Scores ...................................................................... 35 Figure 12 Change in ELQA Letter Sounds Scores ............................................................................. 36 Figure 13 Change in ELQA Receptive Vocabulary Scores ............................................................... 37 vi Education Northwest List of Tables Table 1 ELQA Subtests and Number of Items ..................................................................................... 4 Table 2 Teaching Experience, by Group ............................................................................................... 5 Table 3 Student Demographics, by Group ........................................................................................... 6 Table 4 Fall ELQA Data, by Group ........................................................................................................ 7 Table 5 Anticipated Weeks of TMTD Coaching, by Month and Group......................................... 11 Table 6 Average Number of Coaching Sessions, by Month and Group ........................................ 12 Table 7 Frequency With Which Coaches Addressed Oral Language Topics, by Group ............. 16 Table 8 Frequency With Which Coaches Addressed Phonological Awareness Topics, by Group .................................................................................................................................................. 16 Table 9 Teacher Knowledge Survey, Items Answered Correctly, Fall and Spring, by Group .................................................................................................................................................. 19 Table 10 Frequency and Duration With Which Teachers Used TMTD Materials ........................ 20 Table 11 Teachers’ Preferred Use of TMTD Materials, by Rank ..................................................... 20 Table 12 Average Teacher Use of TMTD Units, by Section ............................................................. 21 Table 13 Percentage of Teachers Participating in Coaching Activities........................................... 22 Table 14 Reported Teacher Skill Levels in Use of Strategies to Build Early Literacy Skills, by Group ................................................................................................................................. 23 Table 15 Use of Instructional Practices to Build Phonological Awareness Skills With Tier I Students .................................................................................................................................... 24 Table 16 Reported Oral Language and Early Literacy Supports to Parents, by Group ............... 25 Table 17 Teachers’ Perceptions of Their Work With Parents to Support Oral Language and Early Literacy, by Group .......................................................................................................... 26 Table 18 Frequency With Which Teachers Communicate With Parents, by Group .................... 26 Table 19 Effect Sizes (ES) for Early Literacy Interventions .............................................................. 38 Table 20 Differences in Rate of Change in Scores of Students of High Intensity and Low Intensity Teachers .............................................................................................................................. 39 Table TA-1 Coaching Conditions for TMTD Schools ....................................................................... 45 Table TA-2 Study Attrition at School and Teacher Levels ............................................................... 46 Table TA-3 Average Fall 2013 Student ELQA Scores ....................................................................... 47 Table TA-4 Average Spring 2014 Student ELQA Scores .................................................................. 48 Neuhaus Education Center: Teachers Make the Difference vii Table TA-5 Differences between Students of TMTD and Comparison Teachers in the Average Student ELQA Scores ........................................................................................................ 49 Table TA-6 Differences between Students of TMTD and Comparison Teachers in the Average Rate of Change in Student ELQA Scores, Controlling for Student and Teacher Characteristics ..................................................................................................................... 50 Table TA-7 Differences between High and Low Intensity Teachers in the Average Student ELQA Scores ........................................................................................................................ 51 Table TA-8 Differences between High and Low Intensity Teachers in the Average Student ELQA Scores, Controlling for Student and Teacher Characteristics ........................... 52 viii Education Northwest Acknowledgements The authors would like to thank Dr. Regina Gooden and Vanessa Grant from Neuhaus Education Center for the significant time and energy they gave in making this evaluation possible. In addition, they invested great time and energy into the teachers’ training and they worked diligently with us to plan and coordinate the collection of many sources of data that provide insight into the various aspects of the project. We would also like to thank the coaches for taking the time to document coaching experiences, and the teachers for completing the surveys. Finally thanks are extended to staff members at the Houston Independent School District and the ELQA staff at the University of Oklahoma for assisting us in acquiring the necessary student data needed for this report. Neuhaus Education Center: Teachers Make the Difference ix Introduction Background Neuhaus Education Center’s (Neuhaus) Teachers Make the Difference (TMTD) project was developed to improve the early literacy outcomes of prekindergarten (PreK) students in the Houston Independent School District (HISD), a population of children who are at risk of future failure in reading. The project aims to develop a common language among PreK educators around literacy instruction and to improve instructional practices to deepen children’s oral language and strengthen early literacy skills. It attempts to do so by impacting the intermediary variables that lead to student learning—specifically teacher knowledge and instructional practices. Neuhaus articulated a theory of action wherein student literacy outcomes would improve if teachers had deep knowledge of literacy and instruction. To do this, the project delivered six days of teacher workshops, dispersed over the course of the school year, each focused on evidence-based early literacy instruction. Neuhaus staff members delivered the workshops in ways that complemented the core curricular materials used by HISD (i.e., Frog Street) with the intent that teachers would use Neuhaus training to supplement the adopted district curriculum. The purpose of the workshops was to improve teacher knowledge and practices that were most likely to impact student literacy learning. More specifically, the workshops aimed to build teacher knowledge about oral language, phonological awareness, letter/sound knowledge, and concepts of print (Burns, Griffin, & Snow, 1999). Additionally, the workshops developed teacher knowledge about instructional practices that can be used to develop student literacy skills in the same areas, such as assessment and intervention design. The workshops also addressed family engagement. Finally, to support teachers in implementing what they learned, Neuhaus provided teachers with themed literacy units with specific protocols for incorporating learning from the workshops into their instructional practices. In addition to the training workshops, each participating teacher was also provided with jobembedded coaching to increase their ability to apply the instructional strategies that had been learned. Coaching was based on the instructional coaching model developed by the University of Oklahoma’s (OU) Center for Early Childhood Professional Development. Coaching techniques included side-by-side, shadow, demonstration, and observation. Coaches provided data concerning some aspects of implementation. For each coaching session, coaches reported on the types of coaching activities in which they engaged with teachers as well as the topics of literacy that were covered during the visit. Neuhaus Education Center: Teachers Make the Difference 1 Study Design The study used a quasi-experimental design, but incorporated aspects of random assignment to create three groups: • Group 1: Teachers who receive workshop training and high-intensity coaching • Group 2: Teachers who receive workshop training and low-intensity coaching • Group 3: Teachers who receive neither workshop training nor coaching Schools of the teachers in groups 1 and 2 were randomly assigned to the two coaching intensity conditions and then teachers within a school were randomly assigned to participate in the study (some teachers received the workshop training and coaching, but not all are included in the data analyses). Schools in Group 3 were selected through a convenience sampling process. Neuhaus engaged the HISD central office in identifying schools that could be approached for participation. Then, Neuhaus sought the permission of the principal, teachers, and students based on the options given to them. Only those who gave permission were included in the study. The Technical Appendix contains more detail about random assignment and coaching groups. The evaluation asked the following questions: • To what degree was the project implemented as intended? o How much professional development did teachers receive? o How much coaching did teachers receive? How did the two levels of coaching intensity differ? o Did teachers report applying workshop content in their classrooms? o What were teachers’ perceptions of the content and delivery of TMTD? • To what degree do participating teachers increase their knowledge of early literacy skills and concepts? • Do students’ early learning outcomes differ for students whose teachers receive professional development compared to those whose teachers do not? Specifically, do student outcomes differ, as measured by the Early Literacy Quick Assessment (ELQA)1? Do results differ for students with different background characteristics? • Do students’ early learning outcomes differ for students whose teachers receive different levels of coaching? Specifically, do student outcomes differ as measured by the ELQA? Do results differ for students with different background characteristics? Originally the evaluation also included analyses of the district’s PreK curriculum-based assessment, Frog Street. It was removed because initial analyses indicated the data were not reliable, we suspect for three reasons: (1) Frog Street was administered by teachers rather than coaches; (2) it was implemented by HISD the first time in fall 2013; and (3) Frog Street software allowed for multiple testing of students, which made comparisons unreliable. 1 2 Education Northwest Data Collection The study relied on data collected from multiple sources, including surveys, a coaching log, workshop attendance, and student assessments administered by the project and the HISD. Surveys Two surveys were administered to teachers: a program-developed Teacher Knowledge Survey and an evaluation-developed Teacher Survey. The Teacher Knowledge Survey contains 30 items addressing teachers’ knowledge and practice concerning the early literacy skills of oral language, phonological awareness, letter recognition, and concepts of print. Workshop participants completed the survey during the first and fifth workshops. Staff members at Neuhaus entered teacher responses from the survey into a spreadsheet, scored it, and forwarded the data to Education Northwest for analyses. Education Northwest developed a survey that addressed issues related to the TMTD workshops and coaching, use of TMTD materials, classroom practice, family involvement, and demographics. The evaluation survey was administered by Neuhaus to teachers in spring 2014. TMTD teachers completed the entire survey; comparison teachers completed the sections on demographics, classroom practice, and family involvement. TMTD teachers completed the survey online; comparison teachers completed a paper version of the survey. A copy of the Teacher Survey is in Appendix A. Coaching Log In fall 2013, Education Northwest, in consultation with Neuhaus, developed an online coaching log for coaches to document their coaching visits with participating teachers. The log tracks the coach, campus, and teacher involved in coaching, the date and minutes of coaching, coaching techniques used, and early literacy topics covered. Coaches could enter data into the log through mid-June 2014. A copy of the Coaching Log is in Appendix B. Workshop Attendance Neuhaus collected attendance data at each of the six workshops. This information was forwarded to Education Northwest in spring 2014 for analysis. Student Assessments Neuhaus administered OU’s ELQA to students in participating teachers’ classrooms in fall 2013 and spring 2014. ELQA is a computer-based assessment for use with preschool children. It is designed to be administered at the beginning of the school year and at four additional specific time periods throughout the rest of the year to measure progress on specific early literacy skills. The assessment is delivered to children orally by an adult. While some assessment items provide the opportunity for children to interact with the screen, the final answer is marked by the adult as correct or incorrect. Neuhaus Education Center: Teachers Make the Difference 3 The ELQA has seven subtests which measure different skills that are most predictive of later reading ability. Each subtest has a set number of items, and each item is worth one point and is equally weighted within the respective subtest. OU does not provide an aggregate score across all items and does not provide any documentation for or against combining the subtest scores. Table 1 shows the seven subtests and the associated number of possible items. Table 1 ELQA Subtests and Number of Items Number of Items Print Concepts 10 Picture Naming 25 Rhyming 10 Letters: Uppercase 26 Letters: Lowercase 26 Letter Sounds 26 Receptive Vocabulary 10 Total Possible Items 133 The ELQA was administered in fall 2013 and spring 2014 to students of all teachers who participated in the study (High Intensity, Low Intensity, and comparison). Students in the TMTD Teacher classrooms were tested by the end of September. Students in comparison teacher classrooms did not complete ELQA testing until mid-October, because parental permission was obtained less quickly. All students were assessed on the ELQA in April and May 2014. The ELQA was administered to students by the Neuhaus coaches to ensure consistency and validity. Student ELQA results were only included in the analysis if the student had both fall and spring scores. Education Northwest worked in collaboration with the HISD and UO to obtain student-level pretest and posttest data from all teachers participating in the study. Participant Demographics Demographic data were collected from teachers and students. Teacher Demographics Although evaluators at Education Northwest received a teacher demographic file from HISD, too many study teachers were missing from it. The following analyses are based on demographics from the Education Northwest Teacher Survey. Survey data show that teachers in the TMTD and comparison groups were similar. Comparing all of the TMTD teachers (High and Low Intensity) to the comparison teachers, we 4 Education Northwest found they taught for similar lengths of time overall, in their district, in their school, and at their grade level. Furthermore, we found teachers were similarly prepared to teach in terms of degrees and certifications, and that they were similar in terms of gender and ethnicity. Table 2 shows there were three differences between the High and Low intensity teachers: the High Intensity teachers had more years of experience teaching, overall, and taught in the HISD and at their current campus significantly longer. There was no statistically significant difference between the two groups in the length of time they taught prekindergarten. Table 2 Teaching Experience, by Group All Teachers High Intensity Teachers Low Intensity Teachers All TMTD Teachers Comparison Teachers 100 34 34 68 32 Survey participants N (response rate) 77 (77%) 21 (62%) 29 (85%) 50 (74%) 27 (84%) Years teaching Mean (SD) 12.9 (10.0) 17.3 (11.4) 9.8 (8.6)* 13.0 (10.4) 12.9 (9.4) Years teaching in HISD Mean (SD) 10.9 (9.7) 14.7 (11.2) 8.5 (8.4)* 11.0 (10.0) 10.6 (9.3) Years teaching at campus Mean (SD) 7.3 (6.6) 10.4 (7.5) 5.2 (4.9)* 7.3 (6.5) 7.2 (6.9) Years teaching Prekindergarten Mean (SD) 5.7 (5.2) 7.5 (6.2) 4.6 (4.5) 5.8 (5.4) 5.6 (5.1) Study Teachers N * p≤.05 Across the three groups, respondents were similar in terms of their preparation to teach. Onehalf of all teachers (51%) had a four-year degree and two-thirds of teachers (43%) possessed more than a four-year degree; less than 10 percent of teachers (7%) had alternative certification. Almost all respondents had an early childhood education certification (91%) and about twothirds (39%) possessed English as a Second Language certification. Fewer teachers had certifications in reading, bilingual education, and special education (9%, 7%, and 5%, respectively). One-quarter of teachers reported having other teaching certifications. The vast majority of respondents were female (94%). Almost one-half (47%) were African American/Black and one-quarter (25%) were Caucasian; fewer were Hispanic/Latino(a) or Asian/Pacific Islander (16% and 9%, respectively). Neuhaus Education Center: Teachers Make the Difference 5 Student Demographics Demographic and fall student assessment data show that students in the three groups were not similar. In regard to demographic data, the High Intensity Teachers had significantly more Hispanic and White students and significantly fewer Black students and students eligible for free and reduced-price lunch than the Low Intensity teachers. Differences continued to exist after combining the students from the High Intensity and Low Intensity teachers (All TMTD Teachers) and comparing them to the comparison teachers: “Other” and limited English proficiency students made up a significantly smaller proportion of TMTD teachers’ classrooms, but TMTD teachers had significantly more students eligible for free and reduced-price lunch than comparison teachers (Table 3). Table 3 Student Demographics, by Group Student Characteristics Student Participants N All Teachers High Intensity Teachers Low Intensity Teachers All TMTD Teachers Comparison Teachers 892 338 292 630 262 Female 56% (498) 53%(179) 58% (169) 55% (348) 57% (150) Hispanic 47% (442) 55% (187) 36% (105)* 46% (292) 50% (130) Black 40% (358) 32% (108) 54% (157)* 42% (265) 35% (93) White 7% (62) 10% (33) 5% (16)* 8% (49) 5% (13) Other 6% (50) 3% (10) 5% (14) 4% (24) 10% (26)* Special Education 1% (%) ** Limited English Proficiency 10% (86) 7% (23) 8% (22) 7% (45) 16% (41)* Free and ReducedPrice Lunch 93% (831) 93% (313) 98% (285)* 95% (598) 89% (233)* * p≤.05; ** <10 students Table 4 shows fall ELQA scores for the same groups of teachers. It shows that the scores of the students of the High and Low Intensity teachers were similar on five of the seven subtests (Print Concepts, Uppercase and Lowercase Letters, Letter Sounds, and Vocabulary). On two of the subtests, Picture Naming and Rhyming, student scores were significantly higher in the classrooms of the High Intensity teachers than of the Low Intensity teachers. However, the students of the TMTD teachers scored significantly lower than students of the comparison teachers on all subtests, except Picture Naming, where they scored significantly higher. 6 Education Northwest Table 4 Fall ELQA Data, by Group Mean (SD) All Teachers High Intensity Teachers Low Intensity Teachers All TMTD Teachers Comparison Teachers 892 338 292 630 262 Print Concepts 3.27 (2.31) 3.17 (2.33) 3.00 (2.24) 3.09 (2.29) 3.70 (2.33)* Picture Naming 18.13 (5.08) 18.96 (4.51) 17.85 (5.68)* 18.45 (5.12) 17.39 (4.91)* 3.41 (2.27) 3.51 (2.21) 2.68 (1.84)* 3.12 (2.09) 4.11 (2.54)* Uppercase Letters 12.01 (10.01) 10.77 (9.66) 11.46 (10.19) 11.09 (9.91) 14.23 (9.93)* Lowercase Letters 7.63 (10.10) 6.43 (9.67) 6.96 (9.52) 6.68 (9.60) Letter Sounds 4.62 (7.82) 3.59 (6.98) 3.58 (6.69) 3.59 (6.97) 7.10 (9.11)* Vocabulary 6.53 (2.16) 6.49 (2.06) 6.34 (2.30) 6.41 (2.18) 6.81 (2.09)* ELQA Subtests Student Participants N Rhyming 9.91 (10.89)* * p≤.05 Regardless of these differences, surveyed teachers reported working with similar types of students: • 60 percent were Tier 1—having all of their learning needs met with core classroom instruction • 32 percent were Tier 2—having learning needs met through core and supplemental instruction • 13 percent were Tier 3—having learning needs met through intensive intervention beyond core and supplemental instruction Neuhaus Education Center: Teachers Make the Difference 7 8 Education Northwest Teachers Make the Difference Intervention Teachers Make the Difference (TMTD) consists of two components: professional development workshops and coaching. The study sought to answer the following questions about the professional development: • To what degree was the project implemented as intended? o How much professional development did teachers receive? o What were teachers’ perceptions of the content and delivery of TMTD? o How much coaching did teachers receive? How did the two levels of coaching intensity differ? To answer these questions, evaluators analyzed data collected from project documentation including workshop attendance and materials and Education Northwest’s Teacher Survey, administered in spring 2014. Workshops During the 2013–2014 school year, Neuhaus Education Center (Neuhaus) provided six TMTD workshops. Each workshop lasted one day, but multiple workshops were offered to provide teachers a choice in the day they attended. The six workshops were offered as follows: • September 9-12, 2013 • October 8-10, 2013 • November 12-14, 2013 • January 14-16, 2014 • March 11-13, 2014 • May 19-23, 2014 Content The TMTD workshops provided participating teachers time to come together as a group and learn about the early literacy concepts of oral language, phonological awareness, letter recognition, and concepts of print. Each workshop focused on a different concept and provided an overview of the relevant research and the associated Texas Pre-Kindergarten Guidelines. During the workshops, Neuhaus staff members previewed one of the four Language and Literacy Units; provided suggestions for instruction, literacy center activities, and partnering with parents; conducted a read aloud and shared book recommendations; and included time for hands-on activities, such as lesson planning. Additional topics covered during the workshops included the importance of prekindergarten (PreK), the developmental stages of language and Neuhaus Education Center: Teachers Make the Difference 9 literacy, the Early Literacy Quick Assessment (ELQA), the district curriculum (Frog Street), and coaching. Teacher Attendance at Workshops Teachers participated in the professional development offered to them. According to Neuhaus attendance data, the vast majority of teachers (91%) participated in all six workshops (85% of High Intensity teachers and 97% of Low Intensity teachers). On average, High Intensity teachers participated in 5.8 workshops and Low Intensity teachers participated in 6.0 workshops. The difference was not statistically significant. No comparison teachers participated in the workshops. Survey data verify these findings—while reported participation was greater than 90 percent for all workshops for both the High and Low Intensity teachers, the Low Intensity teachers consistently reported 100 percent attendance, while that of the High Intensity teachers varied between 91 and 100 percent. Teacher Perceptions of Workshops Overall, both the High and Low Intensity teachers were pleased with the workshops and felt the time spent on different topic areas was “just right.” The vast majority of teachers (at least 85%) agreed the workshops were: • Of high quality (88%) • Highly useful (86%) • Aligned with other professional development they received this year (86%) • Flexible to meet the needs of participants (86%) • Supported by their principal (90%) Slightly fewer felt the workshops were effective in moving their practice forward (82%) or were ongoing and sustained (66%). Respondents were given an opportunity to provide feedback about the content of the workshops. Specifically they were asked about the handouts and presentations, and the time spent on early literacy content and certain TMTD components. Generally, the majority of respondents (at least 78%) felt the content was “just right.” These included: • Detail in handouts and presentations • Early literacy topics of concepts of print, letter recognition, oral langauge, and phonological awareness • Developmental stages of langauge and literacy and differentiating instruction • ELQA, using data, and organizing interventions • Working with parents 10 Education Northwest • Neuhaus Language and Litearcy Units and the Texas Pre-kindergarten Guidelines • Collaborative problem solving with other teachers • Neuhaus coaching model Content on which at least 10 percent of respondents thought “too little” time was spent included: • Differentiating instruction • Organizing interventions into three tiers • Using data • Learning about parents as partners • Learning about the Neuhaus coaching model There were two areas where Low Intensity teachers, compared to High Intensity teachers, felt there was “too little” time spent—learning about the Neuhaus Language and Literacy Units and the Texas Prekindergarten Guidelines. Coaching Teacher Participation in Coaching TMTD teachers received Neuhaus coaching; however, they did so with different frequency. Between October 2013 and April 2014, High Intensity teachers should have received 22 coaching sessions and Low Intensity teachers should have received seven coaching sessions, as shown in Table 5. Table 5 Anticipated Weeks of TMTD Coaching, by Month and Group Weeks of Coaching Month High Intensity Low Intensity October 4 1 November 3 1 December 3 1 January 4 1 February 4 1 March 2 1 April 2 1 Total 22 7 High Intensity teachers received more coaching than Low Intensity teachers, but less coaching than anticipated. Over the course of the year, 607 coaching sessions were entered into the TMTD coaching log for study teachers. The majority of coaching sessions (73%) were with High Intensity teachers. Neuhaus Education Center: Teachers Make the Difference 11 Neuhaus anticipated that coaches would provide each High Intensity teacher 22 coaching sessions, and each Low Intensity teacher seven coaching sessions, over the course of the school year. In total, coaches logged 440 coaching sessions with High Intensity teachers (59%) and 167 coaching sessions with Low Intensity teachers (70%). On average, High Intensity teachers received 12.9 coaching sessions and Low Intensity teachers received 4.9 coaching sessions. Table 6 shows the average number of coaching sessions High and Low Intensity teachers received each month. Coaches were most successful in providing the anticipated amount of coaching in November, February, and March. While coaching was anticipated in April, very few teachers received coaching in that month. Testing occurred in January, which could account for lower than anticipated coaching sessions. Table 6 Average Number of Coaching Sessions, by Month and Group Mean (SD) Month High Intensity Low Intensity October 2.5 (1.1) 1.0 (0.7) November 2.4 (1.3) 0.8 (0.6) December 1.1 (1.0) 0.5 (0.6) January 1.2 (1.3) 0.6 (0.9) February 3.3 (1.2) 1.0 (0.8) March 2.3 (1.4) 0.9 (0.6) April 0.0 (0.2) 0.2 (0.4) Total 12.9 (4.5) 4.9 (2.8) Survey data corroborates the coaching model. High Intensity teachers were significantly more likely to report receiving coaching more frequently than Low Intensity teachers; regardless, teachers were satisfied with the amount of coaching they received. The majority of High Intensity teachers (72%) reported receiving coaching visits between three and five times a month, whereas the majority of Low Intensity teachers (68%) reported receiving coaching visits no more than once a month. Regardless, the majority of teachers felt the amount of coaching was “just right” (90%). Coaching sessions tended to last about an hour (52.4 minutes). While the average length of coaching visits varied between the High and Low Intensity teachers (45 minutes compared to 60 minutes, respectively), the difference was not statistically significant. Again, the majority of teachers reported the length of the visits was “just right” (88%). Content of Coaching During coaching sessions, coaches could address up to eight coaching topics (phonological awareness, letter recognition, grapheme-phoneme correspondence, oral language, emergent writing, teacher questioning strategies, concepts of print, and checking for student 12 Education Northwest understanding) using up to four coaching techniques (side-by-side, shadow, demonstration, and observation). According to coaching logs, the frequency with which most coaching techniques were used, and early literacy topics were addressed, was similar across groups; however, there were a few significant differences: • Side-by side coaching was addressed with proportionately greater frequency with High Intensity teachers • Shadow coaching was addressed with proportionately greater frequency with Low Intensity teachers • Oral language—including vocabulary development, naming, academic vocabulary and multiple meanings—was addressed with proportionately greater frequency with Low Intensity teachers Figures 2 and 3 display the frequency with which coaches used these different techniques and addressed the various early literacy topics with their High Intensity and Low Intensity teachers. Figure 2 Frequency With Which Coaches Used Coaching Techniques 59% 60% 59% Observation 37% 35% 36% Demonstration 32% Side-by-side 13% 27% 6% 11% 7% Shadow 0% 20% High Intensity Teachers 40% 60% 80% Low Intensity Teachers 100% All TMTD Teachers The two most frequently used coaching techniques were used similarly across the two groups. Figure 2 shows coaches most frequently used the observation (59%) and demonstration (36%) techniques with teachers, followed by side-by-side (27%) and shadow (7%). There were no differences in their use of the observation and demonstration coaching techniques between the two groups. However, side-by-side coaching was significantly more likely to be used with High Intensity teachers (32%) than with Low Intensity teachers (13%), and shadowing was Neuhaus Education Center: Teachers Make the Difference 13 significantly more likely to be used with Low Intensity teachers (11%) than with High Intensity teachers (6%). This was corroborated with survey data. Teachers were asked to report the frequency with which their coach engaged in certain activities. Teachers reported that observations occurred with consistent regularity, followed by demonstration lessons, and side-by-side and shadow coaching: • Observe student engagement (72% “regularly” or “always”) • Observe students learning (69% “regularly” or “always”) • Observe instruction (66% “regularly” or “always”) • Demonstrate lessons (48% received “regularly” or “always,” but 42% received “never” or “occasionally”) • Side-by-side coaching (40% received “regularly” or “always,” but 46% received “never” or “occasionally”) • Shadow coaching (37% received “regularly” or “always,” but 47% received “never” or “occasionally”) 14 Education Northwest Figure 3 Frequency With Which Coaches Addressed Early Literacy Topics 64% Oral language 75% 67% 52% 60% 54% Checking for student understanding Letter recognition 54% 54% 54% Phonological awareness 49% 54% 50% 42% 47% 43% Teacher questioning strategies 36% Concepts of print 45% 38% Emergent writing 25% 24% 25% Grapheme-phoneme correspondence 25% 19% 23% 0% High Intensity Teachers 20% 40% Low Intensity Teachers 60% 80% 100% All TMTD Teachers The topic most frequently addressed by coaches—oral language—was addressed with Low Intensity teachers with proportionately greater frequency than with High Intensity teachers; all other topics were coached with equal frequency between the two groups. Figure 3 shows that when working with teachers, coaches most frequently addressed oral language (67%), checking for student understanding (54%), letter recognition (54%), and phonological awareness (50%). These were followed by the topics of teacher questioning strategies (43%), concepts of print (38%), emergent writing (25%), and grapheme-phoneme correspondence (23%). All topics except oral language were addressed by coaches with the same frequency. Oral language was addressed with proportionately greater frequency with the Low Intensity teachers (75%) than the High Intensity teachers (64%). Neuhaus Education Center: Teachers Make the Difference 15 In regard to oral language, two of the most frequently addressed topics—vocabulary development and naming—were addressed by coaches with Low Intensity teachers with proportionately greater frequency than with High Intensity teachers. Two additional topics— academic vocabulary and multiple meanings—while addressed overall with less frequency, were also addressed with Low Intensity teachers with proportionately greater frequency than with High Intensity teachers. The remaining oral language topics, including listening comprehension, describing, read alouds, background knowledge, inferencing, and story retell were coached with proportionately equal frequency between the two groups (Table 7). Table 7 Frequency With Which Coaches Addressed Oral Language Topics, by Group Percent of Receiving Coaching (N) Oral Language Topics All Teachers High Intensity Low Intensity Vocabulary development 53.4% (324) 50.8% (223) 60.1% (101)* Naming 51.6% (313) 48.7% (214) 58.9% (99)* Listening comprehension 44.0% (267) 42.4% (186) 48.2% (81) Describing 40.9% (248) 38.5% (169) 47.0% (79) Read alouds 30.8% (187) 29.2% (128) 35.1% (59) Academic vocabulary 29.2% (177) 25.3% (111) 39.3% (66)* Background knowledge 26.7% (162) 25.7% (113) 29.2% (49) Inferencing 14.3% (87) 13.7% (60) 16.1% (27) Story retell 12.7% (77) 12.3% (54) 13.7% (23) 4.8% (29) 3.2% (14) Multiple meanings 8.9% (15)* p<.05 Regarding phonological awareness topics, coaches addressed rhyme, segmenting syllables, and phonemic awareness topics with similar frequency (about one-quarter of their sessions included these topics), as shown in Table 8. Onset and rime was addressed less often (7%). Regardless, the frequency with which these topics were addressed was similar across the two groups. Table 8 Frequency With Which Coaches Addressed Phonological Awareness Topics, by Group Percent of Receiving Coaching (N) Phonological Awareness Topics All Teachers High Intensity Low Intensity Rhyming 26.0% (158) 26.2% (115) 25.6% (43) Segmenting syllables 24.7% (150) 23.2% (102) 28.6% (48) Phonemic awareness 24.2% (147) 23.9% (105) 25.0% (42) 7.2% (44) 7.1% (31) 7.7% (13) Onset and rime 16 Education Northwest The Education Northwest Teacher Survey also addressed additional areas of coaching. The survey asked teachers about the frequency with which coaches engaged in certain activities. The following activities were slightly more likely to occur with more regularity than less: • Develop plan of action (57% “regularly” or “always”) • Interpret student assessment data (51% “regularly” or “always”) Teachers were less likely to report getting regular support on differentiating instruction (47% received “regularly” or “always,” but 39% received “never” or “occasionally”). Neuhaus Education Center: Teachers Make the Difference 17 18 Education Northwest Teachers’ Outcomes According to the Neuhaus Education Center’s (Neuhaus) Teachers Make the Difference (TMTD) theory of action, teachers’ participation in the TMTD workshops and coaching leads to teachers’ increased knowledge about early literacy, use of the TMTD materials, and changes in classroom practices and family involvement strategies. The study sought to answer the following questions: • To what degree do participating teachers increase their knowledge of early literacy skills and concepts? • To what degree did teachers report applying workshop content in their classrooms? To answer these questions, evaluators analyzed data collected by Neuhaus’ Teacher Knowledge Survey, administered in fall 2013 and spring 2014, and by Education Northwest’s Teacher Survey, administered in spring 2014. Increased Early Literacy Knowledge To measure changes in teacher knowledge of early literacy concepts, Neuhaus created a Teacher Knowledge Survey. Staff administered the survey to teachers participating in workshops in fall and spring. The survey includes 30 items and addresses the early literacy skills of oral language, phonological awareness, letter recognition, and concepts of print. In spring, TMTD teachers answered significantly more items correctly than they did in fall. These findings held for all teachers taking the survey in fall compared to all teachers taking the survey in spring (unmatched) and for study teachers taking the survey in fall compared to those taking the survey in spring (matched) (see Table 9). Table 9 Teacher Knowledge Survey, Items Answered Correctly, Fall and Spring, by Group Mean (SD) Teacher Group N Fall Spring High Intensity 19 18.5 (3.8) 21.1 (3.7)* Low Intensity 26 18.2 (4.8) 21.7 (3.3)* High and Low Intensity 45 18.3 (4.4) 21.4 (3.4)* All Teachers 86 18.5 (3.9) All Teachers 67 21.0 (3.6)* *p<.01 Neuhaus Education Center: Teachers Make the Difference 19 Use of TMTD Materials All TMTD workshop participants leave trainings 1 thru 4 with a Language and Literacy Unit. Each unit is comprised of a theme (“The Kitchen,” “The Farm,” “People, People Everywhere,” and “Me and the World Around Me”) and is divided into three sections: lessons, extension activities, and book recommendations. Teachers introduce the lessons, use the extension activities to give students practice, and draw from the book recommendations to reinforce learning. Each unit includes topics in oral language, phonological awareness, letter recognition, and concepts of print. High and Low Intensity teachers used the TMTD materials with similar frequency. As shown in Table 10, almost all teachers used the materials at least once a week (86%) and for about 40 minutes each time. Table 10 Frequency and Duration With Which Teachers Used TMTD Materials Frequency of Use Daily Percent (N) Mean (SD) All TMTD Teachers Average Minutes 8% (4) 41.3 (14.4) A few times a week 60% (30) 88.3 (91.3) Once a week 18% (9) 35.0 (14.4) Once every other week 4% (2) 175.0 (205.1) Once a month 4% (2) 40.0 (28.3) Less than once a month 4% (2) 20.0 (14.4) Never 2% (1) NA Teachers most frequently used the TMTD materials with their Tier 3 students during smalland whole-group instruction. Teachers were provided an opportunity to rank which tiers they served and the instructional modes they chose when using TMTD materials. Table 11 displays the percentage of teachers ranking each as “1” (most frequent), “2” or “3.” Table 11 Teachers’ Preferred Use of TMTD Materials, by Rank Percent (N) in each Rank Student Tier and Instructional Mode “1” “2” “3” Tier 1 students 41.5% (17) 7.3% (3) 51.2% (21) Tier 2 students 11.4% (4) 82.9% (29) 5.7% (2) Tier 3 students 58.5% (24) 14.6% (6) 26.8% (26.8) Whole-group instruction 41.5% (17) 31.7% (13) 26.8% (11) Small-group instruction 53.7% (22) 31.7% (13) 14.6% (6) One-on-one instruction 22.2% (8) 30.6% (11) 47.2% (17) 20 Education Northwest Table 11 shows teachers most frequently used the TMTD materials with their Tier 3 students, followed by their Tier 2 students, and finally with their Tier 1 students. Whole- and small-group instructional modes were preferred over one-on-one instruction. Almost all teachers (96%) reported using the TMTD materials as a supplement to Frog Street materials. Teachers were slightly more likely to use the lessons than the extension activities, and were least likely to use the book recommendations. Regardless, the largest percentage of teachers indicated they used “Some” of the materials as opposed to “None,” “Many,” or “Most.” Table 12 provides the average with which teachers reported using the sections, across all units. Table 12 Average Teacher Use of TMTD Units, by Section Lessons Extension Activities Books None 6.9% 11.9% 14.5% Some 42.1% 50.6% 60.9% Many 33.7% 31.0% 16.2% Most 17.4% 6.5% 8.4% Teachers used “The Kitchen,” “The Farm,” and “People, People Everywhere” units slightly more than the “Me and the World Around Me” unit. Finally, when asked to what extent the TMTD materials provided additional instructional support for children across the four early literacy content areas addressed, teachers were most likely to indicate “Substantial” support for oral language (78%), phonological awareness (76%), letter recognition (69%), and concepts of print (61%). Teachers were less likely to report “Minimal” than “Somewhat” support, and were not at all likely to indicate the materials were “Not at all” supportive. Changes in Practice Evaluators looked at two areas where changes in practice could occur— instruction of early literacy skills and family involvement. Instruction Evaluators gauged the impact of teacher participation in TMTD on classroom practice in two ways. First, on the survey, we asked how various coaching activities impacted teacher instruction. Second, and also on the survey, we asked teachers the extent to which they used different instructional activities to build early literacy skills with their students. Neuhaus Education Center: Teachers Make the Difference 21 Table 13 shows the percentage of teachers that reported participating in a variety of coaching activities. Of those teachers, it also shows the extent that each coaching activity impacted their instruction “Somewhat” and “Substantially.” Table 13 Percentage of Teachers Participating in Coaching Activities Coaching Activity Percent of TMTD teachers reporting that it occurred at least “Occasionally” Percent of teachers reporting that it occurred at least “Occasionally,” who felt it . . . Impacted their instruction at least “Somewhat” Impacted their instruction “Somewhat” Impacted their instruction “Substantially” Observe instruction 100.0% (49) 70.2% (33) 34.0% (16) 36.2% (17) Observe student engagement 100.0%(49) 72.9% (35) 35.4% (17) 37.5% (18) Observe students learning 97.9% (47) 76.1% (35) 34.8% (16) 41.3% (19) Help interpret student assessment data 95.7% (45) 75.6% (34) 26.7% (12) 48.9% (22) Develop a customized plan of action 89.8% (44) 83.8% (36) 32.6% (14) 51.2% (22) Help differentiate instruction 83.7% (41) 85.0% (34) 30.0% (12) 55.0% (22) Demonstrate lessons 79.2% (38) 86.5% (32) 21.6% (8) 64.9% (24) Conduct side-by-side coaching 75.5% (37) 91.7% (33) 30.6% (11) 61.1% (22) Conduct shadow coaching 67.3% (33) 90.7% (29) 31.3% (10) 59.4% (19) Teachers reported that participating in all coaching activities impacted their instruction, but that participation in demonstration lessons and side-by-side and shadow coaching most impacted their instruction. Table 13 also shows that the majority of teachers reported their participation in coaching activities positively impacted their instruction (at least 70% of teachers reported “somewhat” of an impact). Survey results reveal that some coaching activities had more impact than others. These included demonstration lessons, side-by-side and shadow coaching, and help differentiating instruction. At least two-thirds of teachers indicated that demonstration lessons and side-by-side and shadow coaching substantially impacted their instruction. Teachers were asked to what extent they provide instructional activities to build early literacy skills in four areas—oral language, phonological awareness, letter recognition, and concepts of print. Within each area, several subskills were identified. Teachers were provided three choices, which we converted to numeric values: “I don’t do this” was converted to 0; “I do this, but could use more professional development/resources” was converted to 1; and “I do this well” was converted to 2. Teachers were asked to respond separately for their Tier 1 and Tier2/Tier 3 students. We summed the subskill scores to obtain a skill-level total, with a higher score 22 Education Northwest indicating more skilled use of instructional practices. Table 14 displays the total score possible for each skill area. Table 14 Reported Teacher Skill Levels in Use of Strategies to Build Early Literacy Skills, by Group Mean (SD) Early Literacy Skills Total Possible Score Tier 1 Students Tier 2/Tier 3 Students All TMTD Teachers All Comparison Teachers All TMTD Teachers All Comparison Teachers 6 5.4 (1.0) 4.9 (1.2) 4.7 (1.2) 5.1 (1.2) 22 19.1 (3.1) 16.7 (3.8)* 16.4 (4.2) 16.2 (4.2) Letter recognition 8 7.8 (0.6) 7.4 (1.3) 7.2 (1.4) 7.0 (1.6) Concepts of print 8 7.6 (0.9) 7.3 (1.3) 7.0 (1.5) 7.1 (1.4) Oral language Phonological awareness *p<.05 TMTD teachers were significantly more likely than comparison teachers to report more skilled use of instructional practices to improve phonological awareness skills with their Tier 1 students. Regardless of group, teachers reported similar skill level in the use of the instructional practices to build oral language, letter recognition, and concepts of print skills with their Tier 1 students. There was no difference between TMTD and comparison teachers in reported skill level in their use of instructional practices to build these same skills with Tier 2/Tier 3 students. However, TMTD teachers reported significantly lower skill levels for using instructional practices to build these literacy with their Tier 2/Tier 3 than with Tier 1 students; comparison teachers did not. Instructional Practices to Build Oral Language Skills On the survey there were three practices teachers could use to increase students’ oral language skills. These included: • Responding to read-alouds that demonstrate understanding of what has been read • Retelling or reenacting a story after it is read aloud • Asking/answering appropriate questions about a book read aloud There were no differences in reported skill level of instructional practices used, between High and Low Intensity or TMTD and comparison teachers, with their Tier 1 and Tier 2/Tier 3 students. Instructional Practices to Build Phonological Awareness Skills The survey included 11 practices teachers could use to increase phonological awareness skills. Table 15 lists these practices and shows which were practiced with equal skill by all teachers and which were practiced differently by TMTD and comparison teachers with Tier 1 students. All practices were reportedly used with equal skill by all teachers with their Tier 2/Tier 3 students. Neuhaus Education Center: Teachers Make the Difference 23 Table 15 Use of Instructional Practices to Build Phonological Awareness Skills With Tier I Students Practices Used with Similar Skill Level Across All Teachers • • • • • • • Recognizing alliteration Combining words to make compound words Combining syllables into a word Deleting syllables from a word Producing words that begin with the same sound as other words (onset) Combining onset and rime to form familiar one-syllable words without pictorial support Blending two phonemes into real words with pictorial support Practices Used at a Higher Skill Level by TMTD Teachers • • • • Recognizing rhyme Separating a normally spoken sentence into individual words Removing a word from a compound word Producing a word that rhymes with a given word Instructional Practices to Build Letter Knowledge Skills The survey included four practices teachers could use to increase letter knowledge skills. In three of them, • Naming at least 20 uppercase letters • Naming at least 20 lowercase letters • Recognizing at least 20 letter/sound correspondences there were no differences in reported skill levels in using instructional practices, between High and Low Intensity or TMTD and comparison teachers, with their Tier 1 and Tier 2/Tier 3 students. TMTD teachers self-reported a higher skill level than did comparison teachers when applying the fourth practice—producing at least 10 letter sounds—with their Tier 1 students. Instructional Practices to Build Concepts of Print Skills Finally, the survey included four practices teachers could use to increase concepts of print skills. There were no differences in reported skill level in using the instructional practices, between High and Low Intensity or TMTD and comparison teachers, with their Tier 1 and Tier 2/Tier 3 students in three of them: • Identifying the front and end of a book • Knowing that spoken words can be represented with printed words • Identifying words on a printed page 24 Education Northwest TMTD teachers self-reported a higher skill level than did comparison teachers when applying the fourth practice—identifying where a printed word begins and ends—with their Tier 1 students. Family Involvement Teachers were asked about their beliefs and the supports they provided to parents to develop the oral language and early literacy skills of their children at home. Almost all teachers believed that parents are capable of improving the early language and literacy skills of their children. Teachers reported having repertoires of strategies to share with parents and spending time communicating those strategies. Most teachers communicated with parents at least once a month. There were no differences between TMTD and comparison teachers. Table 16 shows that the vast majority of TMTD and comparison teachers “agreed” or “strongly agreed” they know strategies that support oral language and early literacy skill development for parents to use with their children and that they spend a significant amount of time communicating those strategies to parents. While the differences between the groups were not statistically significant, the trends indicate that larger proportions of TMTD teachers than comparison teachers “strongly agreed,” and larger proportions of comparison teachers than TMTD teachers “agreed.” Table 16 Reported Oral Language and Early Literacy Supports to Parents, by Group TMTD Teachers Agree Strongly Agree Comparison Teachers Agree Strongly Agree I know what types of things parents/families can do to support the oral language skills of their children. 58.3% (28) 39.6% (19) 72.0% (18) 28.0% (7) I put a significant amount of my time into communicating with parents/families about how to support the oral language skills of their children. 50.0% (24) 39.6% (19) 68.0% (17) 16.0% (4) I know what types of things parents/families can do to support the early literacy skills of their children. 54.5% (24) 34.1% (15) 76.0% (19) 24.0% (6) I put a significant amount of my time into communicating with parents/families about how to support the early literacy skills of their children. 51.2% (22) 34.9% (15) 80.0% (20) 12.0% (3) Finally, teachers were asked to what extent they work with the parents of the children in their classroom to involve parents in oral language and early literacy development activities at home with their child. For these items, teachers used a scale of 1 to 5 with “1” being “I have done nothing/little to engage the parents/families of the children in my classroom in meaningful oral Neuhaus Education Center: Teachers Make the Difference 25 language/early literacy activities at home” and “5” being “I have a repertoire of strategies that I regularly use to engage the parents/families of the children in my classroom in meaningful oral language/early literacy activities at home.” Regardless of group, teachers reported themselves on the higher end of the scales. TMTD teachers reported themselves slightly higher on the scale than did comparison teachers, but these differences were not statistically significant (Table 17). Table 17 Teachers’ Perceptions of Their Work With Parents to Support Oral Language and Early Literacy, by Group Mean (SD) TMTD Teachers Comparison Teachers Oral language 3.9 (1.0) 3.6 (1.0) Early Literacy 3.9 (1.0) 3.7 (0.9) The majority of teachers communicated with parents at least monthly, with just over one-half doing so at least three times a month (Table 18). Frequency did not differ by group. Table 18 Frequency With Which Teachers Communicate With Parents, by Group TMTD Teachers Comparison Teachers Never 2.2% (1) 0.0% (0) Once during the year 0.0% (0) 0.0% (0) 2-3 times during the year 0.0% (0) 15.4% (4) 4-6 times during the year 6.7% (3) 3.8% (1) Once a month 15.6% (7) 19.2% (5) 2 times per month 15.6% (7) 11.5% (3) 3 times per month 6.7% (3) 3.8% (1) 4 times per month 37.8% (17) 26.9% (7) 5 or more times per month 15.6% (7) 19.2% (5) 26 Education Northwest Student Outcomes According to the Neuhaus Education Center’s (Neuhaus) Teachers Make the Difference (TMTD) theory of action, teachers’ participation in the TMTD workshops and coaching leads to teachers’ increased knowledge about early literacy, use of the TMTD materials, changes in classroom practices and family involvement strategies. In turn, the improved knowledge and practices lead to improved student literacy outcomes. The study sought to answer the following questions about student outcomes: • Do students’ early learning outcomes differ for students whose teachers receive professional development compared to those whose teachers do not? Specifically, do student outcomes differ, as measured by the Early Literacy Quick Assessment (ELQA)? Do results differ for students with different background characteristics? • Do students’ early learning outcomes differ for students whose teachers receive different levels of coaching? Specifically, do student outcomes differ as measured by the ELQA? Do results differ for students with different background characteristics? To answer these questions, evaluators analyzed ELQA student data collected by Neuhaus coaches in fall 2013 and spring 2014. The following analyses only include students who had ELQA scores for both the fall and spring administrations. Differences Based on Teacher Participation in TMTD To determine if there were differences in student outcomes based on their teachers’ participation in TMTD, evaluators first conducted a simple descriptive analysis of the average ELQA scores of the students of TMTD and comparison teachers. These descriptive averages were calculated for both fall 2013 and spring 2014. A series of t-tests were utilized to compare differences between the student groups for each ELQA subtest. Evaluators conducted further analysis to examine whether there were differential effects of the TMTD treatment on the rate of change on student ELQA scores from fall to spring. Fall 2013 ELQA Scores The average fall ELQA scores of students of TMTD teachers were significantly lower than those of students of comparison teachers on every subtest, except Picture Naming. This is consistent with the fact that the Houston Independent School District (HISD) targeted teachers in schools that were traditionally lower performing for participation in Neuhaus’ TMTD training. Figure 4 illustrates the differences in scores between groups. It shows the average percentage of items that students answered correct on each subtest, by teacher group. Table TA-3, in the Technical Appendix, shows average scores. Neuhaus Education Center: Teachers Make the Difference 27 In fall 2013, on average, students of both TMTD and comparison teachers scored highest on the ELQA subtest measures of vocabulary: Picture Naming (74% and 70% correct, respectively) and Receptive Vocabulary (64% and 68% correct, respectively). Figure 4 also shows that the students of comparison teachers answered a larger percentage of subtest measure items correctly in all areas, except Picture Naming. Figure 4 Percent of Items Answered Correctly in Fall 2013 31%* 37% Print Concepts 74%* 70% Picture Naming 31%* 41% Rhyming 43%* Letters: Uppercase 26%* Letters: Lowercase 14%* Letter Sounds 55% 38% 27% Receptive Vocabulary 64%* 68% 0% 20% 40% 60% 80% 100% Students of TMTD Teachers (n=630) Students of Comparison Teachers (n=262) *p<.01 Spring 2014 ELQA Scores The average spring ELQA scores of students of TMTD teachers were higher than those of students of comparison teachers on every subtest, except Lowercase Letters. Figure 5 illustrates the differences in scores between groups, again showing the average percentage of items that students answered correctly on each subtest, by teacher group. Table TA-4, in the Technical Appendix, shows average scores. 28 Education Northwest Figure 5 Percent of Items Answered Correctly in Spring 2014 78% Print Concepts 58% Picture Naming 86% 83% 70% 67% Rhyming 91% 90% Uppercase 83% 83% Lowercase 71% 65% Letter Sounds Receptive Vocabulary 80% 79% 0% 20% 40% 60% 80% 100% Students of TMTD Teachers (n=630) Students of Comparison Teachers (n=262) *p<.01; ** p<.05 Figure 5 shows that all students scored highest in the area of Uppercase Letters and that students of TMTD teachers answered a larger percentage of items correctly on all subtests, except Lowercase Letters. Significant differences between the two student groups were found in three areas: Print Concepts, Picture Naming, and Letter Sounds. Rate of Change Evaluators compared the rate of change of students of TMTD teachers to that of students of comparison teachers. We used a Difference-in-Differences (DD) regression analysis to compare the difference in the average change between pre- and posttests for the TMTD and comparison teachers. In a quasi-experimental design, where students are not randomly assigned to treatment and control groups, it is unlikely students will begin the school year with similar skill levels. As a result, the differences between groups cannot be entirely attributed to the treatment. DD regression normalizes the differences that exist between the groups aside from the treatment and measures the effect of the treatment as the difference in slope between the actual outcome and the expected outcome that would have occurred under normal conditions. When compared side by side, the differences between average fall and spring ELQA scores in Figure 6 illustrate more progress overall for students of TMTD teachers. Neuhaus Education Center: Teachers Make the Difference 29 Figure 6 Comparison of Average ELQA Scores in Percent Correct 100% 91% 86% 80% 90% 83% 83% 83% 80% 78% 74% 70% 70% 71% 67% 65% 64% 58% 60% 79% 81% 78% 68% 55% 48% 41% 43% 39% 38% 37% 40% 31% 31% 27% 26% 20% 14% 0% Print Concepts Picture Naming Rhyming Letters: Uppercase Letters: Lowercase Letter Sounds Receptive Total Score Vocabulary Fall 2013 Students of TMTD Teachers (n=630) Spring 2014 Students of TMTD Teachers (n=630) Fall 2013 Students of Comparison Teachers (n=262) Spring 2014 Students of Comparison Teachers (n=262) Figures 7 thru 13 show the results of the DD regression analyses for each subtest. They suggest that teacher participation in TMTD impacted student outcomes differently in six areas: Print Concepts, Rhyming, Uppercase Letters, Lowercase Letters, Letter Sounds, and Receptive Vocabulary. 30 Education Northwest The change from fall to spring on the average ELQA Print Concepts score among students of TMTD teachers was 2.69 points higher than that of students of comparison teachers. Figure 7 shows that, in fall, on average, students of TMTD teachers answered significantly more items correctly than did students of comparison teachers (3.7 and 3.1 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups answering an additional 2.06 items correctly. However, students of TMTD teachers answered 7.84 items correctly (2.69 items more than expected). This increased rate of change was significant (p<.01) and allowed students of TMTD teachers to score statistically higher than students of comparison teachers at the end the year, despite having begun the year significantly behind them. Figure 7 Change in ELQA Print Concepts Scores Average ELQA Print Concept Scores 10 8 Treatment Effect = 2.69 points 6 4 2 0 Fall 2013 Spring 2014 Students of TMTD Teachers Actual 3.09 7.84 Students of TMTD Teachers Expected 3.09 5.15 Students of Comparison Teachers 3.70 5.76 Neuhaus Education Center: Teachers Make the Difference 31 The change from fall to spring on the average ELQA Picture Naming score among students of TMTD teachers was 0.35 points lower than that of students of comparison teachers. Figure 8 shows that, in fall, on average, students of TMTD teachers answered significantly more items correctly than did students of comparison teachers (18.45 and 17.39 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups scoring an additional 3.36 items correctly. Students of TMTD teachers answered 21.50 items correctly (0.35 items less than expected). This decreased rate of change was not significant and allowed the students of TMTD teachers to remain significantly ahead of their peers in the classrooms of comparison teachers. Figure 8 Change in ELQA Picture Naming Scores Average ELQA Picture Naming Scores 25.00 No Treatment Effect 20.00 15.00 10.00 5.00 0.00 Fall 2013 Spring 2014 Students of TMTD Teachers Actual 18.45 21.50 Students of TMTD Teachers Expected 18.45 21.81 Students of Comparison Teachers 17.39 20.75 32 Education Northwest The change from fall to spring on the average ELQA Rhyming score among students of TMTD teachers was 1.24 points higher than that of students of comparison teachers. Figure 9 shows that, in fall, on average, students of TMTD teachers answered significantly fewer items correctly than did students of comparison teachers (3.12 and 4.11 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups scoring an additional 2.62 items correctly. However, students of TMTD teachers answered 6.99 items correctly (1.24 items more than expected). This increased rate of change was significant (p<.01) and allowed the students of TMTD teachers to finish the school year at a skill level similar to that of their peers in the classrooms of comparison teachers despite having started the year behind them. Figure 9 Change in ELQA Rhyming Scores Average ELQA Rhyming Scores 10 8 Treatment Effect = 1.24 points 6 4 2 0 Fall 2013 Spring 2014 Students of TMTD Teachers Actual 3.12 6.99 Students of TMTD Teachers Expected 3.12 5.74 Students of Comparison Teachers 4.11 6.73 Neuhaus Education Center: Teachers Make the Difference 33 The change from fall to spring on the average ELQA Uppercase Letters score among students of TMTD teachers was 3.41 points higher than that of students of comparison teachers. Figure 10 shows that, in fall, on average, students of TMTD teachers answered significantly fewer items correctly than did students of comparison teachers (11.09 and 14.23 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups scoring an additional 9.18 items correctly. However, students of TMTD teachers answered 23.69 items correctly (3.41 items more than expected). This increased rate of change was significant (p<.01) and allowed the students of TMTD teachers to finish the school year at a skill level similar to that of their peers in the classrooms of comparison teachers despite having started the year behind them. Average ELQA Uppercase Letters Scores Figure 10 Change in ELQA Uppercase Letters Scores 25 Treatment Effect = 3.41 points 20 15 10 5 0 Fall 2013 Spring 2014 Students of TMTD Teachers Actual 11.09 23.69 Students of TMTD Teacher Expected 11.09 20.27 Students of Comparison Teachers 14.23 23.41 34 Education Northwest The change from fall to spring on the average ELQA Lowercase Letters score among students of TMTD teachers was 3.19 points higher than that of students of comparison teachers. Figure 11 shows that, in fall, on average, students of TMTD teachers answered significantly fewer items correctly than did students of comparison teachers (6.68 and 9.91 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups scoring an additional 11.64 items correctly. However, students of TMTD teachers answered 21.52 items correctly (3.19 items more than expected). This increased rate of change was significant (p<.01) and allowed the students of TMTD teachers to finish the school year at a skill level similar to that of their peers in the classrooms of comparison teachers despite having started the year behind them. Average ELQA Lowercase Letters Scores Figure 11 Change in ELQA Lowercase Letters Scores 25 Treatment Effect = 3.19 points 20 15 10 5 0 Fall 2013 Spring 2014 Students of TMTD Teachers Actual 6.68 21.52 Students of TMTD Teachers Expected 6.68 18.32 Students of Comparison Teachers 9.91 21.55 Neuhaus Education Center: Teachers Make the Difference 35 The change from fall to spring on the average ELQA Letter Sounds score among students of TMTD teachers was 4.97 points higher than that of students of comparison teachers. Figure 12 shows that, in fall, on average, students of TMTD teachers answered significantly fewer items correctly than did students of comparison teachers (3.59 and 7.10 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups scoring an additional 9.92 items correctly. However, students of TMTD teachers answered 18.47 items correctly (4.97 items more than expected). This increased rate of change was significant (p<.01) and allowed the students of TMTD teachers to finish the school year at a significantly higher skill level than their peers in the classrooms of comparison teachers despite having started the year behind them. Figure 12 Change in ELQA Letter Sounds Scores Average ELQA Letter Sounds Scores 25.00 20.00 Treatment Effect = 4.97 points 15.00 10.00 5.00 0.00 Fall 2013 Spring 2014 Students of TMTD Teachers ActualTMD Study Group Actual 3.59 18.47 Students of TMTD Teachers Expected 3.59 13.51 Students of Comparison Teachers 7.10 17.02 36 Education Northwest The change from fall to spring on the average ELQA Receptive Vocabulary score among students of TMTD teachers was 0.45 points higher than that of students of comparison teachers. Figure 13 shows that, in fall, on average, students of TMTD teachers answered significantly fewer items correctly than did students of comparison teachers (6.4 and 6.8 items, respectively). Had all things been equal, students of TMTD and comparison teachers should have improved at a similar rate by spring, with students in both groups scoring an additional 1.08 items correctly. However, students of TMTD teachers answered 7.95 items correctly (0.45 items more than expected). This increased rate of change was significant (p<.05) and allowed the students of TMTD teachers to finish the school year at a skill level similar to that of their peers in the classrooms of comparison teachers despite having started the year behind them. Average ELQA Receptive Vocabulary Scores Figure 13 Change in ELQA Receptive Vocabulary Scores 10 8 Treatment Effect = 0.45 points 6 4 2 0 Fall 2013 Spring 2014 Students of TMTD Teachers Actual 6.41 7.95 Students of TMTD Teachers Expected 6.41 7.49 Students of Comparison Teachers 6.81 7.89 The results of the DD regression for each ELQA subtest are provided in Table TA-5, in the Technical Appendix. Evaluators also ran the above DD regression models controlling for student and teacher characteristics and found similar trends. The significance of the findings remained the same for each subtest. However, the R-squared value for each subtest regression was higher in most cases, which demonstrated that the addition of student and teacher characteristics to the model explained a greater proportion of the variance in student scores, while not diminishing the treatment effects of the TMTD professional development. See Table TA-6 in the Technical Appendix for the regression coefficients. Neuhaus Education Center: Teachers Make the Difference 37 Effect Sizes Effect sizes are a way of quantifying the differences between two groups. Here we use the standard deviation as a metric. Using TMTD as an interpretation example, when there is no difference between groups (an effect size of 0.0), it means that 50 percent of the students of comparison group teachers score below the average student of the TMTD teachers; for TMTD to improve scores by one standard deviation (an effect size of 1.0), means 84 percent of the students of comparison group teachers score below the average student of the TMTD teachers (Coe, 2002). How do effect sizes for TMTD compare to other interventions? Effect sizes (Hedges g) for the change in scores from fall and spring between students of TMTD and comparison teachers and for code-focused, early literacy interventions with children aged birth thru five derived from the National Early Literacy Panel Report (NELP), are shown in Table 19 (National Center for Family Literacy, 2008). Table 19 Effect Sizes (ES) for Early Literacy Interventions Measure Change in Scores from Fall to Spring of Students of TMTD and Comparison Teachers NELP* Fixed Print Concepts** 1.05 0.44 Picture Naming* 0.10 0.35 Rhyme 0.41 Letters: Uppercase 0.38 Letters: Lowercase 0.31 Letter Sounds 0.56 Receptive Vocabulary* 0.23 PreK Low SES Urban 0.40 0.31 0.67 0.40 0.29 0.27 0.26 0.26 0.81 * NELP calculated effect sizes on posttest scores in studies that established baseline equivalence. Since the students of TMTD and comparison teachers were not equivalent at pretest, effect sizes from the change in scores from fall to spring are used instead. **NELP measures included Print Knowledge, Rapid Automatic Naming, Alphabet Knowledge, and Oral Language, respectively. TMTD had larger or comparable effect sizes compared to interventions in the NELP report. In the area of Print Concepts, students of TMTD teachers outperformed those in the NELP studies (effect sizes of 1.05 and 0.44, respectively). Students of TMTD teachers did not do as well in the area of Picture Naming where the effect size was considerably smaller, although these two measures might not be similar. According to Cohen (1988), medium effect sizes were found for the students of TMTD teachers in the areas of Rhyme and Letter Sounds, but there are no comparison effect sizes for these measures in the NELP report. Effect sizes, especially those from studies with Low SES children, between students of TMTD teachers and those in the NELP studies were comparable in the areas of Letters (0.35 and 0.40, respectively) and Receptive Vocabulary (0.23 and 0.26, respectively). 38 Education Northwest Differences Based on TMTD Coaching Intensity The above comparisons were between all students of TMTD teachers, regardless of coaching intensity, and the students of comparison teachers. Neuhaus provided two levels of coaching to participating teachers: high intensity and low intensity. To determine if there were differences in the effects of the TMTD coaching conditions, evaluators again used the DD regression analysis to determine if there were differential rates of change across students of High and Low Intensity teachers. Table 20 displays the results of these analyses, and shows that, in most cases, receiving higher intensity (i.e., more frequent) coaching did not result in any statistically significant differences in student outcomes. For example, in Print Concepts, the score for students of High Intensity teachers improved by 0.117 points more than expected. This was not significantly different from the average score for students of Low Intensity teachers. Students of High Intensity teachers had a lower rate of change than expected in comparison with the Low Intensity teachers in two areas: • Rhyming: The average rate of change for the students of High Intensity teachers was 1.3 points lower than that of students of Low Intensity teachers (p<.01) • Letter Sounds: The average rate of change for students of High Intensity teachers was 2.4 points lower than that of students of Low Intensity teachers (p<.01) Table 20 Differences in Rate of Change in Scores of Students of High Intensity and Low Intensity Teachers Average Student Scores Subtest Print Concepts Picture Naming Rhyming Letters: Uppercase Letters: Lowercase Letter Sounds Receptive Vocabulary TMTD Teachers High Intensity (n=338) Low Intensity (n=292) High Intensity (n=338) Low Intensity (n=292) High Intensity (n=338) Low Intensity (n=292) High Intensity (n=338) Low Intensity (n=292) High Intensity (n=338) Low Intensity (n=292) High Intensity (n=338) Low Intensity (n=292) High Intensity (n=338) Low Intensity (n=292) Fall 2013 3.17 3.00 18.96 17.85 3.51 2.68 10.77 11.46 6.43 6.96 3.59 3.58 6.49 6.34 Spring 2014 7.97 7.70 21.81 21.14 6.78 7.23 23.20 24.27 20.91 22.23 17.38 19.74 7.91 8.01 Difference from Expected Rate of Change 0.117 0.0 -0.374 0.0 -1.270* 0.0 -0.337 0.0 -0.761 0.0 -2.355* 0.0 -0.226 0.0 * p<0.01 Neuhaus Education Center: Teachers Make the Difference 39 The results of the DD regression for each ELQA subtest are provided in Table TA-7, in the Technical Appendix. Evaluators also ran the above DD regression models, controlling for student and teacher characteristics, and found similar trends. The significance of the findings remained the same for each subtest. However, the R-squared value for each subtest regression was higher, which demonstrated that the addition of student and teacher characteristics to the model explained a greater proportion of the variance in student scores, while not diminishing the treatment effects of the TMTD professional development. See Table TA-8, in the Technical Appendix, for the regression coefficients. 40 Education Northwest Discussion and Recommendations Discussion Neuhaus Education Center (Neuhaus) articulated a theory of action wherein student literacy outcomes would improve if teachers had deep knowledge of literacy and instruction, and used that knowledge along with TMTD supplemental instructional materials in their classrooms. To accomplish this, the logic model specified that Neuhaus would provide prekindergarten (PreK) teachers ongoing professional development in the form of Teachers Make the Difference (TMTD) workshops and coaching. Neuhaus was largely successful in their efforts. They provided, and virtually all teachers attended, six days of workshops dispersed across the 2013–2014 school year. Teachers were pleased with the workshops and felt the time spent on a variety of early childhood literacy topics was “just right.” In addition, all teachers received coaching, although the amount of coaching Neuhaus anticipated providing was not delivered. While High Intensity teachers received more coaching than Low Intensity teachers, they only received 59 percent of their anticipated coaching sessions; Low Intensity teachers received 70 percent of their anticipated coaching sessions. There were few other differences in the coaching provided. Both the High and Low Intensity teachers received coaching that included observation and demonstration, and that focused mostly on the early literacy areas of oral language, letter recognition, and phonological awareness, and assessing student understanding. Teachers gained knowledge of early literacy concepts, as evidenced by increased scores on the Neuhaus Teacher Knowledge Survey in spring. Teachers also self-reported using the TMTD materials in their classrooms—at least weekly, with their Tier 3 students, and/or during smalland whole-group instruction. However, there were few reported differences between TMTD and comparison teachers regarding classroom instruction. TMTD and comparison teachers’ reported having similar skill levels for using instructional practices for building skills in oral language, letter recognition, and concepts of print with their Tier 1 and Tier 2/Tier 3 students and in building skills in phonological awareness with their Tier 2/Tier 3 students. However, while Neuhaus focused on using TMTD materials with Tier 2/Tier 3 students, TMTD teachers, reported significantly lower skills when using instructional practices with their Tier 2/3 students than with Tier 1 students. Finally, survey data suggest that TMTD and comparison teachers engaged similarly in family literacy activities and communication. There appears to be a disconnect: teachers received the professional development and coaching and used the materials in their classrooms. While there were few differences in reported practice between TMTD and comparison teachers, the students of TMTD teachers improved their skills at a faster rate than did students of comparison teachers, to the extent that while the students of TMTD teachers were significantly behind their peers in almost all the ELQA subtest measures in fall, they were at similar skill levels or significantly ahead of their peers in the classrooms of comparison teachers on those same measures in spring. Furthermore, TMTD had Neuhaus Education Center: Teachers Make the Difference 41 a larger effect size in Print Concepts and similar effect sizes in Upper and Lower Case Letters and Receptive Vocabulary compared to interventions in the NELP report. A limitation of this evaluation is that implementation of teacher practices was analyzed using self-reported data for both TMTD and comparison teachers. In the absence of calibrated observation of teacher practices, it is difficult to know if TMTD and comparison teachers actually implemented similar practices, but obtained different student outcomes, or if comparison teachers’ perceptions of their own practices were calibrated to a different standard, thereby confounding any true differences in instructional practice. Due to the differences in outcomes between TMTD and comparison teachers’ students, the latter seems possible. The Neuhaus logic model did not specify an ideal amount of coaching, but Neuhaus added higher intensity coaching to test assumptions about how much coaching was needed. While the addition of variation in levels of coaching did not result in meaningful differences, implementation barriers to coaching occurred, and we suggest the lack of differences in the findings in coaching intensity be taken with caution. Low Intensity teachers were more likely to receive side-by-side and shadow coaching and coaching focused on oral language in the specific areas of vocabulary development, naming, academic vocabulary, and multiple meanings. However, these areas do not align to the areas in which students of Low Intensity teachers improved at a higher rate than the students of High Intensity teachers—rhyming and letter sounds. Furthermore, it is unlikely that the increased use of the two coaching techniques—sideby-side and shadow coaching – while reportedly high impact, could have been responsible for these student’s increased growth in these areas. This evaluation should not be read to suggest that more coaching is not helpful. The lack of differences may warrant further examination of coaching intensity in the future, preferably combined with a more intensive examination of implementation practices that delves deeper than what can be obtained by self-reported data. The evaluation identified one additional area of concern. Coaches reported working on vocabulary development during many of their sessions, anecdotally suggesting this was an area of student need. Therefore, the higher initial performance of students on these subtests suggests two possible scenarios. A first possibility is that the subtest gave a true assessment of student performance in vocabulary, and coaches misidentified the need to focus on vocabulary. A second possibility is that the subtest gave a false positive (i.e., was too easy for the students), and the coaches accurately identified student vocabulary needs using other means. Given that the majority of the students included in the sample were economically disadvantaged, the first option seems unlikely. Hart & Risley (1995) found that students from economically disadvantaged backgrounds tend to have significantly delayed vocabulary development. Therefore, given the lower performance of the students on the other ELQA subtests, the latter possibility seems most likely. 42 Education Northwest Recommendations: We offer Neuhaus the following recommendations: 1. Further examine the reliability and validity of the ELQA. If Neuhaus wants TMTD teachers to use ELQA data to make decisions about grouping, differentiating instruction, and providing interventions, they have to be sure that the measures those decisions are being based on are trustworthy. 2. Offer coaching at one intensity level. Select a frequency that is cost effective, aligned to the HISD and TMTD calendars (i.e., to take into consideration assessment dates), and easily maintained by Neuhaus coaches. 3. Some TMTD teachers reported insufficient training in five areas. The amount of time devoted to these in the workshops and/or coaching might be increased: o Differentiating instruction o Organizing interventions into three tiers o Using data o Learning about parents as partners o Learning about the Neuhaus coaching model 4. Increase coaches’ use of demonstration lessons and side-by-side and shadow coaching as the vast majority of teachers (more than 85%) who received those coaching techniques reported it “Somewhat” or “Substantially” impacted their instruction. 5. Consider providing additional coaching in the area of differentiating instruction. This is one area where TMTD teachers reported their skills in using instructional practices were lower with Tier 2/Tier 3 students than with Tier 1 students. 6. In continued evaluations of TMTD, include more objective measures of teacher implementation in TMTD and comparison teacher classrooms. For example, implement a systematic observation process grounded in standard practices that are expected of each participant. The survey items in the Education Northwest Teacher Survey were not able to distinguish differences in implementation, even when student outcomes suggest teacher practices were likely different. Neuhaus Education Center: Teachers Make the Difference 43 44 Education Northwest Technical Appendix School and Teacher Assignment Neuhaus offered workshops and follow-up coaching to schools within HISD. In summer 2013, 67 schools accepted and were randomly assigned to receive high or low intensity coaching. Schools were assigned to conditions to reduce the chance of cross-contamination of coaching models. Teachers in schools that elected to participate were offered an opportunity for the training and coaching. Teachers in schools in the first group, “High Intensity,” received weekly coaching, whereas those in the second group, “Low Intensity,” received monthly coaching, as shown in Table TA-1. Table TA-1 Coaching Conditions for TMTD Schools High Intensity Schools Low Intensity Schools Barrick* Kashmere Gardens Anderson Janowski* Bellfort* Kelso* Ashford Kate Bell* Benavidez Lockhart Bonham Montgomery Bonner* Lyons* Browning Neff* Braeburn* Martin Luther King Jr* C.Martinez* Paige Brookline* McNamara* Codwell Peck* Burbank* Mitchell* Cook* Petersen Cage Ninfa Laurenzo DeAnda Pilgrim Academy Crockett* Oates* DeChaumes* Red Davila Rodriguez* Farias Reynolds* Durham* Shearn* Fonwood* Rucker Emerson Sherman* Franklin* Rusk Garcia* Tijerina Golfcrest Smith* Garden Oaks* Wainwright* Gross* Southmayd* Halpin Whidby* Henderson* Sutton Hobby Woodson* Herrera* Walnut Bend* Isaacs* Young* JR Harris * Schools that participated after random assignment However, by spring, 27 schools (11 in the High Intensity group and 16 in the Low Intensity group) originally assigned to High and Low Intensity coaching conditions either had no eligible teachers or school administrators no longer wanted their teachers to participate. In addition, one school, originally assigned to the High Intensity group, was inadvertently included in the Low Intensity group (Bellfort), and one school, originally assigned to the Low Intensity group, was inadvertently included in the High Intensity group (Herrara). Neuhaus Education Center: Teachers Make the Difference 45 Originally a smaller number of teachers who participated in the training were going to be randomly selected to participate in the study, so as to limit the number of teachers from whom study data were collected; but in actuality, Neuhaus collected study data from all teachers. In fall, out of a total of 100 eligible English-only teachers, 35 teachers in High Intensity schools and 35 teachers in Low Intensity schools were randomly selected to participate in the study. Teachers were not notified about their assignment. During the school year, we systematically replaced study teachers as they dropped from the training or study. Attrition occurred when teachers were moved from English to Spanish instruction classrooms, from general education to special education classrooms, resigned, or decided not to participate in training for other miscellaneous reasons. When a teacher originally assigned to the study group left, the teacher with the next highest randomly generated number in the respective group was selected to take their place. By the end of the year, 34 teachers in the High Intensity group and 34 teachers in the Low Intensity group remained in the study. Attrition is summarized in Table TA-2. Coaching, survey, and student assessment data from these teachers are included in analyses. Table TA-2 Study Attrition at School and Teacher Levels Randomly Assigned High Intensity Low Intensity All TMTD Teachers Summer 2013 33 34 67 Spring 2014 22 18 30 33.0% 47.0% Schools Attrition Differential Attrition 14.1% 44.8% NA Study Teachers Fall 2014 35 35 70 Resigned 2 1 3 SPED 2 3 5 Dual Language 2 4 6 No Pretest Data 0 2 2 Other 2 1 3 Replaced 7 10 17 34 34 68 22.9% 31.4% Spring 2014 Attrition Differential Attrition 46 Education Northwest 8.5% 27.1% NA Finally, Neuhaus and HISD worked to identify 19 schools not participating in TMTD training to serve as a comparison group. Schools in the comparison group were selected through a convenience sampling process. In order to avoid study contamination, no teachers for comparison were teaching in schools that had any other teachers who participated in the TMTD. HISD had focused initial training on schools that were the lowest performing, which left somewhat higher performing schools as the only option. Neuhaus engaged the HISD central office in fall 2013 to identify schools and gain permission to request volunteers for participation in the comparison group. Then, Neuhaus sought the permission of the principal, teachers, and students based on the options given to them. Only those who gave permission were included in the study. No incentives were provided. Neuhaus conducted all data collection on behalf of the comparison group teachers. Student Outcomes To determine if there were differences in student outcomes based on their teachers’ participation in TMTD, evaluators first conducted a simple descriptive analysis of the average ELQA scores of the students of TMTD and comparison teachers. These descriptive averages were calculated for both fall 2013 and spring 2014. A series of t-tests were utilized to compare differences between the student groups for each ELQA subtest. Tables TA-3 and TA-4 reflect the average ELQA scores in fall 2013 and spring 2014. Table TA-3 Average Fall 2013 Student ELQA Scores Students of TMTD Teachers (n=630) Possible Score Avg. # Correct (SD) Avg. % Correct Students of Comparison Teachers (n=262) Avg. # Correct (SD) Avg. % Correct Difference (sig.) Print Concepts 10 3.09 (2.29) 31% 3.70 (2.33) 37% t=3.61 *** Picture Naming 25 18.45 (5.12) 74% 17.39 (4.91) 70% t=-2.85 *** Rhyming 10 3.12 (2.09) 31% 4.11 (2.54) 41% t=6.03 *** Letters: Uppercase 26 11.09 (9.91) 43% 14.23 (9.93) 55% t=4.31 *** Letters: Lowercase 26 6.68 (9.60) 26% 9.91 (10.89) 38% t=4.40 *** Letter Sounds 26 3.59 (6.97) 14% 7.10 (9.11) 27% t=6.25 *** Receptive Vocabulary 10 6.41 (2.18) 64% 6.81 (2.09) 68% t=2.47 *** Total Score (sum) 133 48% n/a 52.43 39% 63.25 *** p < 0.01, ** p < 0.05, * p < 0.1 Neuhaus Education Center: Teachers Make the Difference 47 Table TA-4 Average Spring 2014 Student ELQA Scores Print Concepts Picture Naming Rhyming Letters: Uppercase Letters: Lowercase Letter Sounds Receptive Vocabulary Total Score (sum) Possible Score 10 25 10 Students of Comparison Teachers (n=262) Avg. % Avg. # Correct Correct 5.76 (2.54) 58% 20.75 (3.64) 83% 6.73 (3.11) 67% Difference (sig.) t=-11.99 *** t=-2.56 *** t=-1.13 26 23.69 (5.81) 91% 23.41 (5.33) 90% t=-0.68 26 21.52 (8.07) 83% 21.55 (7.19) 83% t=0.05 26 18.47 (8.71) 71% 17.02 (8.39) 65% t=-2.30 ** 10 7.95 (1.76) 80% 7.89 (1.51) 79% t=-0.52 78% n/a 133 *** p < 0.01, ** p < 0.05, * p < 0.1 48 Education Northwest Students of TMTD Teachers (n=630) Avg. % Avg. # Correct Correct 7.84 (2.30) 78% 21.50 (4.07) 86% 6.99 (3.05) 70% 107.96 81% 103.11 Difference-In-Difference Regression Tables Evaluators utilized a Difference-in-Difference Regression to compare changes in achievement for students of TMTD teachers and students of comparison teachers. Table TA-5 provides the results between students of TMTD teachers (regardless of coaching intensity) and comparison teachers. Table TA-5 does not control for student and teacher characteristics. Table TA-5 Differences between Students of TMTD and Comparison Teachers in the Average Student ELQA Scores N=892 Print Concepts Picture Naming Rhyming Letters: Uppercase Letters: Lowercase Letter Sounds Receptive Vocabulary Total ELQA Score (sum) Difference between average change in the number of Items correct (SE) of students of TMTD and Comparison Teachers Difference between average score in the number of items correct (SE) for students of TMTD and Comparison Teachers, regardless of time period Difference between the average fall and spring scores in the number of items correct (SE) for all students, regardless of group 2.694*** (0.243) -0.605*** (0.172) 2.053*** (0.204) -0.345 (0.470) 1.090*** (0.332) 3.366*** (0.395) 1.238*** (0.279) -0.984*** (0.197) 2.622*** (0.234) 3.409*** (0.840) -3.124*** (0.594) 9.179*** (0.706) 3.190*** (0.933) -3.222*** (0.660) 11.65*** (0.784) 9.908*** 0.380 4.966*** (0.848) -3.512*** (0.599) 9.916*** (0.712) 7.103*** 0.412 0.445** (0.201) -0.380*** (0.142) 1.080*** (0.169) 6.809*** 0.119 15.60*** (3.065) -10.74*** (2.168) 39.86*** (2.576) Slope Intercept (Constant) 3.702*** 17.39*** 4.111*** 14.23*** 63.25*** Rsquared 0.448 0.114 0.306 0.347 0.433 *** p < 0.01, ** p < 0.05, * p < 0.1 Note: Results may be interpreted as follows: Students of TMTD teachers improved by an average of 2.694 (SE=0.243) points more than students of comparison teachers in Print Concepts. When pooling fall and spring scores for each group, students of TMTD teachers had an average score in Print Concepts that was 0.605 (SE=0.172) points less than students of comparison teachers. When pooling all student scores, regardless of group, the average student Print Concepts score improved 2.053 points (SE=0.204). The model accounts for 45 percent of the variance in student scores (R-squared=0.448). Neuhaus Education Center: Teachers Make the Difference 49 Evaluators also ran the above Difference-in-Difference Regression including student and teacher characteristics as covariates and found similar trends. Table TA-6 illustrates the analysis for the rate of change in the number of items correct. Student characteristics included gender, ethnicity, eligibility for Free and Reduced-price Lunch, English Language Learner status, and attendance rate. Teacher characteristics included number of years teaching in district, number of years in the profession, and educational attainment. The overall trends and significance of the findings did not change. However, the student and teacher characteristics allowed the model to account for a greater proportion of variance, which indicated even more favorable results for the TMTD teachers. Table TA-6 Differences between Students of TMTD and Comparison Teachers in the Average Rate of Change in Student ELQA Scores, Controlling for Student and Teacher Characteristics N=892 Print Concepts Picture Naming Rhyming Letters: Uppercase Letters: Lowercase Letter Sounds Receptive Vocabulary Total ELQA Score (sum) Difference Between the Average Change of Students of TMTD and Comparison Teachers Not Controlling for Teacher and Student Controlling for Teacher and Student Characteristics (n=892) Characteristics (n=751) Difference in Slope Difference in Slope number of items Intercept Rnumber of items Intercept Rcorrect (SE) (Constant) squared correct (SE) (Constant) squared 2.694*** (0.243) 3.702*** 0.448 2.334*** (0.265) 1.554 0.481 -0.345 (0.470) 17.39*** 0.114 1.238*** (0.279) 4.111*** 0.306 3.409*** (0.840) 14.23*** 3.190*** (0.933) 4.966*** -0.181 (0.503) 17.23*** 0.184 1.388*** (0.305) 5.662*** 0.332 0.347 4.017*** (0.910) 9.873* 0.398 9.908*** 0.380 3.829*** (1.018) 6.633 0.418 (0.848) 7.103*** 0.412 5.490*** (0.929) 4.494 0.440 0.445** (0.201) 6.809*** 0.119 0.321 (0.219) 7.369*** 0.177 15.60*** (3.065) 63.25*** 0.433 17.20*** (3.318) 52.810*** 0.476 *** p < 0.01, ** p < 0.05, * p < 0.1 Note: Results may be interpreted as follows: When not controlling for student and teacher characteristics, the average change in Print Concepts scores was 2.694 points greater for students of TMTD teachers than for students of comparison teachers. The model accounts for 45 percent (R-squared=0.448) of the variance in student scores. When controlling for student and teacher characteristics, the average change in Print Concepts scores was 2.334 points greater for students of TMTD teachers than for students of comparison teachers. The model accounts for 48percent (R-squared=0.481) of the variance in student scores. 50 Education Northwest Evaluators utilized a Difference-in-Difference Regression to analyze the rate of change in student scores between the students of High and Low Intensity teachers. Table A-7 provides the results between these two groups. Table TA-7 does not control for student and teacher characteristics. Table TA-7 Differences between High and Low Intensity Teachers in the Average Student ELQA Scores Difference between average score in the number of items correct (SE) for students of High and Low Intensity Teachers, regardless of time period Difference between the average fall and spring scores in the number of items correct (SE) for all students, regardless of group Slope Intercept (Constant) Rsquared (0.259) 0.155 (0.183) 4.685*** (0.190) 3.014*** 0.519 (0.518) (0.293) 1.044*** 0.819*** (0.366) (0.207) 3.223*** 4.542*** (0.379) (0.215) 17.91*** 2.687*** 0.106 0.364 (0.917) -0.732 (0.649) 12.77*** (0.672) 11.50*** 0.378 (1.002) (0.887) -0.551 0.000651 (0.709) (0.628) 15.24*** 16.15*** (0.734) (0.650) 6.983*** 3.591*** 0.414 0.477 (0.223) 0.127 (0.158) 1.646*** (0.164) 6.361*** 0.131 (3.283) 0.863 (2.322) 58.25*** (2.405) 52.05*** 0.479 Difference between average change in the number of Items correct (SE) for students of High and Low Intensity Teachers N=892 Print 0.117 Concepts Picture -0.374 Naming Rhyming -1.270*** Letters: -0.337 Uppercase Letters: -0.761 Lowercase Letter Sounds -2.355*** Receptive -0.226 Vocabulary Total ELQA -5.206 Score (sum) *** p < 0.01, ** p < 0.05, * p < 0.1 Note: Results may be interpreted as follows: Students of High Intensity teachers improved by an average of 0.117 (SE=0.259) points more than students of Low Intensity teachers in Print Concepts. When pooling fall and spring scores for each group, students of High Intensity teachers had an average score in Print Concepts that was 0.155 (SE=0.183) points more than students of Low Intensity teachers. When pooling all student scores, regardless of which coaching group, the average student Print Concepts score improved 4.685 points (SE=0.190). The model accounts for 52 percent of the variance in student scores (Rsquared=0.519). Evaluators also ran the above Difference-in-Difference Regression including student and teacher characteristics as covariates and found similar trends. Table TA-8 illustrates the analysis for the rate of change in the number of items correct. Student characteristics included gender, ethnicity, eligibility for Free and Reduced-price Lunch, English Language Learner status, and attendance rate. Teacher characteristics included number of years teaching in district, number of years in the profession, and educational attainment. The overall trends and significance of the findings did not change. However, the student and teacher characteristics allowed the model to account for a greater proportion of variance, which indicated even more favorable results for the Low Intensity teachers. Neuhaus Education Center: Teachers Make the Difference 51 Table TA-8 Differences between High and Low Intensity Teachers in the Average Student ELQA Scores, Controlling for Student and Teacher Characteristics N=892 Print Concepts Picture Naming Rhyming Letters: Uppercase Letters: Lowercase Letter Sounds Receptive Vocabulary Total ELQA Score (sum) Difference Between the Average Change of Students in the High and Low Intensity Groups Not Controlling for Teacher and Student Controlling for Teacher and Student Characteristics (n=630) Characteristics (n=545) Difference in Slope Difference in Slope number of items Intercept Rnumber of items Intercept Rcorrect (SE) (Constant) squared correct (SE) (Constant) squared 0.117 -0.259 3.014*** 0.519 0.315 (0.276) 1.996 0.540 -0.374 -0.518 17.91*** 0.106 -0.481 (0.552) 16.59*** 0.166 -1.270*** -0.293 2.687*** 0.364 -1.130*** (0.317) 2.268 0.374 -0.337 -0.917 11.50*** 0.378 -1.007 (0.972) 5.680 0.423 -0.761 -1.002 6.983*** 0.414 -1.062 (1.068) 1.718 0.447 -2.355*** -0.887 3.591*** 0.477 -2.370** (0.946) 3.919 0.502 -0.226 -0.223 6.361*** 0.131 -0.130 (0.240) 6.869*** 0.165 -5.206 -3.283 52.05*** 0.479 -5.864* (3.486) 39.04* 0.513 *** p < 0.01, ** p < 0.05, * p < 0.1 Note: Results may be interpreted as follows: When not controlling for student and teacher characteristics, the average change in Print Concepts scores was 0.117 points greater for students of High Intensity teachers than for students of Low Intensity teachers. The model accounts for 52 percent (R-squared=0.519) of the variance in student scores. When controlling for student and teacher characteristics, the average change in Print Concepts scores was 0.315 points greater for students of High Intensity teachers than for students of Low Intensity teachers. The model accounts for 54 percent (R-squared=0.540) of the variance in student scores. 52 Education Northwest Appendix A: Teachers Make the Difference Spring 2014 Survey NEUHAUS EDUCATION CENTER Teachers Make the Difference Survey Spring 2014 You received this survey because you are participating in a study of the effectiveness of the professional development program, Teachers Make the Difference (TMTD), offered through the Neuhaus Education Center (Neuhaus). Neuhaus contracted with program evaluators at Education Northwest (in Portland, Oregon) to conduct the study. Evaluators designed this survey to obtain your feedback on the project, to gain insight into the experiences of the teachers who participated in TMTD workshops and coaching, and to ascertain any differences in the teaching practices of teachers in the treatment and comparison groups. Thank your for participating. Your feedback is very important to us. SECTION 1: DEMOGRAPHICS AND BACKGROUND At what campus do you teach (select one)?*  Franklin  Ashford  Montgomery  Barrick  Garcia  Neff  Bell  Garden Oaks  Oates  Bellfort ECC  Gross  Peck  Bonham  Halpin  Reynolds  Bonner  Henderson JP  Rodriguez  Braeburn  Herrera  Rucker  Brookline  Hobby  Shearn  Burbank  Isaacs  Sherman  C. Martinez  Janowski  Smith  Cook  Jefferson  Southmayd  Crockett  JR Harris  Sutton  Davila  Kelso  Tijerina  DeAnda  King  Wainwright  DeChaumes  Lockhart  Walnut Bend  Durham  Lyons  Whidby  Emerson  McNamara  Woodson  Fonwood  Mitchell  Young Neuhaus Teachers Make the Difference Spring 2014 Survey 53 What is your name?* We only ask this to know who has not completed the survey, so we can follow-up with them. Your name will not be attached to your responses in any way. How many years. . . Have you been teaching? Have you been teaching in Houston Independent School District? Have you been teaching at your current campus? Have you been teaching pre-kindergarten? __________ __________ __________ __________ What is the highest education degree you possess?  4-year degree  More than 4-year degree  Alternative certification (e.g., Teach for America) What certifications do you have? (select all that apply)  Early Childhood Education  Reading specialist  Special Education  ESL  Bilingual education  Other: _________________________________________________ What is your gender?  Female  Male What is your ethnic group?:  African American/Black  Alaska Native/American Indian  Asian/Pacific Islander  Caucasian  Hispanic/Latino(a)  Multi-racial  Other What percentage of your children are in the following instructional tiers? Tier 1 (have all earning needs met with core classroom instruction) Tier 2 (have learning needs me through core and supplemental instruction) Tier 3 (need intensive intervention beyond core and supplemental instruction Total (must add up to 100%) 54 Education Northwest _______% _______% _______% _______% SECTION 2: PARTICIPATING IN PROFESSIONAL DEVELOPMENT WORKSHOPS In which professional development workshops did you participate? th th Day 1, September 10 -12 th Day 2, October 8 -10th th Day 3, November 12 -14th th Day 4, January 14 -16th Day 5, March 11-13th Yes      No     If no, why didn’t you participate in one or more of the professional development workshop(s)? Overall, the professional development at the Neuhaus workshops was: Of high quality Highly useful Effective in moving my practice forward Ongoing and sustained Supported by my principal Aligned with other professional development I received this year Flexible to meet the needs of participants Strongly Disagree      Disagree Agree Strongly Agree                        Please indicate the extent to which the following were provided in an appropriate amount (select one). Too little Just Right  Too Much  The amount of detail provided in the handouts  The amount of detail provided in the presentations    The time spent learning about concepts of print    The time spent learning about letter recognition    The time spent learning about oral language awareness    The time spent learning about phonological                      The time spent learning about the development stages of language and literacy in children The time spent learning about differentiating instruction to meet the needs of a diverse group of learners The time spent learning about how to organize interventions into three tiers The time spent learning about how to use data to improve instruction The time spent learning about parents as partners in the development of oral language and early literacy The time spent learning about the Early Literacy Quick Assessment (ELQA) Neuhaus Teachers Make the Difference Spring 2014 Survey 55 Indicate the extent to which the following were provided in an appropriate amount Too little Just Right Too Much The time spent learning about the Neuhaus Language and Literacy Units    The time spent learning about the Texas prekindergarten Guidelines    The time spent on collaborative problem solving with small groups of teachers    The time spent learning about the Neuhaus coaching model    What was one aspect of the large-group professional development workshops that was most effective in moving your practice forward? Do you have any additional comments about the large-group professional development workshops? SECTION 3: PARTICIPATING IN COACHING Which of the following best describes how often your Neuhaus coach visited you in your classroom (select one)?*  5 or more times per month (more than once a week)  4 times per month (once a week)  3 times per month  2 times per month  Once a month  Less than once a month (e.g., 4 to 6 times during the year)  2 to 3 times during the year  Once during the year  Never If no, why didn't you participate in coaching? Coach Visit Frequency and Length The frequency of these visits was (select one)…  Too little  Just right  Too much Per visit and on average, how many minutes did your coach visit with you? (enter a whole number between 0 and 480.) _______________________________________ 56 Education Northwest The length of these visits was (select one)…  Too little  Just right  Too much Coaching Frequency and Impact The following are different activities your coach might have engaged in while visiting your classroom. Please indicate the frequency with which each activity occurred with your Neuhaus coach, using the following scale on the pull-down menu: • Never • Occasionally (less than every other visit) • Sometimes (every other visit) • Regularly (almost every visit) • Always (every visit) Also indicate the extent to which the activity positively impacted your instruction, using the following scale on the pull-down menu: • Not at all • Minimally • Somewhat • Substantially Occasionally Sometimes Regularly Always Not at all Minimally Somewhat Substantially Impact Never Frequency Develop a customized plan of action          Observe instruction          Observe students learning          Observe student engagement          Demonstrate lessons          Conduct side-by-side coaching          Conduct shadow coaching          Help differentiate instruction          Help interpret student assessment data          Neuhaus Teachers Make the Difference Spring 2014 Survey 57 Comments on Coaching What was one aspect of the coaching that was most effective in moving your practice forward? Do you have any additional comments about the Neuhaus coaching? SECTION 4: USE OF NEUHAUS MATERIALS Think about the four Language and Literacy Units you received from Neuhaus (The Kitchen; The Farm; People, People Everywhere; and Me and the World Around Me). On average, how frequently did you use the Neuhaus materials (select one)?*  Daily  A few times a week  Once a week  Once every other week  Once a month  Less than once a month  Never Why didn't you use the Neuhaus Language and Literacy Units? Use of Neuhaus Materials If “Daily,” on average, how many minutes did you devote to Neuhaus lessons per day? _________________________________________________ If “A few times a week” or “Once a week,” on average, how many minutes did you devote to Neuhaus lessons per week? _________________________________________________ If, “Once every other week” or “Once a month,” on average, how many minutes did you devote to Neuhaus lessons per month? _________________________________________________ If “Less than once a month,” on average, how many minutes did you devote to Neuhaus lessons this school year? _________________________________________________ Using Neuhaus Materials Please rank the instructional periods during which you primarily used the Neuhaus materials ("1" should be the most frequent period). ________Tier 1 (core classroom instruction for all children) ________Tier 2 (supplemental instruction for at-risk or struggling children) ________Tier 3 (intensive instruction for children with persistent difficulties) 58 Education Northwest Please rank the instructional modes during which you primarily used the Neuhaus materials ("1" should be the most frequent mode)? ________Whole-group instruction ________Small-group instruction ________One-on-one instruction Please complete the following sentence. I used the Neuhaus materials (select one)…  instead of Frog Street.  as a supplement to Frog Street. Think about the different parts of the Neuhaus Language and Literacy Units. They include different types of resources (i.e., lessons, extension activities, and recommended books). For each unit, how many of each type of resource did you typically use? Many Most None Some Many Most None Some Many Most Recommended Books Some Extension Activities None Lessons The Kitchen             The Farm             People, People Everywhere             Me and the World Around me             To what extent did the Neuhaus materials provide additional instructional support for children in regard to the following topics (select one)? Not at All Minimal Somewhat Substantial Oral language     Phonological awareness     Letter recognition     Concepts of print     Do you have any additional comments about the Neuhaus materials? Neuhaus Teachers Make the Difference Spring 2014 Survey 59 SECTION 5: CLASSROOM PRACTICE Think about the oral language skills that preschool children develop over time. To what extent do you provide instructional activities for the children in your classroom to build the following skills? Please respond separately for your Tier 1 and Tier2/3 children. Select from the drop down menu which uses the following scale: 1 - I don't do this 2 - I do this, but could use more professional development (PD)/resources 3 - I do this well Tier 1 Children Tier 2/3 Children 1-Don't do this 2-More PD 3-Do this well 1-Don't do this 2-More PD 3-Do this well Responding to read-alouds (e.g., books, poems, nursery rhymes) in ways that demonstrate understanding of what has been read       Retelling or reenacting a story after it is read aloud       Asking/answering appropriate questions about a book read aloud       60 Education Northwest Phonological Awareness Think about the phonological awareness skills that preschool children develop over time. To what extent do you provide instructional activities for the children in your classroom to build the following skills? Select from the drop down menu which uses the following scale: 1 - I don't do this 2 - I do this, but could use more professional development (PD)/resources 3 - I do this well Tier 1 Children Tier 2/3 Children 1-Don't do this 2-More PD 3-Do this well 1-Don't do this 2-More PD 3-Do this well Recognizing rhyme       Recognizing alliteration       Separating a normally spoken sentence into individual words       Combining words to make compound words       Removing a word from a compound word       Combining syllables into a word       Deleting syllables from a word       Producing a word that rhymes with a given word       Producing words that begin with the same sound as other words (onset)       Combining onset and rime to form familiar one-syllable words without pictorial support       Blending two phonemes into real words with pictorial support       Neuhaus Teachers Make the Difference Spring 2014 Survey 61 Letter Knowledge Think about the letter knowledge skills that preschool children develop over time. To what extent do you provide instructional activities for the children in your classroom to build the following skills? Select from the drop down menu which uses the following scale: 1 - I don't do this 2 - I do this, but could use more professional development (PD)/resources 3 - I do this well Tier 1 Children Tier 2/3 Children 1-Don't do this 2-More PD 3-Do this well 1-Don't do this 2-More PD 3-Do this well Naming at least 20 uppercase letters       Naming at least 20 lowercase letters       Producing at least 10 letter sounds       Recognizing at least 20 letter/sound correspondences       Blending two phonemes into real words with pictorial support       Concepts of Print Think about the concepts of print skills that preschool children develop over time. To what extent do you provide instructional activities for the children in your classroom to build the following skills? Select from the drop down menu which uses the following scale: 1 - I don't do this 2 - I do this, but could use more professional development (PD)/resources 3 - I do this well Tier 1 Children Tier 2/3 Children 1-Don't do this 2-More PD 3-Do this well 1-Don't do this 2-More PD 3-Do this well Identifying the front and end of a book       Knowing that spoken words can be represented with printed words       Identifying words printed on a page       Identifying where a printed word begins and ends       62 Education Northwest Think about the oral language, phonological awareness, letter knowledge, and concepts of print skills mentioned in the last few questions. What challenges do you have developing them in your children? What professional development do you have access to that supports you as an early literacy teacher? To what extent do you agree with the following statement (select one)? When I have identified an instructional need for a child or a group of children, I know where to go to get supplemental materials. Strongly disagree Disagree Agree Strongly agree     SECTION 6: PARENTAL OR FAMILY INVOLVEMENT To what extent do you believe that the parents/families of your children are capable of improving the early language and literacy skills of their children (select one)?  Not at all  Very little  Somewhat  To a great extent Parents and Oral Language Please answer the following questions thinking about oral language skills (e.g., vocabulary, word knowledge, etc.). To what extent do you agree with the following statements (select one)? Strongly disagree Disagree Agree Strongly agree I know what types of things parents/families can do to support the oral language skills of their children.     I put a significant amount of my time into communicating with parents/families about how to support the oral language skills of their children.     Neuhaus Teachers Make the Difference Spring 2014 Survey 63 On a scale of 1 to 5 with “1” being low and “5” being high, to what extent do you work with the parents of the children in your classroom to involve them in oral language development activities at home with their child (select one)? 1 2 3 4 5 I have done nothing/little to engage the parents/families of the children in my classroom in meaningful oral language activities at home. I have a repertoire of strategies that I regularly use to engage the parents/families of the children in my classroom in meaningful oral language activities at home. What are three strategies that you recommend parents/families use to develop the oral language skills of their children? Parents & Early Literacy Please answer the following questions thinking about early literacy skills (phonological awareness, letter sounds, etc.). To what extent do you agree with the following statements (select one)? Strongly disagree Disagree Agree Strongly agree I know what types of things parents/families can do to support the early literacy skills of their children.     I put a significant amount of my time into communicating with parents/families about how to support the early literacy skills of their children.     On a scale of 1 to 5 with “1” being low and “5” being high, to what extent do you work with the parents of the children in your classroom to involve them in early literacy development activities at home with their child (select one) 1 I have done nothing/little to engage the parents/families of the children in my classroom in meaningful early literacy activities at home. 2 3 4 5 I have a repertoire of strategies that I regularly use to engage the parents/families of the children in my classroom in meaningful early literacy activities at home. What are three strategies that you recommend parents/families use to develop the early literacy skills of their children? 64 Education Northwest Communicating Ideas When thinking overall about all of the strategies you recommend to parents/families for how to support either oral language or early literacy at home, please answer the following questions. How do you communicate ideas to support oral language or early literacy at home to the parents/families of your students? Which of the following best describes how often you communicate these ideas to the parents/families of your children (select one)?  5 or more times per month (more than once a week)  4 times per month (once a week)  3 times per month  2 times per month  Once a month  Less than once a month (e.g., 4-6 times during the year)  2-3 times during the year  Once during the year  Never What challenges do you have communicating with parents about their child’s development? Thank You! Neuhaus Teachers Make the Difference Spring 2014 Survey 65 66 Education Northwest Appendix B: Neuhaus Education Center Teachers Make the Difference 2013– 2014 Coaching Log Neuhaus Education Center Teachers Make the Difference 2013–2014 Coaching Log 1. What is your name?* 2. What is the name of the campus where the teacher you coached works?* 3. What is the name of the teacher you coached?* 4. What was the date of coaching?*  The teacher was absent on the date of coaching. 5. How long did you coach this teacher during this visit, in minutes?* 6. What coaching technique(s) did you use (mark all that apply)?*  Side-by-side  Shadow  Demonstration  Observation 7. What topics did you cover during your coaching visit (mark all that apply)?*  Phonological awareness  Letter recognition  Grapheme-phoneme correspondence  Oral language  Emergent writing  Teacher questioning strategies (teaching metacognition)  Concepts of print  Checking for student understanding  Other: 8. What phonological awareness topics were addressed (mark all that apply)?*  Rhyming  Segmenting syllables  Onset and rime  Phonemic awareness (blending, segmenting, and/or manipulating phonemes)  Other: Neuhaus Teachers Make the Difference Spring 2014 Survey 67 9. What oral language topics were addressed (mark all that apply)?*  Academic vocabulary  Naming  Describing  Multiple meanings  Vocabulary development  Listening comprehension  Read alouds  Story retell  Inferencing  Background knowledge  Other: Any additional comments/notes about this coaching visit? Do you have any more teachers to log?* ( ) Yes, at this campus ( ) Yes, at a different campus ( ) No, I'm done Thank You! 68 Education Northwest References Burns, M. S., Griffin, P., & Snow, C. E. (Eds.). (1999). Starting out right: A guide to promoting children's reading success. Washington, DC: National Academy Press. Coe, R. (2002, September). It’s the effect size, stupid: What effect size is and why it is important. Paper presented at the annual conference of the British Education Research Association, University of Exeter, England. Retrieved from http://www.leeds.ac.uk/educol/documents/00002182.htm Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Hart, B., & Risley, T. R. (1995). Meaningful differences in the everyday experience of young American children. Baltimore, MD: Paul H. Brookes. National Center for Family Literacy, National Early Literacy Panel. (2008). Developing early literacy: Report of the National Early Literacy Panel. A scientific synthesis of early literacy development and implications for intervention. Washington, DC: National Institute for Literacy. Neuhaus Teachers Make the Difference Spring 2014 Survey 69