Pennsylvania Institute for Instructional Coaching — A Partnership Between the Annenberg Foundation and the Pennsylvania Department of Education
Evaluation Plan PDF Print E-mail

The incremental roll-out strategy proposed for the Institute offers a unique opportunity to build data-driven decision-making into the day-to-day operations of the Institute. As part of the model, external evaluators will assist in the development and implementation of a continuous improvement framework as well as the design of appropriate data collection strategies and data management structures.

Each year, indicators linked to the objectives of the Institute will be identified, and data will be collected to measure performance against these indicators. This continuous improvement framework will enable the Institute leaders to assess progress toward their objectives, identify areas in which revisions are needed or data suggest needs for further research, and devise program revisions suggested by evaluation data. Elliott Medrich and FHI 360 will jointly design and execute the continuous improvement assessment process, in collaboration with the executive director and Institute partners.

During the first year, Institute evaluation activities will focus specifically on the identification of strategies and processes that contribute to the delivery of highquality training for instructional coaches, mentoring and other types of support for coaches, higher levels of regular one-on-one coaching in participating schools, greater student engagement—and ultimately to improved student outcomes. Questions that might be examined through the evaluation include:

  • What are the best ways to measure how well coaches interact with teachers?
  • Are Institute programs designed to assure the most productive outcomes for those who are coached?
  • How can coaches and mentors increase the number of teachers who participate regularly in one-on-one coaching?
  • What are the different approaches to delivery of mentoring and what are the best ways to measure the contributions and effectiveness of mentoring?
  • What are the best ways to ascertain that coaches are effectively conveying skills to teachers—and that teachers are acquiring and using these skills?
  • What evidence can be used to improve processes for recruitment, selection, training, and assessment of coaches?
  • What delivery vehicles are most effective in the training of coaches and mentors? What changes are needed in the content of this training?
  • What delivery vehicles are most effective for working with school and district leaders to generate support for instructional coaching?

Answers to questions such as these can help the Institute determine the efficacy of its processes and programs from year to year and suggest ways to rethink and recalibrate before new cohorts are added to the roster. Furthermore, assessment for continuous improvement will enable the Institute to begin to build the evidentiary base needed to attract additional schools, districts, and funders.

Once the major evaluation questions are identified for a given year, the evaluation partners will use a variety of assessment strategies to gather the data needed to gauge progress toward key objectives. These might include: field observations of mentors, coaches, teachers, classrooms, and training sessions; surveys; interviews and focus groups; document reviews; and analysis of grades and standardized test scores.

In most instances, qualitative data collection and analysis will be connected to quantitative data collection and analysis in an effort to answer the why and how questions typically suggested by initial tabulations of quantitative data. For example, the evaluators might want to interview teachers to determine why some participated in regular one-on-one coaching and others did not. Observations in the classrooms of the two groups might reveal other differences.

In-depth case studies will be conducted in all schools during the transition year, and cross-case analyses will be completed to identify emerging patterns and trends. During the case study visits, Elliott Medrich and the FHI 360 researchers will collect data on students, teachers, and coaches through interviews, focus groups, and observations. Additional data, primarily on student engagement and achievement, will be obtained from school records and other documents. From the beginning, Elliott Medrich and the FHI 360 research team, in collaboration with the executive director, will build a strong partnership with the schools and districts around the critical need for evidence and its use in program improvement and program expansion across the state.

Back to Research Highlights...