Measuring Your Progress:
The Basics of Phase 4
What is the purpose of this phase?
You and your team will identify the types of measurement and data that best inform and support your improvement work.
When should this happen?
After you run a few initial tests, unless your project can’t proceed without defining measures first.
Who is involved?
The improvement team leader and all team members.
What will you do?
Turn your theory of improvement into measures that gauge your success over time; backward plan to ensure you’ll have the evidence you need to make critical decisions.
You are ready for Phase 4 when you have:
created a problem statement defining the problem that your improvement team will solve
defined an aim statement for your improvement and completed a driver diagram that has allowed you to come up with a theory of improvement for achieving it
identified clear and measurable change ideas that you hypothesize will help you reach your aim
completed 2–3 PDSA tests (aka your first PDSA burst), which is ideal for learning how the PDSA testing process can be supported by strong measures, although some improvement measures may require measures before starting
reflected on your PDSA burst, drawing conclusions and recording learning and its implications for future work
Provide big-picture feedback on your progress. They are a “north star” anchoring your aim and allowing you to assess whether your project is getting results.
Provide feedback on your particular approach to reaching your outcome measure. They should align directly with the primary and secondary drivers on your driver diagram.
“Short-term” PDSA measures
The pieces of “embedded evidence” that are easily available on a day-to-day basis and help you define how your change will occur.
Measuring Your Progress:
Summary of the Concepts
Improvement science offers an approach to data and measurement that does not require complex software or extensive knowledge of statistics. Nor is it concerned with standardized tests or teacher accountability and evaluation. At its heart, measurement is a tool for understanding; the goal of measurement in improvement science is to help educators answer their most pressing questions and guide their decisions as they work toward improvement.
With improvement science, the data we collect and measure exists to guide us as we test our change ideas and analyze the results. We use the data to understand how our progress matches our predictions, who we are helping (and who is being overlooked), and what we still need to do to reach our eventual goal. During Phase 4, your team will develop accurate, reliable tools for collecting evidence of how a change is working. Ensuring your team has the right information will enable them to make strategic adjustments that lead to consistent, sustainable improvement.
Good measurement ensures the data we collect suits the question we are trying to answer and the decisions we make. Improvement science offers a “measurement system” made up of three types of measures: outcome measures, driver measures, and “short-term” PDSA measures. The measurements are intended to be layered onto the driver diagram you created in Phase 2; each measure will map to a different component of the diagram. By aligning the measurements you collect with your working theory of improvement, you will be able to gauge your results.
Visualizing Your Measurement System:
Three Tools for Understanding
1. Outcome Measures
Outcome measures guide your project like a “north star” and are usually defined when the aim is established, and before testing begins.
Outcome measures should be clearly stated and motivating, and it should be obvious to everyone on the team whether or not they have been achieved. They are usually assessed on a big-picture timescale: from yearly to quarterly, depending on the problem.
2. Driver Measures
Driver measures provide feedback on your particular approach to reaching your outcome measure. They should be tied into the specific theory you have settled on for solving your problem. They should be aligned with the drivers on your driver diagram and exist over a range of timescales, from quarterly to monthly or monthly to weekly.
“Long-term” primary drivers track directly with primary drivers identified in Phase 2: Creating a Theory of Improvement. They are usually collected infrequently, quarterly or monthly. Long-term primary drivers ask:
- What outcomes do we expect from our theory when it is working?
“Medium-term” secondary drivers track directly with your secondary drivers; they give you regular feedback on the details of your change. They are assessed more frequently, monthly or biweekly. Medium-term secondary drivers ask:
- Who and what is involved in effecting the change?
- How will those results be achieved?
3. PDSA Measures
“Short-term” PDSA measures get you into the nitty-gritty practical requirements that have to happen to enact your change. Think of short-term PDSA measures as “embedded evidence” or pieces of data easily available on a day-to-day basis within a classroom or school. They are collected on a daily or weekly basis and analyzed after each PDSA test.
Putting It All Together:
Using Your Measurement System
The measures described above will help illuminate your progress. Focus on the information your team feels is the most important. Choose one or two of the measures to collect, knowing you can gradually add measures later as the improvement effort progresses.
We are used to hearing about finished studies with polished data showing clear results. The reality is that identifying and selecting appropriate measures is often messy and incremental. Give your team the freedom to choose and focus on the types of measures that seem to be the most beneficial in a given moment. This will enable your team to eventually build a robust measurement system that authentically fits your needs and moves you toward your aim.
Overview of the Tools
What is it? A structured brainstorming activity to help your team turn their theory of improvement into predictions of how desired changes can be measured.
What does it do? Guides your team through creating a list of embedded evidence that informs the measurement of your progress.
Why use it? It will help you prioritize evidence collection; this evidence will be critical to have on hand when you need to make decisions that lead your team to your aim.
What is it? A planner to help guide your team through:
- selecting the measures needed to collect data;
- the collection process;
- the analysis and interpretation of the results.
What does it do? Ensures your team has the evidence you need to make decisions.
Why use it? To prevent your team from spending too much time on superfluous data work, or collecting the wrong data.