“Education has a learning problem.”
~ Tony Bryk, PhD
How do we improve practice? How do we know if a particular approach is helping us achieve the outcomes we care about? And how do we scale our learning so that we can impact whole systems, not just individual classrooms? For decades, many of us working in schools or in educational research have been asking these questions. And yet, many would argue that the current educational research infrastructure is not doing enough to improve our nation’s schools (Bryk, Gomez, & Grunow, 2011; Bryk, 2009; Donovan, 2013). Critics of current educational research argue that it is frequently disconnected from practice (Anderson & Shattuck, 2012; Penuel, Fishman, Cheng, & Sabelli, 2011; The Design-Based Research Collective, 2003) and does not scale across contexts (Bryk et al., 2011; Bryk, 2009). For educational research to be more relevant, some scholars argue that it should be conducted by teams of scholars and practitioners, and should focus on improving problems of practice (Bryk et al., 2011; Donovan, 2013; Gutiérrez & Penuel, 2014). In short, it should help us learn how to learn, so that we are working together to continually improve the education we offer to young people. Improvement research is one model that might enable such focus by orienting our collective work around three key questions (Langley et al., 2009): What is our goal? How will we know if we’ve met our goal? What innovations can we introduce into the system that might lead to meeting that goal?
As educators, it is important to understand the system in which we work, and our unique place in it. Donald Berwick, one of the founders of improvement research and past president and CEO of the Institute for Healthcare Improvement, famously noted, “Every system is perfectly designed to achieve exactly the results it gets.” If there are deep inequities and a lack of engagement in our schools, this is not happenstance. It is by design. It is the result of myriad conscious and unconscious choices. To improve education, we need to better understand our system—the mechanisms and assumptions that drive it and that often go unquestioned—as well as the experiences of those working and living within it. Then we need to have the courage to act, to make different choices and to transform the system to get the results we care about most.
Scholars and practitioners have had success using improvement research principles to solve important problems of practice across different disciplines and complex (often dysfunctional) systems, including the auto industry, airline industry, and health care (Smith, Saunders, Stuckhardt, & McGinnis, 2013). There is a new and growing interest in bringing improvement research methodology to bear on long standing problems of practice in education, and in so doing, to rethink and reshape the educational system that produces poor outcomes for too many of our students (Bryk, 2009). Thus far, improvement research has been utilized to improve national community college student performance in developmental mathematics, which has served as a gatekeeper and an obstacle for many students attaining a college degree (Bryk et al., 2011). It has been used to improve the quality and consistency of feedback for new teachers in K-12 districts in Austin and Baltimore (Bryk, 2014; Park, 2014), with the broader hope of better supporting and retaining promising young educators in some of our most disadvantaged communities. And it has been implemented to further develop diverse K-12 students’ academic mindsets so that they feel school is a place where they belong and can be successful (Bryk et al., 2013). In this article, we briefly discuss some of the tools of improvement research and share insights from our initial attempts to improve the college application process for students at High Tech High North County.
While improvement research is relatively new to education, it has a rich theoretical foundation. Morgan (2006) identified multiple metaphors that may be used to better understand organizations. The dominant view of organizations throughout the 20th century was as machines to be optimized. This view was articulated and developed by Frederick the Great of Prussia in the mid 1700s and Frederick Taylor in the early 1900s. This perspective has come to be known as scientific management and includes ideas such as: (a) managers are responsible for thinking while workers are responsible for doing; (b) find the most efficient and precise way to complete a task; (c) select and train workers on the task; and then (d) monitor workers to ensure compliance (Morgan, 2006). Scientific management has been critiqued as creating organizations that are rigid bureaucracies that develop employee apathy and encourage passivity (Morgan, 2006). As a result, other competing theories of organizations developed.
In opposition to viewing organizations as machines to be optimized, a competing metaphor is to view organizations as brains, which is to say flexible, resilient, and inventive learning organizations (Morgan, 2006). One contributor to the theory of learning organizations is the statistician and management consultant Edwards Deming, known for helping to transform Japanese industry in the 1950s through a series of ideas opposed to scientific management (Holt, 1993). In contrast to scientific management principles, where the manager’s job is to monitor for compliance, Deming (1986) dismisses the feasibility and wisdom of such inspection. Instead, he argues that employees should be managed such that they can monitor and inspect their own work (Schmoker & Wilson, 1993). Rather than attempting to improve the final product through inspection at the end, Deming argued for building quality control into the process itself (Holt, 1993) and argued for a process of continuous improvement: “improve constantly and forever” (Deming, 1986, p. 23).
Improvement research draws upon several tools for facilitating this ongoing improvement by people in the system: the driver diagram, the Plan-Do-Study-Act cycle (PDSA), and practical measurement (Langley et al., 2009, p. 119).
Driver diagrams are designed to help improvement scientists (i.e. us) think clearly and systematically about their assumptions and theory of action as they attempt to develop changes that will lead to improvement. Driver diagrams are made up of four elements: aims, drivers, change ideas, and measures.
Practitioners often have hunches about what is contributing to the problem and about the efficacy of particular interventions. The driver diagram pushes practitioners to steer away from silver bullet solutions, to stay focused on a systems approach and to design interventions that support their ultimate goal (Bryk et al., 2011).
The PDSA cycle is a tool for systematically testing iterations towards improvement. In the PDSA cycle, teams (a) create a plan, which includes making specific predictions about what will happen as a result of a change; (b) implement the change in a small scale manner; (c) study the results of the change, including whether the predictions came true or not; and (d) act to refine the change or develop a new one altogether. PDSAs encourage rapid testing and the development of protocols and practices that can be implemented and adapted across diverse contexts.
Yeager et al. (2014) describe two traditional purposes for educational measurement—accountability and theory development. They argue that for practitioners on the ground, a different type of measurement is needed, which they define as “practical measurement.” Practitioners engaged in improvement science require measures that directly measure the target and are designed to work within the constraints of day-to-day educational practice. For example, a survey might need to be answerable in only three minutes and as such must have carefully selected questions that eliminate redundancy and give improvement researchers precise information about their targeted goal (Yeager et al., 2014).
It is worth noting that improvement research challenges existing epistemologies of “What counts as research?” In this way, improvement research is aligned with translational research and action research (Dolle, et al., 2012), both of which are common approaches to educational research. In translational research, information flows down from scholars to practitioners, as theories are developed by scholars and then implemented by practitioners. In action research, in contrast, information flows up. Individual teachers or groups of teachers study local problems of practice in systematic ways. This knowledge may become aggregated to form more general theories. However, because action research is so grounded in a particular context, it has been criticized for failing to work across contexts and thus build scalable knowledge (Dolle, et al., 2012). Dolle et al. argue that improvement research is well situated as a new form of educational research to address the shortcomings of translational research and action research by focusing on improving problems of practice at scale.
High Tech High (HTH) is a group of twelve public charter schools in San Diego County serving approximately 4,900 K-12 students and employing 310 teachers. Since its inception, HTH has aimed to be a school organization that is diverse and integrated, particularly by race, class, gender, and for students with special needs. Orfield (2009) describes how our nation’s K-12 schools have actually become less diverse since Brown v. Board of Education. He laments, “For decades the basic message has been that we do not need to do anything about [problems related to resegregation] and should eliminate the legal and policy tools that we possessed” (p. 32). In contrast, HTH aims to create an intentionally integrated school community with the broader goal of helping to build an integrated society. Since California Proposition 209 forbids schools from using race in their admissions process, HTH uses zip codes as a proxy for race. Due to the unfortunate reality of housing segregation, this ensures that the student population mirrors the diverse communities we serve (Kluver & Rosenstock, 2003).
HTH has long held a strong interest in helping all students have the opportunity to attend college and succeed there. While 98% of HTH graduates go on to college and 75% go directly to four year colleges, a lower percentage of African American and Latino boys go directly to 4 year colleges (estimated to be between 55% – 65%). For this reason, in June of 2014 a team of educators from HTH and the High Tech High Graduate School of Education (HTH GSE) developed an aim to improve the percentage of African American and Latino boys going directly to four year colleges. (A link to the driver diagram guiding this work can be found at the end of this article.) As the quote at the bottom of our driver diagram suggests, our working theory is “definitely incomplete and possibly incorrect.”
Since that time, a number of change ideas have been tested in an effort to achieve this aim. Educators at High Tech Elementary Chula Vista and High Tech High Chula Vista have been working on improving the equity of participation during collaborative group work. Teachers at High Tech Middle have focused on building and sustaining growth mindsets (Dweck, 2007) with 7th graders through goal-setting and explicit teaching about fixed vs. growth mindsets. High Tech High Media Arts worked on a project in the fall to have all 9th grade students pass all of their classes first semester. While the number of students who passed all their courses did not improve, the team learned about the importance of explicitly teaching ninth graders organizational skills. They also recognized that there was more to be done in cultivating African American and Latino male students’ sense of belonging in school, and ultimately in college.To have more students going to college and succeeding there, we need to first graduate more students from high schools who are college and career ready, meaning that they are aware, eligible, and prepared. However, growing research is showing that this alone is not enough; students need to be determined to go—meaning that they see themselves as people who belong in college and who possess the academic mindset to enroll, persevere and ultimately succeed (Farrington et. al., 2012). They also need to actually apply.
To that end, High Tech High North County worked on a project to increase the percentage of seniors applying to four year colleges. Knowing that the percentage of students who start at a community college and graduate from a four-year college is statistically quite low, Isaac, the school director and co-author of this article, decided to focus on the college application process.
He began by applying to two local universities himself and discovered that the process was more confusing than he expected. He then talked to colleagues at other high schools to find out what processes they had in place to help students with the application process. He and colleagues at his school discussed the most promising strategies and put them into place. For example, on a staff day, seniors came in and worked on their college applications alongside their advisor. Teachers also embedded application support into advisory and hosted lunch workshops. As a result, more students applied to four-year colleges than ever before, with an increase from 84% in 2014 to 93% in 2015. After the application process was completed, the school surveyed seniors to find out what was most challenging about the process. Students reported that they were most overwhelmed by the personal statement required by many colleges. To address this issue, all juniors are completing personal statements as part of their humanities class and will have a working draft for next year. (See an illustration of the process above)
One next step is to continue to refine the college application process at HTHNC. Another is to track whether more students applying to four-year colleges actually translates to more students attending as well as succeeding at four-year schools. Another is to disaggregate data from this project to track African American and Latino boys more closely. Another is to share lessons learned across other High Tech Highs as well as other schools around the country. By more closely tracking outcome and process measures of the college application process across our organization, we may be able to help more students, especially underserved populations, attend and succeed in college.
A key aspect of improvement research is the desire to create knowledge that will work across many contexts; in other words, to move beyond idiosyncratic knowledge and take innovations to scale (Silva & White, 2013). This is tricky. Educational innovations often work on a small scale, but lose effectiveness when scaled up (Bryk et al., 2011). What works in one setting may not work in another.
However, by engaging in improvement research with others who are focused on the same aim—in a network improvement community —we are able to understand what leads to variability in performance across contexts. The focus is not simply on whether an innovation works, but on identifying the conditions that enable an innovation to work. By bringing together a diverse group of schools, districts, and organizations to work on a persistent problem of practice, we are able to develop change ideas that work in a variety of settings, better enabling us to scale the work.
One promising practice we have already adapted to share our learning across a network is the “improvement review” (Reinertsen, Pugh, & Nolan, 2003). In an improvement review, a team shares their aim statement, driver diagram, measures, and poses a question to the larger group. The group then uses a structured protocol to ask the team questions, and to discuss their work and potential next steps. The review encourages presenting teams to get organized, complete partially finished work, and move forward with clarity. For other participants, the review facilitates a deeper understanding of, and commitment to, our shared work and the improvement process. We have found that protocols like this support the creation of professional communities of practice focused on teaching and learning, collaboration, and dialogue (Riordan, Caillier & Daly, 2014). These protocols also are critical to fostering a culture of improvement across a network where people embrace the work as part of their professional identity and their organization’s core mission (Russell et. al., submitted).
A basic tenet of improvement research is that we are stronger together than we are alone—and that by working together in systematic ways we can make significant progress on persistent problems of practice. In the past, and in many schools today, educators work in isolation. We hope to change that. If you care about getting all students to college and engaging them in work that matters along the way, consider this an invitation.
To see the driver diagrams guiding the authors’ work visit:
the Google Slides presentation
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16–25.
Bryk, A. S. (2009). Support a science of performance improvement. Phi Delta Kappan, 90(8), 597–600.
Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In M. T. Hallinan (Ed.), Frontiers in Sociology of Education (pp. 127–162). The Netherlands: Springer.
Bryk, A. S., Yeager, D. S., Hausman, H., Muhich, J., Dolle, J., Grunow, A., … Gomez, L. (2013). Improvement research carried out through networked communities: Accelerating learning about practices that support more productive student mindsets. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching.
Bryk, A. S. (2014). Improvement science workshop: Getting ideas into action. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching.
Bryk, A. S., Gomez, L. M., Grunow, A. & LeMahieu, P. (2015). Learning to Improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard Education Press.
Deming, W. E. (1986). Out of the crisis. Cambridge, MA: Massachusetts Institute of Technology.
Donovan, M. S. (2013). Generating improvement through research and development in education systems. Science, 340(6130), 317–319.
Dweck, C. (2007). Mindset: The new psychology of success. New York: Ballentine Books.
Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching Adolescents to Become Learners: The Role of Noncognitive Factors in Shaping School Performance—A Critical Literature Review. Consortium on Chicago School Research. 1313 East 60th Street, Chicago, IL 60637.
Gutiérrez, K. D., & Penuel, W. R. (2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43(1), 19–23.
Holt, M. (1993). The educational consequences of W. Edwards Deming. Phi Delta Kappan, 74(5), 382–388.
Jones, I., Gallagher, R., Daley, B., & Caillier, S. (2015, March 4). Improving four-year college application rates. Presented at the Carnegie Foundation Summit on Improvement in Education.
Kluver, J., & Rosenstock, L. (2003). Choice and diversity: Irreconcilable differences? Principal Leadership, 3(8), 12–18.
Langley, G., Moen, R., Nolan, K., Nolan, T., Norman, C., & Provost, L. (2009). The improvement guide: A practical approach to enhancing organizational performance. San Francisco, CA: Wiley.
Morgan, G. (2006). Images of organization. Thousand Oaks, CA: Sage Publications.
Orfield, G. (2009). Reviving the goal of an integrated society: A 21st century challenge. The Civil Rights Project/Proyecto Derechos Civiles at UCLA. Los Angeles.
Park, S. (2014). BTEN feedback management system. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching.
Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337.
Riordan, R., Caillier, S. & Daly, B. (2014). Developing teachers for deeper learning. In J. Bellanca (Ed.), Deeper Learning: Beyond 21st Century Skills (pp. 137-156). Bloomington: Leading Edge.
Russell, J. L., Bryk, A. S., Dolle, J., Gomez, L. M., LeMahieu, P., & Grunow, A. (submitted). A framework for the initiation of network improvement communities.
Schmoker, M. J., & Wilson, R. B. (1993). Total quality education: Profiles of schools that demonstrate the power of Deming’s management principles. Bloomington, IN: Phi Delta Kappa Educational Foundation.
Smith, M., Saunders, R., Stuckhardt, L., & McGinnis, J. M. (Eds.). (2013). Best care at lower cost: The path to continuously learning health care in America. Washington, DC: National Academies Press.
The Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8.
Yeager, D., Bryk, A., Muchick, J., Hausman, H., & Morales, L. (2014). Practical measurement. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching.
Tags:
