Susan Moore Johnson
Carl H. Pforzheimer, Jr. Professor of Teaching and Learning
Graduate School of Education
I’ve been thinking about the problem you posed in your letter since I received it, hoping that I would find a meaningful way to respond. It’s a fascinating issue that would make for a good book and an intense discussion among any group of researchers. I’m confident that you’ll receive a provocative and informative set of responses. I’m reluctant to list five studies that have made a difference in education because I don’t think that any single study has made a sufficient difference. So, I’ve decided to write about just one key point.
In order for research to have sustained, positive impact on student learning (broadly conceived), it must influence people making decisions at many levels: policymakers at the federal, state, and district levels; and practitioners at the district office, school site, and classroom. Thus, no single study is sufficient to inform all of these people, each of whom has a different piece of the action. Rather, a set of variously designed studies conducted over time would be needed to ensure that these agencies and actors could all work in concert to achieve the best possible outcomes. The research that gets the biggest headlines tends to focus on policy and to report causal findings. However, even the most convincing policy research is of little consequence unless there are additional studies that explain why and how an intervention makes a difference. Otherwise, money and time spent on the intervention will be wasted when school and classroom practices remain unchanged.
A good example is the research on class size. For decades, teachers have reported in surveys and interview studies that smaller classes (15-20 students) enable them to meet the varied needs of their students and to provide more attention and response to their work. However, it wasn’t until the STAR study in the late 1980s that the importance of class size was confirmed in a large-scale randomized experiment. The findings provided ammunition for school administrators and union leaders to recommend funding class-size reductions, which are very expensive. Subsequent reductions in K-3 classes in California in 1996 were correlated with improved student performance, although there were so many other potential explanations for the change, the seeming success of this policy did little to advance the cause of smaller classes. The unintended consequence of this policy, which created a teacher shortage and the hiring of many unlicensed teachers, further complicated this picture. Although the STAR findings suggest that creating smaller classes (13-17 students) in K-3 is worthwhile, it tells us nothing about what to do in the upper grades or how to decide whether to commit limited funds to reduce class size or improve professional development. Moreover, there is little research to inform teachers of smaller classes how to make the best of that opportunity. In fact, many teachers continue to teach as they always have. Therefore, although the STAR study was very influential in establishing the importance of class size, there has been insufficient subsequent research to elaborate on these findings in ways that would inform school districts, schools, and classroom teachers about how to act.
Value-added studies present similar opportunities and limitations. Parents and teachers have long known that some teachers are more effective than others and that this fact has far-reaching implications for their children and students. Sanders’ TVAAS research powerfully demonstrated that teachers do matter and that the impact of these differences have a multiplicative effect for students. Although Sanders’ research methods have been challenged, his general findings have been confirmed by a RAND review. States, districts, and advocacy groups have used these findings in different ways. Probably the most powerful impact has been in pointing to the inequities in teacher assignment patterns. For example, work at Education Trust has shown that first-year teachers tend to be less effective than teachers with 3-5 years of experience, and students in low-income, low-performing schools are more likely to be assigned less effective teachers than students in high-income, high-performing schools. In response to such data, local school districts are currently exploring ways to right this inequity. However, little is yet known about what makes for a more effective teacher (beyond very general findings about SAT scores, experience, and training), what instructional practices are used by more effective teachers, and how less effective teachers might become more effective teachers. Some school districts have jumped quickly to use value-added assessments in a performance-based pay system, an approach for which there is no solid research.
My point in discussing these two examples is that, even in the most methodologically sophisticated, large-scale studies that allow for causal inferences, much more must be learned before they can have a meaningful and sustained impact on the educational experiences of students. Additional studies (large and small, qualitative and quantitative) must be conducted that test, add to, and elaborate on these findings. Most importantly, this research must inform people at all levels about how to effectively implement these findings—what should policymakers and practitioners from the legislature to the classroom do to make the most of what is known? Any discussion of findings should situate a particular study within this larger picture, specifying what is known or not yet known, and suggesting what the next research steps should be.
Researchers realize that any proposed study must be grounded in a discussion of prior, relevant work. However, my sense is that this often is no more than a symbolic gesture to ensure funding. Few studies truly build on prior work. Seldom, in my experience, do researchers join their efforts to examine a problem and proceed to systematically study it piece by piece (using a variety of methods and disciplinary perspectives) in ways that provide the kind of detailed recommendations needed to ensure impact at all levels of policy and practice. This would necessarily call for a problem-based approach to research that extends over a long period of time. Such an approach would call for far more funding than is currently available in education. It would require everyone involved to understand various research methods and recognize how to integrate the findings from different types of studies. And it would call for a level of systematic collaboration that currently does not exist in the field.