A Memorandum for the Spencer Foundation Board of Directors
By Ellen Condliffe LagemannJanuary 24, 2002
New Approaches to Education Research
In recent years, there has been growing agreement that, in order to serve the education profession, education research needs to produce knowledge that includes, but goes beyond, the kinds of knowledge that are produced by the academic disciplines. A key attribute of the professions is that they claim to have special, valid knowledge that enables their members to accomplish something useful in the world. Professionals cure or prevent disease, build skyscrapers and bridges, write wills and divorce settlements, file tax returns, and teach children. As professionals, educators need to be able to assert that they know how to educate. They need to be able to claim that they know how to foster learning in diverse groups of people. Education research must provide knowledge that helps educators to do this. Its role is to discover and to test and certify the knowledge, skills, and tools educators can use to facilitate learning.
Professional knowledge has to be applied in particular situations - it has to be applied "in practice." That means that the knowledge must illuminate and enable educators to act on problems in the context of particular systems and schools with specific teachers and students. Such situations are inherently complex and affected by many variables at many levels. Disciplinary research, by contrast, tends to focus on developing knowledge of important constructs and variables and on understanding relationships among those constructs and variables. It aims at identifying regular or law-like relationships that will always exist, "other things being equal." To be sure, some disciplines, history being one example, sometimes focus on one specific event or situation and try to understand everything about that case. Despite that, such disciplines still differ from professional knowledge in that their goal is to foster understanding rather than doing.
The knowledge produced by education research often incorporates understandings derived from the disciplines, but the test of its value lies in whether, if acted upon, it is likely to achieve the desired result(s). Put otherwise, education research must translate and combine understandings from the disciplines about all the factors that may affect a particular educational situation. These understandings then represent statements about the tolerances and contingent probabilities within which there is a chance that acting on the knowledge will achieve what one hopes to accomplish. It is this function of combining knowledge of many sorts and thereby providing evidence about the chances of accomplishing one's goals that should distinguish education research from other kinds of investigation. It is the capacity to predict outcomes from specific actions taken in practice that should set education research apart from other kinds of scientific inquiry.
Understood this way, education research is often called "use-inspired" research, a term coined by Donald Stokes in Pasteur's Quadrant.[1] Borrowing from the title of a classic book by Charles E. Lindblom and David Cohen, it can also be called "usable knowledge." Ideally, this type of research takes into account all of the conditions that exist in the setting or case under study, thus helping to insure that the limit to its generalizability will be clear. In an article about teacher research, MagdaleneLampert noted that trying to understand "what is learned from a single 'case' in all its complexity" and how that learning applies to other situations in which similar problems arise is one of the "enduring puzzles" scholars of education face. [2] In fact, Lampert's new book, Teaching Problems and the Problems of Teaching, is a very creative effort to describe and understand that puzzle, and the challenges of creating usable knowledge in education.
Use-inspired research should generate knowledge that can be embodied in tools, materials, curricula, or what David Cohen, Steve Raudenbush, and Deborah Loewenberg Ball call "instructional regimes" that can be studied in practice. Instructional regimes must be understood as involving constant interaction between and among a variety of resources. Some of these resources will be brought by the teacher(s), some by the student(s), and some will be embedded in the available instructional materials and activities. The effects of an instructional regime will depend on the experience, knowledge, norms, and approaches to learning of the students and the knowledge, skill, and strategic actions of the teachers. In addition, the interaction of these resources within instruction will affect and be affected by the resources available in the context in which the instructional regime is implemented. Matters of time, district requirements about homework, state and/or district standards, school leadership, and parental involvement all can influence the ways in which student and teacher resources will interact. However, their influence will be mediated by the character of those very interactions, and this is even clearer in the case of such factors as district wealth and expenditure or average class size. As Cohen, Raudenbush, and Ball state, "the instructional effects of conventional resources depend on their usability, their use by those who work in instruction, and on the environments in which they work."[3] As in dynamic treatment regimes in medicine, which allow doctors to adjust doses depending on a patient's current status, instruction may be modified by teachers according to their perceptions of student needs, interests, and aptitudes and by students according to their perceptions of a teacher's expectations, competence, or even friendliness.
All these complexities of life in "real" classrooms mean that survey research methods would not be likely to find significant or interpretable relationships between general measures of school resources - for example, per pupil expenditures, class size, teacher credentials, or quality of facilities - and measures of pupil attainment. They could only do so if they could take into account or hold constant all the mediating factors that shape actual instruction. In light of this, it seems clear that studying the influence of resources on instruction in the ways social scientists have traditionally approached this relationship is not likely to yield useful understandings. Instead of asking how available resources affect learning, one now needs to ask about which instructional approaches will be effective in reaching which goals and what resources are needed to implement this approach.
What does all this suggest in terms of how we think about education research? It suggests, first, that it will be desirable to organize different kinds of experiments, some to demonstrate "efficacy," which will show how an intervention works under ideal conditions, and some to demonstrate "effectiveness," which will show how an innovation works in different "real" situations, constrained by resources. "Active" experimental or quasi-experimental research can be usefully supplemented with other more "passive" kinds of studies. Observational and correlational research is needed to provide the insights, hypotheses, and descriptions that will tell us about the constraints operating in different situations. Survey research and micro-ethnographies will also alert us to new instructional regimes emerging in practice.
New "active" approaches to education research help explain how and why education research must go beyond the purposes of traditional disciplinary research in the social and behavioral sciences if it is to yield the kind of applied, context-specific guidance that the education profession needs. This point is further supported by the argument presented in the National Research Council's report, Scientific Inquiry in Education. [4] There, the argument is made that science is science regardless of the field to which it is applied. "Scientific research, whether in education, physics, anthropology, molecular biology, or economics, is a process of rigorous reasoning supported by a dynamic interplay among methods, theories, and findings."[5] That said, the principles that define science must be specialized to suit differences among fields. "Education has its own set of features- not individually unique from other professional and disciplinary fields of study, but singular in their combination - that give rise to the specialization of education research."[6] Among the features that make science in education different from science in other fields, none is more important than the fact that education varies constantly depending on the "physical, social, cultural, economic, or -historical environment."[7] With that in mind, the report recommends support for theory-building studies in those areas where "basic theoretical understanding is weak."[8] This would be fundamental, disciplinary work. In addition, it urges much more attention to what is called "implementation research, the genre of research that examines the ways in which the structural elements of school settings interact with efforts to improve instruction."[9] It also urges efforts to understand how interventions that are successful in one setting can be "scaled up" in others. These are examples of the kinds of context-specific research that can be defined as use-inspired or usable research in education.
Of course, much that is now being said about use-inspired education research has been said before, albeit somewhat differently. For example, it has long been acknowledged by people savvy about education that education research is more akin to a design or engineering science than it is to the disciplines. Indeed, this seems to have been Lyle Spencer's view of things. He earned his wealth translating research into curriculum materials and tools that could facilitate instruction. That aside, matters of status and other disincentives to develop specialized research strategies that are especially appropriate to problems of practice in education have persistently steered education researchers away from approaches that have demonstrable, positive influence on the ways in which educators can operate. It is because scholars of education have increasingly come to recognize this that they have now begun working to develop new approaches to education research. These have often involved design experiments in which a researcher develops an innovation and then investigates it while implementing it. [10] According to traditional disciplinary canons of rigor, this mode of research can be seen as subjective and open to bias. And yet, such research is likely to be extremely productive in education, where research must be usable to be effective. The challenge now is to find new ways to address matters of bias so the results of such work can be trusted.
Not surprisingly, perhaps, more and more people concerned about science policy understand this. In fact, discussion of basic research that can be socially useful was the theme of a fascinating conference convened by the David and Lucille Packard Foundation and the Alfred P. Sloan Foundation in November 2000. At that meeting, Gerald Holton coined the term "Jeffersonian Research" to describe "a style of research that locates itself in areas of ignorance of how to meet societal need." [11] Admitting that such research had frequently gone on in the past, Holton nevertheless maintained that heretofore there had been too little explicit discussion of why such research was vital. He expressed concern that U.S. spending for basic research has dropped precipitously -- in 2000, for example, it was about 0.6% of the GNP, the same level it was at in 1953. He also worried that the public is not well enough aware of what science can do to help solve difficult human problems. It is for this reason that he advocated trying to stimulate public discussion of "Jeffersonian Research."
At the Packard-Sloan conference, there was also considerable talk about ways to use research to improve education, both in this Jeffersonian sense of concentrating on basic work likely to inform the design of practical solutions and on the problems of design itself. In his report of the conference, Lewis Branscomb noted that there was consensus that "we need new avenues for capturing the wisdom of practice, and we need a new kind of professional who can bridge the worlds of research and practice. "[12] Teachers do not have the time to translate research into the tools, texts, curriculum, and tests that could enhance their instructional competence, and schools do not generally encourage experimentation. Lacking a demand for research, available knowledge does not get put to use. To the participants at this meeting, finding ways to change that situation was an urgent national priority.
Interestingly, in the Summer 2001 issue of Items and Issues, the newsletter of the Social Science Research Council, includes an article by Charles R. Hale about what he calls "Activist Research" in the field of global security and cooperation. According to Hale, activist research "a) helps us better to understand the root causes of inequality, oppression, violence and related conditions of human suffering; b) is carried out, at each phase from conception through dissemination, in direct cooperation with an organized collective of people who themselves are subject to these conditions; c) is used, together with the people in question, to formulate strategies for transforming these conditions and to achieve the power necessary to make these strategies effective."[13] Across fields, there is clearly a press to define research strategies that will harness knowledge more effectively to the needs of the "real" world.
Why Do We Need Use-Inspired Research and Need to Better Understand Use-Inspired Research in Education?
Of course, increasing recognition of the importance of linking research more effectively to practice and policy in education comes within a specific context. It comes at a time when the United States has committed itself (at least rhetorically) to a radical new goal. In the past, we thought it sufficient to offer all children an equal education. If some prospered and others did not, that was deemed to be in the nature of things owing to differences in ability. Now, by contrast, we are saying that we want to educate substantially all children to high levels. This is an appropriate aspiration for this country, but it poses challenges that are too little recognized. It will require a much deeper understanding of instruction than we now possess, and it will necessitate that we solve (or at least chip away at) traditional problems of linkage between theory and practice. By studying the development, organization, evaluation, and diffusion of usable knowledge in education, Spencer can contribute to the knowledge base needed to realize the nation's educational goals.
In addition to such study, we also need to find ways to better educate the public about education. Too few people fully understand how difficult it is to teach reading and mathematics. Too few people understand how complex educational problems are.
Knowledge cannot guarantee that the public will support intelligent education policies based on research, but it might foster greater skepticism toward data-resistant ideologues who seize on issues like bilingualism, social promotion, and remedial education and force ill-informed changes in policy. I believe that Spencer should play a central role in helping to develop a new science of education and to better educate the public about education. I think we can do this by developing an initiative called "Usable Knowledge in Education."
A Spencer Initiative to Promote Usable Knowledge in Education
Spencer does not have the resources to carry on the kind of research David Cohen, SteveRaudenbush, and Deborah Loewenberg Ball are doing. On occasion, the Foundation might want to participate in that kind of work, but the Foundation's comparative advantage does not lie in funding such studies. Instead it lies, I believe, in studying and learning about new approaches to research and sharing what we learn with other funders and policymakers. In a sense, it lies in assuming a meta-analysis and diffusion role. Let me try to clarify what I mean with a few examples.
To my knowledge, no one is studying how different efforts to carry on use-inspired research are working out. No one, for example, has studied the Consortium for Chicago School Reform to determine what is working well and what is not working well. Beyond that, no one is monitoring progress to be sure that researchers in one venue - again, say, the Consortium - are talking with researchers trying to do similar work in other settings.
Another example is found in the work of Barbara Neufeld, a field researcher, who has been investigating how standards-based reform plays out in classrooms, individual schools, and districts. Neufeld seems to be creating knowledge about practice that may inform practice and its management. In a sense, she is an ambassador from practice who uses research to improve practice. How does Neufeld's approach to developing usable knowledge compare to the approaches used by researchers at the Consortium and other people carrying on research in practice?
Another example that is also worth pondering because it has been quite successful is the National
Reading Panel, established by Congress in 1997. The Panel focused on the question of what research findings (as opposed to products) about reading had sufficient evidence behind them to justify basing instruction on them. Even though proponents of whole language approaches to reading instruction were critical of the Panel's work, it was generally balanced and reasonably well received. What could be gleaned from the Panel's success that could be useful in trying to learn how we can evaluate knowledge in education?
The Center for Policy Research in Education (CPRE) has also learned a lot about how one gets policy research to policymakers in ways that will actually inform the decisions they make. Someone should be studying and learning about CPRE and attempting to find ways to replicate its approach.
The Mellon and Russell Sage Foundations funded what was called "The Literacy Project" for ten years. Beginning as an effort to apply cognitive science to problems related to literacy, the program evolved into an experiment in sustaining and diffusing complex innovations. It moved in this direction because the scholars involved realized that the fundamental research in cognitive science they were doing had little effect on learning. The research was solid, but it did not translate into classroom improvements. Hence, the scholars shifted their direction and began experimenting in classrooms using cognitive science as a way to evaluate the innovations they generated, rather than as a basis for the design of innovations. Once successful innovations were in place - innovations like Ann Brown and Joe Campione's "Communities of Learners" - the next logical question was how to sustain and diffuse them. The scholars who were involved in "The Literacy Project" are currently completing a book on what they learned. We should look for ways to build on their work.
I have provided these examples to try to make concrete what we mean by new styles of use-inspired research. It is important to recognize, however, that our efforts to learn about new styles of research - to play a meta-analysis and diffusion role - should be thought of as part of building programs of research. This involves synthesizing what is known about the attainment of specified educational goals; on the basis of that, setting priorities for what needs to be learned; carrying out the necessary research and design; evaluating the knowledge that results; diffusing the knowledge that appears to be warranted; again synthesizing what is now known; and so on in cyclical fashion. This is really the only way usable knowledge will aggregate in education and the practice of education become research based.
Given these interests and concerns, we propose launching an initiative focused on the development, organization, evaluation, and diffusion of usable knowledge, especially but not exclusively, in K - 12 education. The initiative would have two overarching goals:
- Developing understanding about ways to design, manage, and support education research that will actually enable educators to ensure that substantially all students meet or exceed standards in core subjects and skills. This will necessitate coming to understand the ways in which research can support the development of both knowledge and instructional tools.
- Helping the public, policymakers, and practitioners understand what is known about effective education, what is not known or not true, what needs to be learned, and how the trustworthiness of knowledge in education may be assessed.
These goals will guide the Foundation's initiative in the coming months and, over the long run, contribute to the development of new approaches to education research.
Please address all inquiries to usableknowledge@spencer.org. Thank you.
[1] Donald E. Stokes, Pasteur's Quadrant: Basic Science and Technological Innovation (Washington, D.C.: Brookings, 1997).
[2] Magdalene Lampert, "Knowing Teaching: The Intersection of Research on Teaching and Qualitative Research," Harvard Education Review, 70 (Spring 2000): 93.
[3] David K. Cohen, Stephen W. Raudenbush, and Deborah Loewenberg Ball, "Resources, Instruction, and Research," p. 18.(In Press). To be published in Boruch, R. & Mosteller, F. (eds.) Evidence matters: Randomized trials in education research Brookings Institute. Also published as a working paper, Center for Teaching Policy. Seattle: University of Washington. (An earlier version of this paper was presented at the fall meeting of the American Academy of Arts and Sciences, October, 1999.)
[4]Scientific Inquiry in Education, edited by Richard J. Shavelson and Lisa Towne (Washington, D.C.: National Academy Press, 2001).
[5] Ibid, pg. 1
[6] Ibid, pg. 3
[7] Ibid, pg. 3
[8] Ibid, pg. 87
[9] Ibid, pg. 88
[10] Allan Collins, "The Changing Infrastructure of Education Research," in EllenCondliffe Lagemann and Lee S.Shulman, Issues in Education Research: Problems and Possibilities (San Francisco: Jossey Bass, 1999), pp. 289-298
[11] Gerald Holton, "What is the Imperative for Basic Science that Serves National Needs?" In Science for Society, a Report of the November 2000 Conference on Basic Research in the Service of Public Objectives, p.77.
[12] Ibid., p. 18.
[13] Charles R. Hale, "What is Activist Research?" Items Issues II, No. 1-2 (Summer 2001), p.13.