Strategic Initiatives

Data Use and Educational Improvement

The Spencer Foundation is pleased to announce a new initiative on Data Use and Educational Improvement. This research program will support scholarship that examines the conditions, contexts, and underlying factors and processes that affect how educational organizations use data and information for improvement. We invite proposals that reflect the priorities outlined below.

There is a growing movement to have data inform policy and decision making, and there are underlying assumptions inherent in the movement—that educators will actually acquire and recognize appropriate data; that they will have the skills to interpret it; and that they have the capacity and expertise to respond. There is great potential for data to support and even transform educational policy and practice in various settings. But for many educators the promise of improvement through the use of data is constrained and even thwarted by a wide array of factors. Too often, we are unaware of how these factors affect our ability to use data for improvement.

It is not difficult to hypothesize about what might be happening in various settings; we’ve all heard stories about school principals drowning in data, or college administrators warned by legislators that their institutions will be held accountable on various performance measures intended to improve certain outcomes. But the field knows relatively little about what is actually happening once educators encounter data. We hope to remedy that gap by supporting rigorous, high quality scholarship with a research agenda described below.

We think the time is right for this agenda. There are many—including the federal government—who have committed significant resources to creating and maintaining systems to collect data and generate analyses for practitioners and policy makers all over the country. In addition, many institutions are engaging in their own data collection and analyses, especially in higher education. There is optimism on the part of those who promote data use for improvement. In K-12 education, school districts across the country are investing in systems to create enhanced access to data. Some are embarking on training to encourage teachers, principals, and district leaders to integrate attention to data in their ongoing practice. In higher education, increasing calls for accountability have motivated additional examinations of performance and outcomes, as well as questions about what counts as improvement.

At the same time, there has also been an increase in recent scholarship on how data are used. But this research is uneven. It focuses on some aspects of the data use phenomenon, while largely ignoring others. For example, there is a growing body of research on classroom use of assessment, but much less attention to how other levels of the system use evidence or how these other levels of the system influence data use in classrooms. And college administrators may see data collection and dissemination as a priority, but there is only anecdotal evidence at best about how few faculty members have embraced it.

The research base is also fragmented. Researchers from multiple disciplines and research perspectives investigate the phenomenon using contrasting conceptual frameworks and different language for similar concepts. More worrisome, a great deal of the research on data use is normative rather than analytic. That is, it focuses on trumpeting the cause of data use, or a particular approach to data use, rather than analyzing what actually happens when people at different levels of the system use evidence in their practice. The research of this type is characterized by great optimism about the transformative power of data use, but provides little evidence of when and under what conditions this optimism is warranted.

This Initiative attempts to remedy the paucity of knowledge about how data are actually used by supporting scholarly work that advances our understanding of the various phenomena influencing efforts for improvement. Spencer and the scholars we support can make different and valuable contributions to this emerging field of study by emphasizing our strength in inquiry. The Initiative will provide a catalyst for deeper and more rigorous research that improves our understanding of how data are used and how that use informs broader organizational improvement. The goal is not to discover and advocate for a specific intervention, though we hope that the scholarship we support is useful to both practitioners and other researchers. Rather, our goal is to create new knowledge about the conditions, contexts, and other factors that affect how data are used; stimulate additional attention to what happens after data are gathered and shared; and advance theory about data use, all as part of an effort to ensure that the promise of data use for educational improvement does not go unfulfilled.

What are we looking for?

Research questions and projects in Data Use and Educational Improvement

This initiative questions the assumption that the simple presence of data invariably leads to improved outcomes and performance, and that those who are presented information under data-driven improvement schemes will know how best to make sense of it and transform their practice. Our intent is to attend to the factors and underlying processes that can explain which data, in what contexts, and under what circumstances contribute to particular outcomes. We also focus on the capacities needed by school and district practitioners as well as college and university faculty and staff to translate, interpret, use, and evaluate the effectiveness of the particular data they use to drive improvement.

Many people inside and outside of K-12 and higher education have embraced data—in the form of standardized tests, or students’ evaluations, value-added analyses, or something as simple as attendance records—as the solution to many of the ills of our educational systems. The data-related activities and outcomes of other fields, from business to health care, have contributed to the increased interest in using data to improve educational outcomes in all settings. However, this rush to develop “data-driven decision making” processes is too often an accountability response that is not coupled with a genuine shift in organizational processes to learn from the data being gathered and used. Teaching and learning in classrooms, schools, and across departments easily get lost in the administrative mandate to execute data-driven processes. We are interested in research that goes beneath the undifferentiated demand for more data, that challenges the uncritical embrace of the effectiveness of data use, and that promotes the efficacy of using and understanding data to make actual improvements in K-12 and higher education.

We identify here a few of the important concepts that will inform the research agenda for this Initiative. These are fertile areas for new thinking, generative questions, and alternative ideas. For example, a traditional understanding of capacity might simply suggest that additional professional development workshops will address any perceived deficiency in skill or ability in individual faculty members. Instead, we want to consider the interactive processes that affect how individuals think about new information and how their background and knowledge shape that encounter. We also care about the influence of the organization itself. The culture of a school or university division is likely to have an effect on how data are used or not, on whether they can be acted upon for improvement. This Initiative is also concerned with the theory that will guide the research on data use. Are current theories sufficient to explain what we might find in this type of research? We think there are tremendous opportunities for growth in how we explain and measure what is happening in various settings where data are used. While this list is not comprehensive, these ideas—and the questions they help generate—are essential to improving our understanding of how data are used and how that use is connected to improvement.

Capacity: Many assume that if data are delivered to educators they will immediately be able to use them for improvement purposes. We want to identify the skills, knowledge and dispositions necessary for educators to understand, use, and learn from the data they encounter in order to make instructional improvement. There are also ways to think about capacity at an organizational level, about what structures or systems facilitate or inhibit certain processes, or whether an organizational culture treats the use of data as an add-on or as part of the regular work of its members.
Sensemaking: We are greatly concerned with how individual teachers and faculty members, as well as principals and department heads, learn how to use data and how they can work together to understand, interpret and apply data in their specific professional contexts. The knowledge, beliefs, and dispositions of the individuals involved influence how they interact with and make sense of new information. We want to learn more about how.
The link between data and action: Most often educators are given data with the assumption that they will immediately act on the information provided in ways that will result in demonstrable improvements. For instance, when a principal receives information showing a downward trend on 3rd grade performance or a Dean receives completion rates for majors from a particular department, the assumption is that these professionals will make immediate and positive changes. We are interested in what actually occurs when an educator receives such information and what results from their subsequent actions. We are also interested in the larger context and its effect on the ability to act, from strong external accountability to a specific political environment.
Theoretical underpinnings of data use: We want to encourage work through a variety of methodological and theoretical lenses that examines what makes data useful in education and other related fields, what tools might be necessary to help educators use data, and which organizational contexts contribute best to the effective use of data in educational settings.

There are several key questions that will help reach the overall goal of this Initiative: to create new knowledge about the conditions, contexts, and other factors that affect how data are used; to stimulate additional attention to what happens after data are gathered and shared; and to advance theory about data use for improvement purposes. The questions listed here are intended to be examples of questions that reflect that goal as well as the concepts described above. Specific research projects will undoubtedly have different and more specific questions, but these can serve as a general guide for the types of questions we find compelling.

How do the kinds of data and the forms in which they are presented affect how data are used?
What skills and dispositions are needed for professionals in elementary and secondary education, and in higher education, to interpret, understand and use data to enable improvement in practice and outcomes?
What sorts of training or professional development and workplace norms are needed to help education practitioners use data and information more effectively?
In what ways do educators’ beliefs and motivations shape their understanding and use of data and information in school and college/university settings?
What organizational cultures or structures influence how people make sense of data/information in their particular professional contexts?
How do principals, deans, and teachers take data and then apply this information in decision making about instruction, resource allocation, course design, and other pertinent concerns?
How do incentives from accountability practices influence data use? What is the role of power and politics in data use?
How do lessons from data use at one level of a system flow through the system?
What can we learn about successful and unsuccessful data use from other fields that can shape such use in education?
How do differences in leadership affect an organization’s use of data?
What conceptual frameworks and/or theories can be applied to the development and use of data in education in order to actually lead to improvements in teaching and learning?

Funding Considerations

We are interested in building a portfolio of work that represents the areas above and invite proposals that reflect these priorities. Our intent in the short run is to build the knowledge base both practically and theoretically in a way that can inform improvement in schools, colleges, and universities. In the long run we want to improve the quality of research on data use in schools and higher education and build a better understanding of how best to make data work for education improvement.

We envision a range of projects for consideration that address the questions we pose above as well as those questions not represented that provide the greatest chance to develop actionable knowledge on data use. We care about not only the questions guiding projects, but also the design of the studies. In this regard, we welcome proposals that meet the following priorities for the Initiative.

Encouraging work that is rigorous in its design yet understandable for broad audiences across education research, policy and practice.
Engaging scholars from a broad range of disciplines who specialize in understanding how data can lead to improvement in educational settings.
Encouraging research teams that include academics as principal investigators along with practitioners, policy analysts, and other relevant stakeholders and actors.
Supporting work that focuses specifically on the underlying processes of data use along with the organizational contexts and factors that influence these processes.
Designing empirical work that both generates and tests hypotheses.
Supporting comparative work on data use in education and in other fields, as well as work from international contexts.
Encouraging work across a variety of theoretical perspectives and lenses and across the spectrum of research designs and methodologies.
Supporting work that examines other types of data than standardized test score results.
Creating mentoring and networking opportunities for early-career scholars working in this substantive area.
Conducting work that leads to educational improvement by providing practitioners in schools and university settings with conceptual and analytic frameworks and tools to use data in constructive ways.

Expected Outcomes

Educators in K-12 and higher education nationwide are being called to use data and to adopt data driven decision making as tools to improve their practices and ultimately to improve student outcomes. The logic seemingly makes sense—performance and outcome data given to teachers, professors, principals, and department heads will help these professionals make necessary changes in their classroom and organizational settings. Yet there is limited, if any, inquiry on what might actually work in these different settings and why. This leaves educators drowning in data and high expectations that improvement will follow without helping them develop a better sense of how to use these data for improvement purposes.

This initiative intends to provide educators in K-12 and higher education with research evidence that leads to a clearer sense of how and why certain data work in some settings and not in others. Work funded by this initiative will build the empirical base and explore multiple theoretical perspectives on data use in education. In addition, the research Spencer supports will provide educators with better frameworks and tools to use data and to understand both the potential and the limitations of data use in their settings. By clearly identifying the factors, contexts, and situations where data use truly leads to improvement, and those cases where it does not, the Foundation hopes to underscore that producing and providing data to educators is a necessary first condition, yet not sufficient to lead to improved educational outcomes.

Application Guidelines for the Initiative on Data Use and Educational Improvement - click here.