Data Use and Educational Improvement

Current Evidence for the Classroom Projects

The following are summaries of the first round of studies funded through the Evidence for the Classroom Initiative. Although these are good examples of the kinds of research we seek to fund through the RFP, the absence of any particular type of study from this list should not be taken as a sign that we would not be interested in funding such a study through the current call. In short, we welcome a wide variety of theoretical and methodological approaches.

Melissa Braaten (Start: 7/1/2013, End: 7/30/2015)
Department of Curriculum & Instruction
University of Wisconsin-Madison
Evidence for the Middle School Science Classroom

Current emphasis on “data-driven instruction” rests on assumed connections between teachers’ analysis of data and subsequent practice. Systematic study will be necessary to understand how teachers’ attention to and use of student learning data is—or is not—related to instruction and decision-making. Because this practice takes place primarily in the privacy of a classroom and only periodically in the more public context of teacher groups or meetings, any study attempting to uncover data-driven decision-making from the perspective of the practitioner requires close, ongoing relationships with teachers. By working to understand the practices of “data use” from the perspectives of teachers, this project aims to understand how tools, colleagues, organizational structures, and other workplace resources provide both affordances and constraints for science teachers’ efforts at “data-driven” decision-making and instruction.

Over 2 years, this study will examine science teachers’ data use in daily practice in two middle schools that are racially, linguistically, and socioeconomically diverse. Using an ethnographic approach and multi-case study design, the project researchers will investigate teachers’ pedagogical decision-making when they obtain, interpret, use, and generate data in science classrooms. The researchers will examine personal, organizational, and contextual factors affecting science teachers’ data use and pedagogical practices, including purposes for using data; tools, resources, norms, and communities shaping teachers’ data use and pedagogical practice; and organizational, cultural, ideological, and political discourse shaping teachers’ data use and pedagogical practice. Based on interim findings from year 1, researchers will work with teachers and school leaders to develop tools and other supports for teachers’ professional learning teams. Iterative cycles of design studies during year 2 will examine how tools and resources shape these professional learning teams and support teachers’ attention to and use of student learning data during science teaching.

By becoming embedded observers, members of the research team will build case studies of: 1) individual science teacher practice and decision-making occurring daily in classroom settings, 2) collaborative teacher meetings and decision-making occurring at each school site, and 3) teacher participation and decision-making in periodic school level institutional meetings. Cross-case analyses will identify patterns and important variations between different instantiations of data use and instructional practice.

This study will: 1) provide rich evidence about the role that student learning data may or may not play in science teachers’ pedagogical reasoning and daily teaching practice, 2) generate findings useful for building meaningful supports for fostering productive data use practices for science teachers both in pre-service and in-service settings, 3) help provide connections between policy and practice.

Jennifer Greene and Thomas Schwandt (Start: 1/1/2013, End: 12/31/2015)
Department of Educational Psychology
University of Illinois at Urbana-Champaign
The Role of Student Characteristics in Teachers' Formative Interpretation and Use of Student Performance Data

Teachers’ practical use of student performance data (e.g., unit tests; interim assessments known as benchmark, screening or diagnostic tests; project-based assessments) in decisions to change instructional practices is a complex and elusive phenomenon. This study is designed to achieve two principal aims: 1) study decision-making in elementary school teacher-teams, and 2) describe aspects of school culture that may influence data use in instructional decision-making. This study proposes that understanding practical use of these performance data requires the simultaneous investigation of three sets of circumstances: 1) How and why teacher-teams draw on knowledge of student characteristics (not simply race, ethnicity, and disability status but also students’ cultural and conceptual knowledge, language practices, and other distinguishing characteristics) in their interpretations of student performance data (and, as importantly, if they do not draw on this knowledge, why not), 2) How instructional decisions based on these two data sources—knowledge of student characteristics and data on student performance— arise through a process of collective decision-making (i.e., what transpires in teacher-team discussions), and 3) How a school’s culture (e.g., organizational and political contexts, accountability system, general climate) influences the decision-making processes of teacher-teams.

Our specific research questions are:

1. In what ways and to what extent are interpretations of student performance data made by teacher-teams situated in their knowledge of student characteristics?
2. How does variation in these characteristics affect the ways in which teachers interpret performance data and use those data to make instructional decisions?
3. How are data on student characteristics brought to bear and negotiated as part of the process of interpreting and using performance data in a team setting?
4. In what ways and to what extent does the culture of the school influence or mediate how teacher-teams interpret and use student performance data?

Ilana Seidel Horn (Start: 1/1/2013, End: 12/31/2015)
Department of Teaching and Learning
Vanderbilt University
Data in Conversation: Professional Learning Opportunities in Teachers' Data-Centered Collaborative Talk

In the context of No Child Left Behind, student performance data have become commonplace, particularly in low-performing schools. In many places, improvement efforts attempt to harness the potential of teacher community along with the notion of evidence-based practice, making teacher workgroups important sites for interpreting student performance data. The proposed three-year project will investigate four middle school mathematics teacher workgroups in two urban districts as they discuss performance data. Focal groups will come from an existing longitudinal research project investigating systemic instructional improvement from district offices to schools and classrooms, providing analytic leverage from existing data and instruments. Through comparative case studies, we will uncover how teachers’ data conversations shape professional learning in different settings. Focusing on videotaped conversations, we will use methods from sociolinguistics to investigate learning in each workgroup. We will mine and augment existing data to situate the interactional analyses, tracing outward from teachers’ conversations to differing school, district and state contexts and inward to individuals’ contributions and expertise. This project will develop a model of teachers’ professional learning opportunities through collective engagement with student performance data. We will investigate teachers’ learning opportunities by examining workgroup cultures, specifically by comparing uses of representations of practice, epistemic stances, and interactional routines on teaching. These guiding concepts come from prior work on teachers’ learning through conversations. By extending these constructs to account for data use and specifying how they come together to shape professional learning in different contexts, we will sharpen our understanding of the learning potential in data-centered teacher conversations. Theoretically, this work will extend our understanding of teachers’ learning by examining relationships between workgroup practices and individual teachers’ development, as well as relationships of this development to broader contexts. Practically, it will illuminate learning consequences of data use, providing insight for school improvement and guidance for leaders.

Carolyn Riehl (Start: 3/1/2013, End: 9/30/2015)
Department of Education Policy and Social Analysis
Teachers College, Columbia University
Degrees of Freedom: Teachers' Use of Student Performance Data in Decision Making and Instructional Change

Teachers’ data-driven decision-making (DDDM) can serve a technical function in guiding instruction. It also is a constitutive practice of the school as a social organization. And in many settings, it has taken on the character of an institutional myth that lends legitimacy in an era of intense pressure for evidence-based practice. The proposed research is intended to develop knowledge about how, if at all, DDDM leads to real instructional change. In particular, we will explore the logics of action that guide DDDM in schools and how those logics are enacted through structures, processes, speech, and action. We posit that at least four logics may be in play: the logic of feedback and diagnosis, the logic of instructional repertoires, the logic of accountability, and the logic of student voice.

For this naturalistic case study, we will select four public elementary schools in New York City. We will study at least two naturally occurring DDDM groups per school.

We will observe the structures, interactions, processes, and tools and resources that comprise the practice of DDDM and analyze how these components reflect different theories of action for DDDM. We will document the substantive content of DDDM discussions and the instructional decisions and plans that are made. We then will make repeated direct observations of classroom teaching, documenting the instructional changes that follow from DDDM. We will triangulate our observations with interviews with teachers and school leaders. A longitudinal design will enable us to observe how teachers’ participation in DDDM and their efforts to make change in their classrooms evolve over time.

In the second year, we will add a design study of student participation in DDDM in one or two schools that volunteer to participate, helping teachers to develop rubrics and other mechanisms for eliciting student self-reflections on their performance, and then observing how those are incorporated into DDDM practices and how they are interpreted by teachers and used for making instructional adjustments.

Overall, we aim to elaborate an evidence-derived theory of action for how the socially situated practices of teachers’ use of student performance data have impact in classrooms and schools. These findings will establish evidence proofs that can shed light on DDDM in other settings.

Brenda Turnbull and Erikson Arcaira (Start: 1/1/2013, End: 6/30/2015)
Policy Studies Associates
Now What? Instructional Responses to Interim-Test Data

This is a study of elementary teachers’ instructional decisions made in response to students’ performance on interim or benchmark tests of reading and mathematics. Through case studies in three elementary schools and close analysis of supports available to the teachers, the team will probe ways in which instructional prescriptions are shaped by the schools’ tools, structures, and routines for data use and for instruction—and learn how these school conditions support rigorous, applied academic work as a response to student performance data.

The central research questions are:

1. How do elementary-grade teachers make instructional decisions in response to students’ performance on formal interim assessments?
   1. How are these decisions influenced by characteristics of the data (content and format) and the organizational conditions for the process of interpreting data (formal and informal structures, and organizational routines)?
   2. How are these decisions influenced by the availability of specific instructional tools, structures, and routines?
2. What supports teachers in choosing instructional responses to interim assessments that go beyond narrowly focused test preparation?

Through in-depth data collection and analysis in reading and mathematics, grades two to five, across schools that have strong student performance and that have made serious efforts to use data productively, the study will identify ways of supporting teachers in assigning academic work that includes applications of skills and knowledge. Because the school sites have quite different philosophies, the types of supports that are common across settings will have some generalizability.

The study addresses a gap in the literature on data use: relatively little is known about teachers’ instructional choices made in response to student data, other than that teachers may struggle to translate data into instructional prescriptions and may fall back on re-teaching the specific items missed on tests. Thus, the answers to the research questions will have significance for research and practice.