Reflections on Reading Spencer Proposals: An Opportunity to Learn

Elizabeth Birr Moje
University of Michigan
June 2008

Every year for the last five years, I have agreed to participate in some sort of Spencer Foundation selection committee, whether the Spencer Foundation’s Dissertation Fellowship Selection Committee or, more recently, the Exemplary Dissertation Award (EDA) Committee. The rational person might wonder why an individual would agree to read multiple rounds of many different dissertation proposals and even dissertations. Indeed, this is a question I have asked myself over the last several years. In this essay, I answer that question by highlighting the opportunity to learn that comes from being part of Spencer’s multidisciplinary selection process. Specifically, I demonstrate what a cross- or multidisciplinary conversation about education research can look like and why such conversation can enhance originality, rigor, relevance, disciplinary knowledge and, especially, clarity of expression in education research. Ultimately, the point of this essay is to underscore the value of a particular kind of collegial exchange around ideas, whether for improving dissertation writing, doctoral education and mentoring or, perhaps, education research writ large.

The Spencer Foundation’s Exemplary Dissertation Award Selection Committee is a unique group. Members are, generally, representative of research institutions (whether universities or labs) across the United States. We are thus influenced by the norms and practices of the academy. Committee members know, for example, how to conduct reviews. We know that criticisms should be constructive, personal biases should be bracketed as much as possible, and that we should recuse ourselves from deliberations and voting when we find such bracketing impossible. We also know to examine proposals for evidence of originality, rigor, relevance, contribution to disciplinary knowledge, and clarity. Selection committee members are, however, also members of other sub-cultures, namely our respective disciplines and fields. We bring different kinds of disciplinary and methodological expertise. We may even view the purposes of education research in different ways, and thus, each of us has a unique take on what counts as original, rigorous, relevant, contributory, and clear. For me, these differences make the review work intellectually stimulating because the process provides an opportunity to learn how members of different disciplines and fields think.

The EDA review context is fascinating, however, because the very differences I have outlined could have made the work difficult, painful or even impossible. We could have reached an impasse in our understanding and have failed to make any recommendations. We could have left the table each day frustrated, embittered, or questioning our own judgments as scholars. We could have learned to devalue the scholarly traditions of different disciplines. None of these things happened, in large part because of the committee’s constitution and because of the way in which our norms for working together were established.

The greatest single asset of the work of the EDA committee, I believe, is that we were focused on a particular, and yet broad-minded, mission, as defined by Lyle Spencer’s commitment to “improving education broadly conceived, wherever learning occurs.” What is powerful about Spencer’s vision is that it is clear and focused (“improving education”), but simultaneously open-minded (“broadly conceived, wherever learning occurs”). This open-minded view of learning and contexts for learning sets a tone for selection committees, so that they can value equally a range of foci and methods, such as the studies represented by this year’s EDA recipients. These EDA recipients included Timothy Reese Cain’s historical analysis of academic freedom and tenure in higher education; Sean Kelly’s mixed method study of race, class, engagement, and literacy development in middle-school classroom interactions; Douglas Lee Lauen’s economic analysis of the causes and consequences of school choice; Karthik Muralidharan’s randomized evaluation of the impact of teacher incentives on children’s learning in school; and Mary Carmel Murphy’s experimental study of social identity threat among university level students.

By Spencer’s definition, these five very different studies, together with all of the excellent studies reviewed for the EDA, counted as studies of education and learning. There was no debate as to whether these studies mattered at the outset; thus, committee members came together around the table with respect for the various foci and concomitant designs and methods for carrying out those studies. What we still had to deliberate on, however, was how all of the studies reviewed mattered differently to different education problems, how their designs, methods, findings, and conclusions were framed in different ways, and how each might serve as an exemplar of the best in education research.

This last point signals a second key dimension of the committee: its multidisciplinary and multi-methodological composition. It is, perhaps, obvious why an interdisciplinary committee would be critical to judging the criterion of contributions to disciplinary knowledge. Whereas we could all weigh in on whether we saw a particular study as having relevance for the improvement of education, it was less simple for outsiders to a given discipline to determine the value of the contribution a study made to that disciplines. In each case, we had to rely on the expertise of members of each discipline to inform those outside the discipline whether the work under review made a new and noteworthy contribution to the disciplinary knowledge base. We also had to rely, however, on the expertise of a multidisciplinary group because not only does knowledge differ by discipline, but so also do the ways of knowing and ways of communicating knowledge of each discipline.

Specifically, what counts as and is used to warrant rigor may differ from one discipline to another. Psychologists, for example, typically describe “subjects recruited” and “instruments administered,” whereas historians will name the archives they have examined. Education scholars might describe in depth a curriculum developed and studied in only one classroom design experiment, whereas an economist will present mathematical models to represent the phenomena under study in a randomized trial. Moreover, embedded in said mathematical model is the mechanism for studying the phenomenon, whereas the curriculum of the education scholar represents the concepts and practices under study, but not the methods for studying its enactment. It can be jarring, for example, for the reviewer who expects a thorough-going portrayal of analytic methods to encounter a single sentence that explicitly names an analysis because the analytic methods are assumed or implied in the report of findings. It can be equally confusing for the reviewer who expects a mathematical model to find pages of text dedicated to the procedures for and examples of discourse analysis. Such differences necessitated an multidisciplinary review process, one that was sensitive to the disciplinary and methodological traditions that were part and parcel of each study.

Differences such as these made the EDA selection committee members interdependent in important ways, leading us to ask questions rather than merely offer ratings and critiques. Because we were expressly committed to giving each study proposal the very highest quality review we could give, we pushed ourselves to be sure that we understood the studies from their disciplinary and methodological perspectives. In one case, for example, we brought in the expertise of a scholar outside the committee to check on a point that none of us could answer.

All this said, it is worth noting that we also agreed that the best proposals were those that could communicate across disciplines and methods, that is, the proposals that acknowledged their particular knowledge bases, as well as their ways of knowing and doing. These proposals also sought to clarify why that knowledge base and those ways of knowing necessitated particular designs, methods, and discourses in the study’s written presentation. This point was particularly important when judging clarity of expression. There were times in our reviews that a member of a particular discipline would acknowledge that a proposal appeared to contribute to disciplinary knowledge via rigorous and original methods, but did not make that knowledge contribution as clear to a broad education community as it could via its expression of ideas. Although we were committed to understanding each proposal and to recognizing that rigor, originality, expression, and contribution may differ across fields and disciplines, we were also committed to locating studies that communicated findings to a range of people interested in improving education because scholars cannot affect change in education if we cannot communicate our findings to a broad audience.

The implications of these observations for dissertation writers should be clear. One obvious implication is that scholars should consider the five criteria by which proposals are evaluated as they write various proposals for consideration by the different Spencer review committees (although these criteria specifically reflect the EDA committee’s deliberations, they are, in effect, the criteria by which all proposals are judged). More to the point of this essay, proposal writers and new scholars should also work toward a broad understanding of what counts as relevance in education research, as well as an understanding of how qualities of rigor, originality, contribution, and clarity within one’s given discipline or field might compare to those of another field. In other words, strong education scholarship demands not only fluency in the standards valued in one’s discipline, but a meta-level awareness of those standards and at least a minimal understanding of the standards of other disciplines. This meta-view can be attained by reading both within and outside one’s discipline and consulting with members of other disciplines, just as the occasion of the EDA process allows selection committee members to do.

These implications are by no means always self-evident. Scholars, especially emerging scholars, often ask those within the field to read papers and proposals and, in the case of proposal writers, to serve as recommenders of the work. But how often do scholars imagine their work being read by a group comprised of psychologists, economists, sociologists, historians, anthropologists, and education scholars? What might be the benefits of a scholar—whether a developing or senior member of the field—attempting to read her own work from the viewpoint of a researcher who studies similar issues, but from different standpoints, for example, historical rather than contemporary, or sociological rather than psychological? How might a study be written if the author recognized that the work had to communicate its value just as clearly to a scholar who develops a mathematical model of whether teacher incentives lead to student learning gains as it does to the scholar who studies closely how teachers make decisions about moment-by-moment instructional moves? To write well to such a diverse group requires a broad and meta-level stance on one’s own work. To write well means not only avoiding an overreliance on undefined technical language of the field or discipline, but also attention to making clear what counts as rigorous data collection and analysis in one’s field. It requires signaling to a readership what counts as warrant and making the significance of one’s work clear to a diverse body of readers. Studies that communicate beyond their disciplinary domains are not only more likely to win awards (or be recommended for funding), but they are also more likely to have an impact on the field and on learning wherever it occurs. Indeed, it might be argued that what makes an exemplary dissertation exemplary, even beyond the qualities by which the EDA selection committee judges proposals, is the study’s ability to communicate to a diverse group of readers, a quality that suggests greater likelihood for the study to have an impact in the world. Ultimately, exemplary studies communicate their rigor and originality to a diverse audience.

The implications of these observations extend beyond dissertation writers to the work of senior scholars who serve as mentors to developing scholars. As mentors, we need to find ways to engage our students across a range of disciplines and methods, not only to produce interdisciplinary or intermethodological studies, but also to improve single method studies conducted entirely within disciplinary boundaries. We need to push students to read broadly and deeply, and we need to connect them to scholars in other field. Informed by my own learning experience on Spencer committees, I have collaborated with my colleague, Bob Bain, to develop a research group that brings together doctoral students and faculty from diverse backgrounds. To date, the group includes participants with backgrounds in literacy, history, teacher education, psychology, learning sciences, natural sciences, science education, anthropology, and English education. We read each others’ work, and members present everything from working analyses to practice conference presentations. Bob and I share work in progress to model how we struggle through challenging analyses and seek advice from a range of colleagues. I also use what I have learned through the EDA selection committee process to advise my own students and to mentor junior colleagues.

In sum, I found the work of the EDA selection committee to be an opportunity for intellectual engagement as well as a space for my own professional development as a scholar and as a teacher of other scholars. Spencer committee processes are designed not only to give all applicants a “fair shake,” but also to push forward the thinking of committee members, all of whom advise doctoral students and conduct their own research, and Spencer Foundation program officers. We learn to be better advisors to our students and better colleagues to one another by coming together around a shared intellectual task. I hope that this essay encourages similar learning among emerging and established scholars alike.