Interview: Researcher Susan Dynarski on Boston Charters, Student Poverty, and Data in Education
Explore previous interviews: Secretary of Education Arne Duncan, former U.S. Sen. Tom Harkin, current U.S. Senator and education committee Chairman Lamar Alexander, and Professor Chris Emdin, who focused on the topic of "white teachers in the hood." The complete 74 Interview archive can be found here.
Aside from her prolific academic journal writing on K–12 and higher education policy, University of Michigan economist Susan Dynarski is also an influential user of Twitter and a regular contributor to the New York Times.
Earlier this month, we extensively discussed the role of research in the mainstream education debate, the lasting impact of No Child Left Behind, and Dynarski’s own research, which centers on Boston charter schools and the measurement of student poverty. (The interview has been minimally edited for clarity and length.)
: How do you believe educational research is utilized in informing policymakers and journalists?
Dynarski: It is a broader issue, similar to how research is used to inform policies in the U.S. or any other country. Education policy, in particular, is quite diffuse. I think you can draw parallels with medical policy and health policy because there is a more centralized mechanism for shaping it. For instance, Medicaid and Medicare dictate what they will pay for, which influences medical practices through randomized trials. However, there is no similar central body that governs education policy in the United States.
Randomized trials are rarely seen in education policy.
What do you mean by "rarely"?
I mean that they are not as common as one would hope, and policymakers often do not explicitly incorporate them.
That is correct. In fact, they are hardly ever incorporated. While the practice is beginning to gain traction in certain areas, it largely depends on individual organizations and their leaders.
I think there has been improvement over time. We now have more data that allows us to comprehend what is happening in schools and districts than ever before in history. The primary legacy of No Child Left Behind, in my opinion, is not accountability but the fact that we can now measure test scores annually in every state. That was not possible before NCLB. Many research papers and statements about children and their performance would not exist without this data.
Federal policy has been encouraging states and districts to incorporate more evidence in their policies, which is a positive development. Initially through NCLB and then during the recession, specific conditions were attached to federal funding to enhance data analysis and capabilities at the state level.
Is there significant variation between states in terms of data collection and accessibility for educational researchers? I ask because I recently reported on California, where they have severely limited their educational data collection systems and have made it extremely difficult, if not impossible, for researchers to obtain data. However, it seems like other states are doing a much better job in this regard.
There is indeed substantial variation. California is an extreme example, as you mentioned.
And it is a small state, so perhaps we should not be too concerned about it!
It is astonishing. The data already exists, and what was discontinued and defunded was the pooling of the data. California is facing difficulties. Researchers are finding ways to cope by forming partnerships and creating local-level databases. Some cities are collaborating—I am aware of at least one consortium of large cities that have united their data for analysis. Despite the state’s failure to create a statewide data set, the college system is actively pursuing research.
There are various models out there. This situation truly exemplifies the states serving as laboratories. North Carolina, for instance, readily provides its data to education researchers. And when I say "readily available," I mean with all the necessary security and privacy measures in place to protect the data. But if a university-based researcher poses a valid research question, North Carolina will facilitate the process.
Do you have any advice for researchers on how to effectively communicate their research in a way that is more understandable and practical for journalists and policymakers?
If you have the necessary resources and the support of a university that can assist you in creating more accessible and concise summaries, it can be highly beneficial. Utilizing platforms like Twitter and blogs can be a great way to condense your work into a more digestible form for wider dissemination. However, caution should be exercised when collaborating with university press offices as they may prioritize attention-grabbing headlines over accuracy and expertise in your field. It is essential to maintain control over the message you want to convey, especially when encountering exaggerated claims made by these press offices. The conflicting interests of gaining press attention and delivering accurate information can create a challenge.
As for the balance between being an objective researcher and a public intellectual who expresses strong opinions in outlets such as the New York Times, there is indeed a tension. Take student loans as an example. This policy area lacked solid academic research and reliable data, leading to a reliance on theoretical speculation and anecdotes. In such cases, it becomes crucial to highlight basic facts to inform the conversation. While I refrain from taking positions on specific legislative pieces, I focus on providing information to guide discussions. I don’t perceive a conflict between these roles, although it can be more challenging for junior faculty to navigate. The protection offered by tenure certainly helps.
When acting as a "public intellectual," it is essential to stick to one’s area of expertise. It is important to acknowledge that my position does not grant me expertise in all subjects. Many individuals make the mistake of becoming general commentators beyond their field. Journalists also struggle with determining the implications of conflicting research findings presented by experts. Sometimes the research itself may be inconclusive, making it difficult to comprehend how two knowledgeable individuals can have opposing views. In such cases, it may be necessary to emphasize the unsettled nature of the question. However, false equivalence should be avoided when the weight of research clearly favors a particular view. Climate change serves as a classic example where the media’s portrayal was too balanced for a significant period of time.
Now, let’s discuss your research on Boston charter schools and your findings. In collaboration with Harvard and MIT researchers, I have been involved in a long-term research project investigating the performance of charter schools in Boston. Our findings indicate a substantial positive impact on student achievement, which is among the highest I have observed across various educational settings.
Notably, middle schools and high schools demonstrate significant improvements in math scores. Attending one of these schools for a few years can help bridge the test score gap between black and white students, for instance. To reach these conclusions, we utilize lotteries as they enable us to compare outcomes for students who either win or lose the lottery for admission. This approach is crucial because students are not randomly assigned to schools due to the selection bias issue arising from their choices.
Additionally, our research reveals that high schools increase college attendance, improve the quality of colleges students attend, enhance SAT scores, and encourage AP course participation. Overall, it appears that charter schools in Boston are performing commendably.
Our research has made a significant impact on the policy debate. Our initial papers on Boston charters played a role in changing the mayor’s perspective on charter schools. Currently, there is a contentious battle in Massachusetts regarding the cap on charter schools. While other states do not have such caps, it remains a significant factor in Massachusetts.
Furthermore, the outcomes in suburban and non-urban areas are not as positive as those in Boston. This implies that the cap on urban areas should be lifted, while still considering the implementation of a cap in suburban areas.
The research conducted on the effectiveness of a program should not be the sole determining factor for someone’s decision to vote in favor of the program. It should be taken into account along with other considerations, such as distributional questions. Priorities may differ for individuals, and it is essential for people to carefully weigh their decision based on their own priorities. It is disheartening that discussions often revolve around disregarding research rather than acknowledging alternative priorities.
Could the positive effects observed in Boston charter schools be attributed to peer effects? Is it possible that the students who choose to attend charter schools are more motivated, and therefore the winners are surrounded by similarly motivated peers?
While it is a possibility, our attempts to investigate this suggest that it is not the main contributing factor. We conducted a recent study on high schools in Boston to address concerns regarding low achievers being pushed out of charter schools, leading to an environment with higher-achieving peers. We compared the baseline achievement of students who were accepted and those who were not, and tracked the achievement of their peers over time. Surprisingly, we found that the baseline achievement of peers in public schools increased at a faster rate than in charter schools. It appears that low achievers are actually disappearing more rapidly from public schools than charter schools.
Do Boston charter schools spend more money per student compared to traditional public schools?
Based on the information we have, it seems that they do not. Massachusetts maintains comprehensive records on this matter, and the data suggests that the spending is similar. However, the allocation of resources differs. Charter schools do not have the same pension and health costs as traditional public schools, allowing them to utilize their resources differently.
What do you think about the argument that the gains observed in test scores may not accurately reflect educational achievement and attainment?
We address this concern through our findings related to college attendance. There are limitations to what we can measure when it comes to assessing the impact of educational programs. When critics argue that certain aspects go unmeasured, such as unquantifiable factors, it is a question that cannot be answered through research.
I would like to mention the work of Dobbie and Fryer on the Harlem’s Children’s Zone, which involved surveys to gather additional information on applicants. They were able to explore effects beyond test scores, such as reductions in teen pregnancy and crime rates.
Have you come across the recent study conducted by Dobbie and Fryer on Texas charter schools, which found significant gains in test scores and college attainment, but minimal income differences that were not statistically significant?
I have not yet seen that study. However, it is important to consider that young adults may still be in the early stages of their careers. When comparing the earnings of individuals who attended college and those who did not, the differences tend to be relatively small until individuals reach their thirties. This is a well-known phenomenon among labor economists. Traditionally, there is a point of transition, typically around age 28, and it could possibly be shifting to later ages.
Boston boasts a particularly vibrant intellectual environment, benefiting from a large pool of highly motivated, well-educated graduates. This could potentially contribute to its success. The educational institutions in the area tend to be independent, rather than being part of national chains, although this has both positive and negative implications. One example of an exceptional chain is KIPP, which successfully disseminates its practices.
However, while I have some theories, I do not have a definitive answer.
Overall, Massachusetts has a well-established policy framework in place. The state effectively monitors the charter school sector. On the flip side, the rigorous application process and thorough school evaluations are required. In contrast, Michigan has a more relaxed approach, allowing any post-secondary institution to establish a charter school. There is minimal oversight at the beginning and little supervision thereafter. The key to improving education through charter schools lies in striking a balance between autonomy and accountability, and Massachusetts has achieved this balance.
Interestingly, contrary to my expectations, Boston and Massachusetts prove that charter schools can excel even when traditional public schools are not performing well.
Indeed.
Now, let’s discuss your recent research on using free and reduced-price lunch as an indicator of poverty. Can you please explain your findings?
Traditionally, the achievement gaps between underprivileged students and their peers are measured by comparing those who qualify for subsidized federal meals with those who do not. However, this method is imprecise, as around half of all children now qualify for subsidized meals, which does not necessarily correlate with poverty. In contrast, only about a quarter of children actually live in poverty-stricken households. The reason for the discrepancy lies in the eligibility rules, which include children from households with incomes up to 185 percent of the federal poverty line.
Our goal was to utilize existing administrative data sets commonly used in research to develop a more nuanced measure of economic disadvantage. In Michigan, we discovered that although half of all children currently qualify for subsidized meals, only 14 percent have consistently qualified since entering the school system. This group of persistently disadvantaged children had considerably lower test scores compared to those who were never disadvantaged, as well as those who experienced occasional economic disadvantage. The achievement gap between persistently and never disadvantaged students was significantly larger, approximately 40 percent, when using these refined measures compared to the traditional ones.
Why should the findings of this research be of interest to more than just researchers?
Policymakers utilize eligibility for subsidized meals as a means to allocate funds to schools. Many states, districts, and federal entities use this measure as a proxy for assessing school disadvantages and to target resources accordingly. In Michigan, we discovered that schools with identical proportions of currently disadvantaged students varied greatly in terms of the percentage of persistently disadvantaged students. This means that we are overlooking the true differences in school and student needs. Our resource allocation could be more effective if we address this issue.
You also suggest that this finding may have implications for the use of test scores in evaluating teachers. Can you elaborate on this?
Certainly. If persistently disadvantaged students are concentrated in specific classrooms—a possibility that aligns with the history of segregation in the United States—then value-added test score measures may unfairly disadvantage those teachers. Essentially, certain teachers and schools face the challenge of instructing students living in severe poverty, which is not accurately captured by traditional measures. This research helps shed light on the depth of poverty and further research is needed to evaluate the practical implications for value-added assessments.
What other K–12 education research are you currently involved in?
Currently, I am involved in a randomized trial in Tennessee that examines the impact of dual credit courses in high schools. Dual credit courses, such as Advanced Placement (AP) courses, are well-known. However, the Tennessee legislature mandated the creation of courses for students who are not academically elite, expanding beyond the typical students who enroll in AP courses. These new courses include subjects like advanced algebra and greenhouse management.
Since the person overseeing research in Tennessee, Nate Schwartz, was once my student, we decided to implement the study as a randomized trial. I suggested this approach, and the state agreed. However, I do not have decision-making authority.
We have an ongoing partnership with the state of Michigan where we are currently conducting research on the impact of the ‘college for all’ curriculum on student achievement. Our initial findings, which are still in the form of a working paper, show no significant effects. However, we have another study coming out soon which examines the influence of a Michigan scholarship program on college enrollment.
So, the college for all working paper essentially found no positive effects, correct?
That’s correct. Our findings showed no significant impact.
Currently, a major part of my work involves collaborating with the state of Michigan to further enhance research and data capabilities. Our goal is to make the data more accessible to other researchers and to explore outcomes beyond just test scores. For instance, we are interested in assessing if charter schools contribute to the safety of students. Many parents claim that charter schools provide a safer learning environment, so to answer this question, we need to gather information about interactions with the juvenile justice system. However, we must balance this research agenda with the need to protect the privacy of the children involved.
One aspect that frustrates me is that those who often argue that test scores aren’t everything are the same individuals who resist collecting more data.
Are you planning to track the progress of Boston charter school students even after they graduate college to measure the long-term outcomes of attending a charter school?
Yes, we are eagerly awaiting the passage of time. Each year, the charter school applicants age by one year!
Have you encountered any challenges in carrying out this research?
Unfortunately, no. The recent paper we published focused on college enrollment and choice, but we still need a few more years to observe the full effects. On average, it takes about six years to complete a degree at a postsecondary institution, so we have to be patient. Moreover, we would love to be able to compare student information on earnings and other outcomes.
