My kingdom for a reliable curriculum review

Started with the best of intentions, the piecemeal system of rating curriculum is frankly bananas

Happy Friday, Bell Ringers. In today’s letter, I wade into the dark and murky waters of the curriculum review process—and boy, do I have regrets. Kidding. [Sort of.] But in talking to people and reading what’s out there, it’s clear that curriculum review platforms are made with the best of intentions, but may not be getting the job done to help states and districts choose effective curricula.

Today’s newsletter is free, so you can really get what is going on behind the paywall. Consider supporting my work so it can keep going!

This week, I wanted to talk more about retrieval practice, because last week’s newsletter, Four big ideas on retrieval practice with Patrice Bain, made such an impact on me. Bain’s big ideas include helping people get over the idea that retrieval is just rote learning gussied up with a fancy word, as well as teaching parents how to use it at home—something she did every chance she got when she was still in the classroom. (Her tip on making daily flashcards on a piece of paper—just basically building a study guide for the end-of-unit test—is brilliant, and my 9th grade son is now using it for biology and history.)

I also took a little Bell Ringer survey (self-selected group, I realize) to hear what people had to say about how common retrieval practice is, especially in U.S. classrooms. I got so many interesting and encouraging responses, you can contribute your own thoughts and read them all here.

---

I’ll be attending this totally free webinar on How Learning Happens with Bell Ringer sources Amanda VanDerHeyden and Nidhi Sachdeva on November 5, if you want to join. It’s being put on by SpringMath Accelerated. Register here.

Sachdeva, who is a researcher and educator at the University of Toronto and York University, is going to be giving an overview of the science of learning, so especially if you’re new here, this is probably going to be a great place to get started.

---

My kingdom for a reliable curriculum review

Last week, Eric Hirsch, the Chief Executive Officer of curriculum review nonprofit EdReports, announced he is stepping down in June of 2026.

“We’ve helped create conversations about what high-quality curriculum looks like and supported so many in making more informed decisions about materials selection,” Hirsch wrote on LinkedIn.

According to their website, in the last decade nearly 1800 districts serving 18 million students have used EdReports to choose learning materials. Originally launched to help states and districts figure out which curricula were aligned with the Common Core State Standards, EdReports has become the de facto authority for many states in either choosing curricula or helping create lists of approved materials that districts are allowed to choose from. A “green light” from EdReports means to people who make decisions that those reading, math or science materials are going to be “high-quality.”

But as EdReports has grown, that prominence has also exposed some pretty big flaws with their process and product. Critics say that what started out as quality recommendations coming from a streamlined review process executed by groups of well-trained educators has, over time, devolved into more uneven reviews.

Reviews are only produced by groups of educators, and no researchers or experts are consulted. And reading and math programs that use techniques unsupported by research have received the coveted “green light,” while others that have been proven successful—often through randomized control trials—have not, creating confusion.

EdReports doesn’t rate materials on whether they have a successful track record in helping students learn, but rather how well they align with Common Core State Standards. And even after they amended their process to try to correct for some of these challenges, experts say EdReports hasn’t gone back and re-reviewed materials using the new criteria—so materials green-lit before remain so today.

“Relying on EdReports blindly is not reliable,” said education journalist Natalie Wexler, who wrote her own widely-read critique examining how EdReports reviews could be misleading to consumers last year.

That would be big news to practically every state leader looking for something good their schools can use. And when I asked a small group of researchers and experts whether a better review platform exists? Also, no. Maybe. With caveats.

I think it goes without saying that curriculum choice, what students actually learn all day at school, is important—research has confirmed it. Using learning materials that are well-designed, organized and sequential, year upon year, add up to more than the sum of their parts, according to cognitive science, because a store of knowledge in long-term memory is so crucial to the kind of thinking and analyzing we want students to be able to do.

But U.S. schools have just recently jumped back on the bandwagon of believing using a curriculum—any curriculum— is good in the first place.

Morgan Polikoff, a researcher at the University of Southern California’s Rossier School of Education, told me earlier this year that teachers creating their own lessons from scratch is “ingrained in the American culture of teaching.” A 2022 survey showed that 80% of teachers say they often create their own materials and 90% of teachers mix and match, using the curriculum and a bunch of other stuff they often find on Google or Teachers Pay Teachers. (This is kind of complicated, because obviously, some teachers assess that the approved curriculum isn’t any good, so they may be searching around for something better to use.)

It’s also apparent that the idea of a curriculum review platform is a good one. Not every single state leader or board of educators who help make decisions on what to use can be an expert in the efficacy of curricula, Common Core, what teachers will be enthusiastic about using, read all the studies supporting the science of learning, as well as be resistant to (or at least skeptical of) the sweet sweet marketing tactics of gigantic, influential publishers. That’s a totally unreasonable ask; they need a shorthand.

So schools need curriculum review, and the curriculum review currently on offer is lacking. I wanted to know two things: first, is this shake-up at EdReports a sign of a more comprehensive reform, and what could that look like? And second, are there other review sites doing this well, or better? If we dared to dream, what should the perfect review platform look like?

Let’s just get the first one out of the way: I reached out to EdReports to find out explicitly whether there are big changes ahead, but they didn’t respond to a request for comment.

Each review platform looks at one specific aspect of curriculum—and that’s a problem

EdReports dominates the market of curriculum review, but there are others doing this work, also nonprofits like them, also working hard to be objective in their ratings. That’s not the problem, said researcher Holly Lane, director of the University of Florida Literacy Institute and author of the UFLI Foundations phonics program.

The problem is how the different platforms have targeted specific curricular aspects to review, giving a kind of lopsided inspection. “Each of the entities that does these reviews does a good job at it,” she said. “But none are comprehensive enough to be informative for people to make decisions in schools and districts. Each one takes a different angle, and looks at different things.”

EdReports is mainly looking at Common Core alignment. Federally funded What Works Clearinghouse does efficacy reviews of curricula—something that’s desperately needed—but makes them really hard to understand. “They’re great at explaining something to other researchers, but not great at breaking it down in a way that a practitioner can understand,” Lane said. The Knowledge Matters Campaign is looking at knowledge-building, and The Reading League tool evaluates evidence-based reading materials.

To make things more complicated, Johns Hopkins’ Evidence for ESSA, which came about to support districts implementing the Every Student Succeeds Act, rates the quality of the evidence being used to test out different curriculum to see if they work. The quality of the studies is really important, especially since education research standards can be wishy-washy and downright unreliable—but also somehow makes trying to find an evidence-based curricula harder, because the quality of the study is quite different from the quality of the material itself, i.e. whether or not a lot of children learned when using it.

Lane makes this point with an example: here is the Evidence for ESSA review of Leveled Literacy Intervention (LLI), a small-group tutoring curricula made by Fountas & Pinnell, which has been called out for recommending teaching practices that aren’t supported by evidence.

(Image source: Evidence for ESSA website)

The green “Strong” rating is given to the quality of the studies used to test this program, Lane said. The effect size for struggling readers, however, is +0.13—a very weak effect size, so small that it’s in a category below having any instruction at all.

That’s extremely confusing. “There are very strong programs out there that, to date, don’t have very strong evidence,”—Lane said on a recent podcast, meaning that the program is following the science on how material should be taught, how it should be assessed, etc., but maybe hasn’t been run through a randomized controlled trial. Then there are programs that have strong study quality, like LLI, “but they’re not strong programs.”

Program evidence is often a low priority for decision makers

Adding more complexity to this stew is to understand what state and district leaders are looking for, what they consider important, when choosing curriculum in the first place. If the goal is to get more districts using materials that are supported by scientific evidence and help large groups of students learn a lot—do decision makers put a high priority on evidence and research when choosing?

In one fascinating survey conducted last year, researchers asked school administrators to explain their thinking when weighing one program over another, and to rate a set of considerations in order of importance that included factors like a need for the program, educators’ buy-in, and the evidence base of a program.

“Research evidence was the lowest weighted piece of information,” said lead researcher Cortenay Barrett Morsi, a psychologist at Michigan State University. “The evidence base was the least weighted. The most weighted considerations were things like community buy-in, do teachers like it, do they think they need it, and do they want it.”

“That makes a lot of sense when we think about why there is so much variability in the programs or interventions or supports that are available to students. There’s going to be a lot of variability in what people like,” she said.

Trying to tease out how administrators thought about “research” and “evidence-based” programs, Morsi found that those words were often used broadly to mean several different things. “People actually conceptualize ‘research’ differently,” she said. Asking about research data, she said administrators often said, “I did my research,” or “my teachers did their research,” and what they meant was they gathered information about that program from others—neighboring districts, for example, to see how well they liked it.

“When you ask them, ‘okay, well, what do you think about this published randomized control trial?’ That’s a little bit of a different thing—that’s the part of it they’re skeptical about.” Many administrators said research-based curriculum they’d used often “didn’t work.”

“They’re not necessarily like, ‘oh, I don’t value research.’ They’re just valuing this other use of the word, if that makes sense,” Morsi said.

Administrators put high value on whether their teachers liked the program and found it usable, and were concerned about “fit”—whether the program aligned with the values and perspectives already going on at the school. “If a teacher has been trained and they are using a very heavily inquiry-based approach to everything they do, then you bring in something [explicit and systematic] like SpringMath—it really doesn’t fit. It doesn’t go with what they’re doing all day long, right?” Morsi said.

This makes a lot of sense, considering how many districts use and rely on EdReports—many administrators value educators’ perspectives, and they may not necessarily be thinking about the evidence base for a curriculum’s use.

In a perfect world…

Lane said she believes the answer to all these problems is an FDA-like entity, government-run, that calls balls and strikes on which curricula pass a minimum standard of evidence that children are going to learn while using them. The people who would run this imaginary Federal Education Quality Administration [my creative name] would have expertise in the science of learning, what stands for quality research, the standards, and curriculum design, as well as the realities of classroom teaching.

Individual districts don’t have the capacity—nor should they—to evaluate every curriculum they’re using in classrooms, going to multiple websites and platforms to put all the pieces together. “It’s like expecting doctors to conduct their own drug trials, it’s not doable,” Lane said.

Reviews based on whether certain curricula helped children grow, like the What Works Clearinghouse produces, are good measures of learning but they have “no teeth” to them, Lane said. A federal minimum standard could force publishers to include basic science in the quality of materials if they wanted to get them on the market. “There’s [currently] nothing to say, like we do with food and drugs, you cannot sell this product to the American public, because it doesn’t work,” she said.

Having a government-run, FDA-like review platform might also help eliminate some of the current curriculum reviews’ biggest problems, like conflicts of interest in funding. When the same funders support curriculum review sites and other big-name education platforms, some suggest that has kept prominent literacy and math advocates from having a more open conversation about review sites’ problems.

“No one will speak out about EdReports,” said Karen Vaites, founder of the Curriculum Insight Project. Vaites, an original supporter of EdReports, has been trying to draw attention to the need for reform for years. “One of the key problems is self-silencing around EdReports, people won’t speak truth to power. They are afraid to put their funding at risk, that’s a well-understood problem.”

Even amid all these challenges, other countries are eyeing the curriculum review model to help bolster their own education reforms. When I spoke with Wexler recently, she was preparing for a speaking tour of Australia, where they are thinking of—you guessed it—launching their own nonprofit-style curriculum review platform. She’s going to address how it’s going in the U.S. in her talks to Australian educators and leaders.

“I think they need to reconsider the model [of EdReports],” Wexler said. “Teachers can have input, that’s fine, as long as they’re trained carefully. But even if you’ve done those things, they need expert oversight and guidance. It makes sense to have experts who really understand what makes curriculum good.”

In addition, a lot has changed in the curriculum space and in general knowledge of cognitive science and psychology research over the last decade, Vaites said. Reviews need to reflect those shifts, like whether a curriculum includes whole books or only excerpts; or including how much of learning takes place on a screen, considering the growing body of evidence that reading on paper produces better reading comprehension.

EdReports could take this opportunity, with Hirsch on his way out, to do a whole-cloth reform.

“I would love EdReports to be reformed. It would be perfectly possible for Ed reports to say we’re going to materially change our process,” Vaites said. Review quality would improve if the review team included both experts and teachers, and they not only reviewed materials but also went out into districts to see how they were being used in real districts.

“I don’t expect to see that happening. And the biggest reason I don’t expect to see it happening is states aren’t demanding it, funders aren’t demanding it. If people demanded it, it’s possible. Nothing about this is impossible. It would take resources and leadership,” she said.