This post has been updated.
Some schools in San Diego County, like Edison Elementary in City Heights, are achieving surprising results in closing the achievement gap. But looking at raw test scores alone, it would be hard to tell.
Roughly 95 percent of Edison’s students live near the federal poverty line. Most schools with that level of poverty score well below a proficient level in reading and math on California’s standardized tests. Based on the test scores and poverty level of all other schools in the county, Edison’s combined reading and math score should be 62 points below the state’s proficiency benchmark, to be exact.
But Edison’s scores break out of the statistical model. The school’s combined score is 5 points above the proficiency cutoff. In other words, it scored 67 points higher than its poverty level predicts it should.
Only three schools in the whole county did better, according to an analysis by Voice of San Diego and the Center for Research and Evaluation at UC San Diego Extension. VOSD and the Center for Research and Evaluation developed a new measurement that shows whether schools are meeting their expectations, based on poverty level.
Using the metric, our analysis found that across San Diego County, some high-poverty schools are doing much better than you’d expect in reading and math. Others are doing significantly worse.
The wisdom of using test scores alone to judge a school has become increasingly frowned upon in recent years. Over and over again, research shows that a school’s poverty level is one of the greatest predictors of its test scores. As poverty level goes up, test scores unflinchingly go down.
Most experts agree that out-of-school factors, including poverty, account for about 60 percent of variation in test scores. School quality accounts for just 20 percent or even less, studies have shown.
In other words, raw test scores don’t say what we think they do. They are telling us more about what’s happening outside the school than what’s happening inside of it.
Our metric controls for that. It shows whether schools are doing better or worse than they should be based on poverty level. That helps cut out at least part of the noise that’s getting picked up in a test score. It helps bring focus back to the signal of school quality.
“The interpretation is that [test scores] show the contribution that schools or teachers are making toward student achievement,” but it’s not that simple, said Dan Goldhaber, director of the Center for Education Data and Research at the University of Washington. “You could have schools with very middling results that are serving very disadvantaged kids and really contributing quite a lot to their education.”
Edison’s test scores aren’t much above proficiency. But the school serves families with very low incomes. Through that lens, having above-average results puts Edison far above average for similar schools.
The most pressing question about schools that defy the odds is how they do it. Back in March 2020, I spent weeks reporting on Edison. The school’s principal at the time, Eileen Moreno, said she pushed all the adults at the school to be “warm demanders.”
That meant she wanted them to demand the best from kids. But she wanted them to do it from a place of love, rather than harshness.
Joseph Murphy, a professor at Vanderbilt University who has studied schools that close the achievement gap, validated that approach. Schools must weave together a sense of caring and academic pressure like a DNA strand, he said.
Most of the schools that scored highest on our income vs. poverty metric were high-poverty schools like Edison. But some low-poverty schools also significantly outperformed expectations.
At La Jolla Elementary, only about 10 percent of students qualify for free or reduced-price lunch. Based on other schools’ performance and poverty level, La Jolla should have scored 52 points above proficiency. Instead, it scored 122 points above proficiency – meaning it performed 70 points higher than expected. That was the third highest score for all San Diego County schools.
The analysis also identified schools that performed well below expectations.
At Dehesa Charter – which was a primarily online school, authorized to serve students all over San Diego County – only about 27 percent of students qualified for free or reduced-price lunch. If it was achieving just average performance, it would score 30 points above proficiency in reading and math. Instead, the school scored 85 points below the proficiency cutoff.
Dehesa Charter closed in June 2019. But its old website now redirects to a new mostly online school called Dimensions Collaborative. Dimensions didn’t have enough available testing data, so it’s not included in our database.
Six out of the 10 lowest-performing schools are considered “dashboard alternative” schools. California’s school dashboard measures school performance, based on a number of factors including test results. Dashboard alternative schools are judged on the same measures as other schools on the dashboard. But they are also judged by several other metrics, including school climate, because they serve large at-risk populations, including students who have been expelled, those who have dropped out and re-enrolled, foster youth and others.
A note on how we performed our analysis: State testing was suspended in 2020, during the pandemic. Rather than use a school’s test scores from 2019 alone, we averaged their test scores from 2017, 2018 and 2019.
That means you’re getting a more stable look at a school’s performance. If it performed particularly poorly one year, but did well in the two other years, our analysis would account for it.
As a proxy for school poverty, we used the percentage of students who qualified for free or reduced-price lunch. We averaged the same three years’ worth of free and reduced-price lunch data to come up with a consistent three-year average.
We excluded schools that did not have three years’ worth of test data or had fewer than 100 students.
The majority of schools scored between 25 and -25, indicating they performed close to average, based on their poverty level.
Schools that scored above 50 and below -50 were extreme outliers.
Exactly how much education leaders should focus on test scores in order to make policy decisions is an ongoing debate.
Back in 2013, when Cindy Marten – now deputy secretary of the U.S. Department of Education – had just become superintendent of San Diego schools, she said it was shameful that America had become obsessed with a single test score as the only indicator of school quality. But later during public appearances, she never failed to point out that San Diego Unified performed better than other big-city school districts on the National Assessment of Educational Progress.
Goldhaber said he does believe tests are one important measure of school quality.
Tests should not be “the be-all end-all,” he said. “Oftentimes test scores are not particularly well-utilized [as a means of advising better public policy]. But you also need them for diagnostic purposes and figuring out what’s working and what’s not working.”
Looking at a school’s suspension rates or how frequently students are absent are also important ways to evaluate school quality. Looking at how safe students feel is another. Test score growth is Goldhaber’s preferred method. Controlling for a school’s poverty level is one more way to dig beneath raw test scores to begin to reveal how well schools are educating their students.
Clarification: This post has been updated to reflect that some of the lowest-performing schools in the analysis are considered “dashboard alternative” schools.