S5E8 – The Impacts of Testing Our Kids and Ranking Our Schools (Part 2)

Andrew: Welcome to the Integrated Schools Podcast. I'm Andrew, a White dad from Denver.

Ali McKay: And I'm Ali, a White mom from Seattle.

Andrew: This is part two of "The Impacts of Testing Our Kids and Ranking Our Schools." And while obviously no one could ever replace the late, great Courtney, my former co-host, I also recognize that having people to think through these topics with me and help me interview our guests is incredibly important. So, for the next couple of episodes, I'm going to be joined by various people from the Integrated Schools community. They’ll share their thoughts, their stories, and also help me interview our incredible guests.

Today we have Ali. Ali, tell us a bit about yourself.

Ali McKay: Sure. I live in Seattle. I've lived here for 21 years now. I've got two kids. They are eight and nine and they go to a global majority school near our neighborhood. I got involved in Integrated Schools about two and a half years ago, and I am our Seattle chapter lead. I also serve on the Parent Advisory Board.

Andrew: Yeah. You've been a huge part of the organization for certainly as long as I've been around, and you wrote what is our most widely read blog post called “The Problem with GreatSchools”, and I’m wondering if you can tell us how that piece came to be.

Ali McKay: Yeah. We actually switched schools, so we first started sending my older, now nine year old to our neighborhood school, which was actually pretty White. It was about 75% White at the time. I think it still is. It's often called, "private school of the public school system." We didn't actually know that so much when we bought the house.  We bought the house way before we had kids. And so we just sent our son to the local school, and kind of didn't think more about it. And it was, it was fine. Then when my younger one was ready to go to school, we had an opportunity to tour another school very close by, that was not the "private school of the public school system." I think at the time, maybe it was rated a three or a four on GreatSchools, but we had an opportunity to tour it because of a program it had for my younger son with low vision. And we did. And it was a really special place.

We felt that it looked more like the world. It had a lot of different kinds of diversity. It had both racial diversity, economic diversity, as well as different abilities. There are some kids receiving special education services, my sons would see kids with white canes, for vision. They would see some kids in wheelchairs, and we felt that there's a lot of different ways that they needed to understand that the world wasn't the pretty White bubble where we were. So, it was a tough choice, but we decided to move my older one and start my younger one there that that year. That was three years ago now. And it was after that move where we're going to the school that, that the neighbors would say, “Oh, well that school, you don't want to send your kids there. It's, it's not a good school.” But we were there and I was like, this is, this is nice. This is a nice school. It's good. You know, it's got some challenges but it's got some great stuff going on. And I just couldn't understand where this huge disconnect was.

And particularly like the ratings systems. Like, what was going wrong here that the school was by the account of that number, like a bad school. And so I actually did some Googling, to see, like, what are people talking about integration? This seems like it's a good thing. Like, why aren't we talking about this?

And I found Integrated Schools and got to start speaking to Courtney with those supposed to be 20 minutes, but turns into an hour and a half conversations that everyone talks about with her. And she and I ended up talking a lot about the ratings. And so she said, “Why don't, why don't you write something?”

And, I did. And she actually worked on it a lot as well. So I give her a lot of credit and other folks who helped edit it. And we put it up and I guess it just struck a nerve.

So, I'm, I'm glad that it's, it's out there. I'm glad people are starting to think about this stuff.

Andrew: It's a great piece. There's a link to it in the show notes and I’m particularly excited that you agreed to come on this show today because we’re talking all about GreatSchools.

Ali McKay: Oh, I'm glad I could join you.

Andrew: So, last episode, we touched on GreatSchools a bit and the other aggregators of school ranking data. But today we're really going to dig into the ways that the GreatSchools ratings can impact both schools and, you know, in some ways, more importantly, the decisions that people are making about where to live and where to send their kids.

Ali McKay: Yeah, exactly. GreatSchools, on their website, they say that they are,, “empowering parents to unlock educational opportunities for their child.” And they state their commitment to helping marginalized and Black and Brown communities find better schools, people who aren't being served well by the school system.

So I think these kinds of conversations. They're a jumping off point. They are a starter for a lot of really interesting and challenging issues.

Andrew: You know, in the, in the past we have avoided even mentioning their names on the podcast, at times because, you know, I think in many ways it feels like GreatSchools contributes to the smog that we talk about, right, the problematic ways that we discuss “good” and “bad” schools, but they are ubiquitous and, and they are meaningfully impacting everything from how principals prioritize their time in the building to gentrification and housing prices.

And so, back in December, we saw an article come out on Chalkbeat called “Looking for a home? You've seen GreatSchools ratings. Here's how they nudge families toward schools with fewer Black and Hispanic students” by Matt Barnum. And we decided it was probably time to dig in.

Ali McKay: Yeah, it was really exciting. Matt is a reporter at Chalkbeat and I learned that he also is a former teacher himself. And so he just has a really amazing grasp on current educational research, but he has a personal knowledge of it as well. I think that really informs his work and it makes it really quite impactful.

Andrew: So we reached out to Matt. He was generous enough to come on the show and tell us what he learned. And Ali was generous enough to join me in the conversation. So, let's take a listen.

---------------------------------------------------------

Andrew: We can just start by you introducing yourself.

Matt Barnum: I'm a national reporter with Chalkbeat, an education news site focusing on national education policy and politics issues and often writing about research and data in education.

Andrew: How did you, how'd you get into education reporting? Why do you care about education?

Matt Barnum: Well, I started my career as a teacher, and then worked at a policy organization. And then I was really frustrated because I became so fascinated by this world of education research that felt very separate from the world of education policy, oddly. And I realized that there were all these people out there whose jobs were literally to find empirical answers to key questions in education.

And yet those answers were not often times fundamentally shaping the debate. I mean, of course, I don't think that empirical answers answer all the key questions or even necessarily the most important key questions about education, but they seem very relevant. And so I sort of became interested in writing more about research because I felt like it was often lacking, in the debate.

And so I've been doing that for now almost five years.

Andrew: That’s great. Maybe you can tell us a bit about this GreatSchools piece. And maybe the first place to start is: why does GreatSchools matter?

Matt Barnum: Yeah. So I became interested in them when I realized that if you Google virtually any school in the country, one of the first things that is going to come up is their GreatSchools profile and their 1 through 10 rating. And that was really striking to me, and they get tens of millions of views on their website each year, and I follow often debates about school rating systems in districts like Denver and then states across the country. And there are all these huge debates: How should you rate schools? And in a way, we have a de facto national school rating system.

GreatSchools is one of them, you know, not the only one. There are others, but they are probably the most influential school rating system in the country. And so I was like, well, this deserves more analysis to see what is going on underneath the hood of these rating systems.

Now, GreatSchools came about in the late nineties and they started in California and they started rolling out across the country and now they're in every single state and most public schools in the country get a 1 through 10 rating.

Andrew: So I guess in some ways the local conversations about how to rate schools become, maybe not irrelevant, but less relevant now that every school has a GreatSchools rating. I guess depending on who is actually looking at those grade school ratings.

Ali McKay: So Matt, do you have any sense of who actually uses GreatSchools, the demographics, race, class, et cetera?

Matt Barnum: That's a good question, GreatSchools has provided me with some data on this and they have some data showing that a large share of their users are low income families. Now I think it was not the majority and it's a difficult thing to say, well, what does it mean to be a user?  You know, their definition, I believe, was based on accessing their website. 

Ali McKay: Right. Which implies that it's not looking at the use through Redfin, Zillow, et cetera.

Matt Barnum: That's right. Yes. So I asked for data on the users of Zillow, Redfin, and I didn't get data.  GreatSchools didn't have access to that data. Zillow did not offer me anything. So, I think that is an important point, we would expect that folks who are accessing these real estate websites, you know, maybe that's a more affluent demographic and there's a good chance that more people are seeing, at least the first cut of the GreatSchools ratings, on those real estate websites than on GreatSchool's website itself. But I did speak to low income parents who did say “I used the rating and it really helped me find a better school for my kid. I was looking at schools all around that were ones or twos, and I found a school that was a five and that may not sound great, but that actually made a very big difference for, for me and my family.” And so GreatSchools, they can point to demonstrable efforts to reach low income families and families of color so they can use their ratings. 

Andrew: So there's this fundamental question of who is seeing the ratings? And we have, I guess, some data showing that, at least on the website, there may be some usage by a diverse set of families, but one could certainly imagine that that proportion shifts more heavily towards wealthier families.

If you include the people seeing these GreatSchools ratings through other places, they show up, like real estate websites.

Matt Barnum: That’s right. I think one other point is that access to the ratings, access to the website, and this is another area where GreatSchools has made efforts to make sure their website is clear and accessible. If you look at GreatSchools, compare that to the average state rating site, GreatSchools is much more easy to navigate. Their website fully translates into Spanish in an easily accessible way. But there is a very important distinction between being able to access the ratings and being able to actually use and leverage the ratings.

So if you're a family who doesn't have the means to move into an area where the higher rated schools are, and we know the higher rated schools are in more affluent areas serving more affluent students, you can see where the 10 rated schools are all you want, but how far does that get you? Right? So, now, again, I talked to low income families who did use the ratings to choose among schools in their area, but there is a limit of how far that can get you.

Andrew: Right, right. So they say that their data is provided in order to help parents choose better schools. Is there any research to indicate that that's actually effective or that that's true at this point?

Matt Barnum: So, there is a study, and this study hasn't been fully published, but  there is a summary of it that exists done by a credible researcher, Peter Bergman of Columbia University, that looked at what happened to families who used federal housing vouchers when they were given access to the GreatSchools rating as part of the housing voucher website, versus families who use federal housing vouchers, but didn't have access to the GreatSchools rating on the website. And so that study found that families who did have access to GreatSchools ratings chose schools that were more highly rated on GreatSchools. So families offered GreatSchools ratings ended up in areas where schools earned an average rating of 4 as opposed to 3.7.

So the rating went up by 0.3. So that's, that's small, but it's real, and was statistically significant. The study is not able to follow those students to see whether that actually changed their achievement trajectory. Did they actually learn more? Now, to the extent that the ratings are accurately capturing school quality, we would expect that to translate into gains in student learning.

Andrew: And, and that's a good point, right? Like if the ratings are actually capturing school quality is an “if”, and so I wonder if, if it would be good to talk a bit about what makes up these GreatSchools ratings. And my understanding at least is it's primarily test scores, but, at least for GreatSchools, the rating is largely based on proficiency scores versus growth scores?

And so maybe we just start out with a quick, what is proficiency and what is growth?

Matt Barnum: Yeah. Right. So, proficiency is the percent of students who score proficient, however the state measures the proficiency bar, on a state exam. So in every state they have their own exam, and they'll give a student a score and they'll say whether that student clears the cut score for what they deem as proficient in that grade and subject.

Andrew: And that's like a snapshot in time. This kid, regardless of where they showed up - do they know the material they're supposed to know for third grade or don’t they?

Matt Barnum: Right, right. A sort of a test of mastery. It doesn't tell you where a student came in and how much they learned over the course of the year. A number of researchers have made efforts to say, okay, well, how can we actually measure how much students have grown or learned year over year. 

Andrew: A growth measure.

Matt Barnum: And that's through complicated statistical metrics to say: essentially compared to other students who started out at the same level of this student, how much did this student improve over time relative to other students?  

Now, I should say that there are complications and controversies in measuring growth.  One, it's a, it's a relative measure, in almost all cases. It's saying how do you perform relative to other students in the state or, or in the district. Two, it’s still, it still relies entirely on test scores. So if you're not a fan of using test scores, using growth doesn't help. It's still is just based on test scores. But it's almost certainly a better measure for how schools are contributing to student learning. Though even there, it still may not be a perfect measure of the school's independent contribution. And growth ratings are much less tied to student demographics than raw proficiency ratings. So you'll often see schools that have low proficiency ratings have high growth ratings, and the correlation is just substantially lower.

Andrew: And so for a long time, GreatSchools was using proficiency as basically like all of their ratings, right? And if your school had 80% of kids who passed the cut score for third grade math, then you get a good rating, and if only 20% of your kids pass that score, then then you don't, right?

Matt Barnum: Exactly.

Andrew: But given the strong correlation between proficiency and demographics and the fact that kids enter school at different levels, we have the emergence of growth ratings and, and this emergence of growth ratings, and I guess the underlying correlations actually caused GreatSchools to adjust their rating system, right?

Matt Barnum: Right. So in 2017 they revamped their rating system. Their prior rating system was almost exclusively based on the test score proficiency rates of students at the school. And so, so they revamped that and at the time, one of their officials said that the new rating system would “more accurately reflect what's going on in a school besides just its demographics.” Which, which was a really interesting acknowledgement that the prior rating system reflected to a large degree, student demographics. So I was really curious to what extent, the ratings were correlated with demographics today, after the overhaul. So I worked with our great data team, a partner, a staff member of ours, Gabrielle LaMarr LeMee, who did the data analysis and did a really great job.

And so we found - Gabrielle found, that there continues to be this really strong correlation  between school demographics and GreatSchools ratings, meaning more precisely, schools with more low income students, schools with more Black and Hispanic students get lower ratings - substantially lower ratings on average. Though we did find that that relationship had gone down modestly after GreatSchools changed the rating system in 2017.

Andrew:  And I, I guess part of that modest improvement came from adding growth scores to the ratings.

Matt Barnum: Yeah. So GreatSchools does include growth in their formula, where it exists, where states create growth ratings for their schools, which isn't all states. But where it exists, GreatSchools uses that in their formula as about 20 to 25% of the rating or about half of what they use as for proficiency.

Andrew:  You know, I think one of the things that I thought it was really interesting from your piece was the analysis that showed that if growth scores were all that mattered, if GreatSchools got rid of proficiency entirely and only looked at this sort of - what are schools contributing to test scores over the course of the year - measure. This growth score. You have this sort of shift from a number of predominantly Black and Brown or low income schools, their rates going up. And then the thing that I think was super interesting was that you also have this number of high performing low poverty schools that they would actually see their ratings go down because their growth scores aren't actually that high.

Matt Barnum: Yeah. In Denver, for instance, the schools that serve the most affluent student body and the fewest Black and Hispanic students in the Denver area, 78% of them were rated above average by GreatSchools.

Whereas if you just went on growth, it went down to 51%, which was a pretty substantial drop. And so that presents a really interesting fact that some of these most affluent schools - the schools that are in the public imagination, “the best schools”, they don't look as good on these growth measures, how much they're helping students learn, as they would on proficiency. And that's not a total shock. There have been studies on, for instance, the most selective schools in the country. The schools that families are clamoring to get their kids in. Studies that have looked at how much these schools actually help students learn or get into or stay in college. They actually don't find a huge effect.

And so often some schools are sort of coasting on the level that students come in at rather than how well they're actually educating their students.

Ali McKay:  Matt, as a teacher, I imagine you have strong feelings and opinions about standardized tests.  So can we use the tests or we should at least think about them in a more, in a more deep way.

Matt Barnum: Well I try to mostly put on my like, journalist hat and my like, reading of  the research hat. And I think there's a lot packed in there, right? And so one question is, to what extent  does testing, and high stakes testing, shape instruction in a negative or potentially positive way?

And, there's some mixed research on this. For instance, there's evidence that teachers will teach to  questions that are on the test. Now, if that's a really good question, if those are the things you want students to know and be able to do, maybe that's not a problem. But if the test does not cover the breadth of material that you want students to know and be able to do, maybe that, that is a big problem.

And then there are these questions about, well, do tests to actually predict, long run success of students, these long run outcomes? And here there's some conflicting research and at least just in the aggregate, test scores do predict long run outcomes. But there's a question of how strongly do they predict those long run outcomes? Are they predicting long run outcomes because they're highly correlated with race and student affluence, which also predicts long run outcomes? So, that's a very long-winded way of saying research may not be able to fully answer that question.

And I think part of that has to do with your values and what you think about tests and what they're measuring.  And it is certainly clear, I think from a research and experiential perspective, that test scores are not fully capturing what we want schools to do for kids and to impart on kids.

Andrew: Are there any places that are capturing some of those other things? I think about all the other things that people think about being important about school and the school experience. Is there any good data on that?

Matt Barnum:  I think, I think there's some decent data. There are survey measures of students and teachers. There are measures of attendance, which I actually think measures of attendance are an underused measure by researchers to, to measure school. We know that schools contribute to whether students show up to school. There are measures of discipline rates, how many students are you suspending? To what extent are there disproportionalities? Now all of these sort of output measures, we really have to wonder - if we end up attaching stakes to these measures, do they corrupt the underlying measure? Do they make schools hide their suspension rates? Do they make schools inflate student attendance in some way? 

And then there are also measures of inputs, like how experienced is your teaching staff? Do you have high teacher turnover? How racially integrated is your school? How much does your school spend per pupil? But those also have some, some challenges too because some of them are just outside the school's control, right? So if you rated a school based on how much it was spending per pupil, you're not measuring the work of the school, you're measuring state policy. If you measure them based on whether they have high or low teacher turnover, well does that encourage a principal not to have any teacher turnover when maybe some teacher turnover is good and healthy, whereas others are bad? You know, it's really hard to parse some of that stuff.

So I think all of these measures have real complications.

Ali McKay: I think about my own school, which is rated a 2 now, and one of the first things our teachers and principal look at every year is the student survey. How are students feeling? Do they feel safe? And I'm really glad that that's like the first thing that they look at. And then they do look at the test scores and the growth in the math proficiency and the English language arts proficiency.

And we actually had huge growth last year. It was amazing. And I think they should be proud, but that GreatSchools score is still says 2. It doesn't even show up when you search Seattle.

So I hear all the data that you're talking about and then I just translate it to my school and how there's so much that I could tell people about our school that a single number absolutely can't capture.

Matt Barnum: Right. Yeah. I mean, so to your point, that was something I noticed and I mentioned in my reporting. There was a school in Denver that had a really high growth rating and it had gotten accolades from the district and from the state for its really high levels of growth for its student population, which was predominantly low-income and Hispanic.

And if you type in the zip code of that school in GreatSchools. That school is not one that actually shows up upon first pass. In fact, there's not a single school that has that zip code that shows up because all the schools in that zip code are not highly rated.

Now, I should say you can sort pretty easily. You can look for, for other schools...

Andrew: Yeah but you, you have to know that that's something you want to do and that...

Matt Barnum: That's right.

Andrew: That you want to go after. I mean, my school is also a 2 incidentally, and yeah, you put in my zip code and it actually… y’know probably three pages of search results before you actually find my school.

Matt Barnum: And I mean, I think one, one interesting question I have is: how much does the particularities of GreatSchools, both the ratings itself and the structure of the search, shape decisions? And I, I don't know the answer to that. I think that's an empirical question and I would love to see research on that.

But it is interesting to think about how seemingly minor decisions about what are the schools that first come up when you search, could shape where families end up sending their kids.

To your earlier point Ali, about this like, captured in a single number issue, I think that is very fundamental to not only the discussion about GreatSchools, but the, just the discussion broadly about school quality and what makes a good school.

And I think a lot of people, educators, parents, really recoil at the idea of a single number to capture something as complex as a school. Right. On the other hand, I think also a lot of people want some summative thing. Like, is this a good or a bad school? And there is demand for that.

And those two things just run up very fundamentally against each other. Though those are two very inherent tensions that I don't think there's ever going to be an easy answer to. We're, we're never going to solve that tension.

Ali McKay: Well, I think about context. For example, what was the history of testing in the first place? You know, the eugenics movement, there's been a lot of research and work on, what were IQ tests designed to measure in the first place? And that history matters. It may not be the same thing that's happening now, but that history matters.

And people want to know “what's a good school?”. But in the same way, people wanted to racially profile on NextDoor, and NextDoor said, that's not okay. Like, we recognize that we are within a system and within a culture, in a society that has some problems.

Matt Barnum: How does, I actually don't know. How does NextDoor do it?

Ali McKay: So my understanding, and I only went through the website quickly just to test it, is when you want to report suspicious activity or a crime, you have to answer a couple additional questions. And they, they say something like, “what is the behavior that the person's doing that's concerning”? Not “what are they just wearing”, or “what color is their skin?”

But, I think they put some hurdles up before people can just fire off something that goes to all the neighbors. And they worked with a researcher, she studies bias, and what she had said is, you need to put some space between people's immediate bias reaction and then their, their output, what they say, what they do. And so that's what NextDoor tried to do, is build some space and time, maybe to interrupt that bias a little bit.

Matt Barnum:  Got it. That's interesting.

Andrew: So NextDoor saw the impact of their platform and chose to create that buffer between the bias reaction and posting. And it seems like GreatSchools either doesn't see a problem or maybe doesn't see it as their role to influence this bigger question of what do we think of as good and bad schools.

Matt Barnum: So, yeah, and obviously I don't want to speak for GreatSchools, but I think it is accurate that the public's conception about what is a good school and a bad school is highly racialized, and I think we cannot get around that. I think that's just an empirical and historical fact.

And if we ask the average person, “what is a good school?” You think about a school in the leafy suburbs where most students are White and come from affluent backgrounds. 

And “what is a bad school?” We think about an “inner city” failing school, right? And by “inner city” we of course, mean schools that are predominantly made up of Black and Hispanic students, and that long predated GreatSchools, obviously. Right? But a question is, to what extent does GreatSchools want to disrupt that? Does it see its role as disrupting that? And I think our data analysis shows that the very least, they're not fundamentally disrupting this, this conception that, that exists.

Andrew:  And in some ways, contributing to maintaining it. There's some balancing act that GreatSchools is, is doing. Like you said, these conceptions of “good schools” and “bad schools” definitely predate the existence of GreatSchools. And at some point, like, the success of GreatSchools; I wonder if there is some bit of it that is tied to the White privileged desire for a “scientific explanation”, a “scientific reasoning” to make the choice that that we were going to make anyway, which was for a Whiter and wealthier school. And if you have to, you know, confront this as “I'm making this choice because it's a Whiter and wealthier school”, that feels really awkward.

If you can say, “Oh, I'm just choosing  the ‘best school’ I can get for my kid”, then it's an easier choice to make. And I wonder how far GreatSchools could push before, particularly the White and privileged people who are, who are largely responsible for driving up real estate prices and contributing to residential segregation, before they just say, okay, we don't care about your GreatSchools data anymore because it doesn't conform to our preconceived notions.

Matt Barnum: I, yeah, I think that is a really important point. I think there;s a question of how elastic, how malleable are family school choices and to what extent is GreatSchools helping families choose a school versus supporting preexisting choices that would already happen. But then that goes back to the question of, well, are families using race or are they using test scores? Or are they using race, but pretending to use test scores? But then it also becomes difficult when our conception of what a good school is is so wrapped up in both race and test scores that are highly correlated with each other, right? And so sort of untangling all these different influences is very difficult.

There was a study that I talked about in the piece, and this was a hypothetical study, so this was sort of a laboratory type experiment, where participants were given a vignette and asked to choose if you hypothetically were a family moving into an area, where would you move into?

And so in one condition they gave them information about a school district’s test score growth, and in another condition, they just gave them information on demographics and then another condition they just gave them information on proficiency. And it turned out that people who were White and more affluent were more likely to consider moving their family in this hypothetical scenario to a more racially diverse and racially integrated area when they were given information about schools test score growth.

So again, that is what people say in a lab setting. We don't know how far that's gonna get us in real life, but I think that does give some reason to think that families’ preferences might be somewhat malleable.

Ali McKay:  Well, and we have that study that came out. When GreatSchools comes out in a city, the residential segregation starts to increase. Like a house with a school that's highly rated, will go up in price more than a house that's in a zone for a school that simply doesn't have a GreatSchools rating, even if the test scores are actually the same.

So in theory, the school’s test score proficiency is the same, but the house’s value starts going up differently because there's a GreatSchool number attached to it.

Matt Barnum: That's right. And so, one: that finds evidence that families’ choices are malleable and that they are really looking at, at least some of them are looking at GreatSchools ratings and that that was going to bid up housing prices in more affluent areas. And also potentially shut out lower income families. And in that sense, increase residential segregation, and although the study doesn't look at this, potentially increase school segregation. 

Now, the standard caveat is - and I cover education research. This is one study. I think the methods are rigorous. But I'm looking forward to more research to understand the full effects of GreatSchools ratings. And the other thing is, this was based on the older rating system that was, that was more strongly correlated with race and income. We show that the newer rating system is still strongly correlated, but slightly less so.

Andrew: What might work better? I mean it seems like you come down pretty clearly on growth being better than proficiency. Not great maybe, but certainly a better measure of school quality.

We know it's hard to find reliable data about other things that parents really care about and we know, as you mentioned, that once people know they're being measured, it affects the incentives in schools.

Is there something that'd be better? And is there a compelling reason to think that we need to rank schools, that we need to be doing something that is not more like a, pass, fail - accreditation, and that is, there is actual value to be had from ranking schools.

Matt Barnum: So let me answer that question first, which is, and I don't want to say yes or no, but I do think as a descriptive matter, that is going to happen. And I think that the genie is out of that bottle. You know, there is demand for it. There is data that does it. And if GreatSchools disappeared tomorrow, other entities would, would fill that space easily. And so it's hard to imagine getting to a place where, where that doesn't exist, especially in our country where there are mandated tests in grades three through eight.

And I'll note that here it's many civil rights groups that have argued most strongly for testing and test based accountability, and they pushed to maintain 3-8 annual testing in the most recent, authorization of the federal education law. So I think the, the cat, the cat is out of the bag. For better or for worse, that this is the world we live in now.

I think the question of what to do instead is definitely not an easy question and I hope people didn't come away from my piece thinking it was an easy question or thinking that there were clear answers.

What we showed was that growth was much less correlated with student demographics and that researchers agree that it is a better measure of school's independent contribution to student learning than proficiency. 

So potential alternatives without endorsing any of them would be: you could weigh growth more. You could weigh it a lot more. That doesn't solve the issue of: are test scores are good or the best measure of school quality?

You could throw a lot of other measures into the system. You could say, let's use attendance. Let's, let's use suspension rates. That is sort of, we don't have many other measures that are widely reported. You have some cities that do surveys and others that don't. Those measures themselves may be highly correlated and probably are highly correlated with student demographics. Maybe slightly less so, maybe not. 

That may solve the breadth of measures issue, but it doesn't solve the relationship with demographics issue. You could have a measure that includes the degree of integration, particularly for schools that are predominantly White and affluent, if you think that's something you want to, to nudge families to more integrated schools, or if that's something families want that isn't actually captured in the ratings.

You could, if you wanted to avoid the like one be-all-and-end-all score, you could say, well, let's before we rate a given school, ask families what they most care about. And so you could say, like you enter GreatSchools, first take this survey about what are the four most important things you're looking for in a school?

And then based on the survey, we'll create this own personalized algorithm for you. Now a problem with that is we still have the, the families will be like, well, “I care about how warm and loving the school is”. Well, we don't have that data. Redirect!

Ali McKay:  Would you like test scores?

Matt Barnum: Right. So it doesn't, you know… We still have the, this data issue. We also have this issue with, you know, if what many families are looking for are schools with high proficiency rates and few Black and Hispanic low income students. You know what - that's not going to solve the issues that are raised if GreatSchools, to a large extent, is giving them what they want. The, the survey is, it's going to also give them, give them what they want, but whether that's socially desirable is a separate question. 

Ali McKay:  At least it would be more intellectually honest...

Matt Barnum: Right

Ali McKay:  But does that move the ball forward? Probably not.

 Matt Barnum:  Now, obviously in this scenario of if GreatSchools were like, we're going to create your own personalized algorithm, they could constrain the choices of, you know, they're not going to ask families like, well, do you want uh...

Andrew: How many Black kids do you want at your school?

Matt Barnum:  Right! Now they can ask things that are like about proficiency test scores that are highly correlated with that. But they can definitely potentially frame those questions in a way that also does some, does some of that nudging, for better or for worse.

Ali McKay: Yeah. And the way that NextDoor does nudging for people around the racial profiling.

Matt Barnum: Mm...

Ali McKay: So Matt, considering that schools’ contribution may not be what we think it is for these test scores.  I think Jack Schneider maybe has said that - and other researchers, that it's something like 20% of a test score can be controlled, you know, by the school. Is there an argument to be made that this data should be used by different demographic groups differently?

Like if you're White and affluent, you would look at it in a different way?

Matt Barnum:  That's an interesting question. I mean, I think for low income families and families of color, even 20% is not a trivial number, and the decision of what school you go to could be much higher stakes, especially if there are other environmental, neighborhood, poverty related factors that are putting negative pressure on your child's learning. And so making sure you're maximizing that, that 20%, and I, more or less agree with that characterization, I would just emphasize 20% is not trivial. And one thing I really did not want people to take away from this story was the idea that well schools don't matter for student learning.  And we have a great deal of evidence that they do matter. And that schools vary in how much they help students learn year over year. And the growth data reflects that. And they vary, and not just in test scores, but how much they help students show up to school, how often students are suspended, how likely they are to enroll in college, et cetera.

And I know that wasn't the premise of your question, questioning that. But I, perhaps your point is that for some families, the decision of where to enroll your child is higher stakes, when you have all these other factors that are pushing against you, and other families, it's lower stakes when you have all these other factors that are pushing in favor of you.

And I think that is an interesting question. Now, whether that message is going to be effective for any parent, to say “well, it's not high stakes where you enroll your kid.” I don't know. I think that, I think that is a, that is a difficult proposition and I think it's doubtful that from a policy perspective or even like a GreatSchools quasi policy perspective, that that argument is, is actually going to hold water and effect, family's choices.

Andrew:  Yeah, I guess I, I've, I maybe disagree with that a little bit, because I think there are definitely some people for whom that's not gonna mean anything. They're like, that's great and all, but I'm still getting the “best for my kid”. But I think, you know, Integrated Schools as an organization, we see a lot of parents who are rethinking what does a good school actually mean? And what is your obligation as a member of society to the greater good? And, and I think you're right, the stakes are different for different, for different families. And so that, I think there are… at least we look at it, the responsibility on White and privileged parents who, you know, the system is set up to continuously push our kids forward, continuously lift them to the next rung of the ladder, that there's a, a different responsibility in terms of how to use these data to make choices about schools. 

And if you have this conversation with somebody who is not thinking about schools, is just, you know, maybe thinking about having a family, looking to buy a house, I think there's a lot of room for nudging to actually be really effective there. When we say, what you should do is find the best house that gets you into the best school possible. And the way we define best school is with GreatSchools ratings, I think it has a real big impact on a lot of the ways that people think about their responsibilities as parents and their role in providing for their kids in a way that, that subtle nudges may actually have a big impact on, to let people start thinking about their other values.

Matt Barnum:  Yeah. I think that's, that's a really interesting, compelling point. I think the fact that Integrated Schools as an organization has resonated with so many parents does counter my point that well, maybe not - the argument doesn't have that much resonance. And so I take your point. I think that the other thing I would say is that our data analysis shows, contrary to popular belief, that there are a lot of schools in high poverty neighborhoods that are helping students learn a great deal and are probably, and this was just based on the test score growth dimension, but are probably excellent schools across many other dimensions.

So it's not even clear how much you're giving up, how much of that 20% you're, you're giving up, if you're a White, affluent family who's enrolling in a high poverty school.

Andrew: Yeah. I mean, I mean personally, I feel like my kids are getting a great education at their 2 school.

Ali McKay:  Yeah. I feel the same. 

So Matt, this was obviously a lot of work that you did, when you were doing this, what did you hope people would do with the article and with the work? Did you have a vision of how it would be used?

Matt Barnum: Yeah. It's always hard. It's actually always hard to answer that question, even though it feels like a straightforward one. It is a straightforward one. I hoped that it would spark conversations and discussions about what is a “great school”, and how different measurements and conceptions about what a “great” and “good school” is, can shape school sortings, stratification, and segregation, and discuss alternatives to predominant conceptions and rating systems. And I was really pleased with how the article was received in that respect. 

Andrew: Awesome. Well, thank you for writing it. Thank you for all the work you do at Chalkbeat and thank you for coming on the show.

Matt Barnum:  Thanks for having me. This has been a really interesting discussion. I really appreciate it guys.

Ali McKay: Thank you.

-------------------------------------------------------------

Andrew: So, big thanks to Matt for coming on the show, for sharing his work with us. Ali, what did you think? What stuck out to you from that conversation?

Ali McKay: There's, there's so much really. I think that, when we talked about that 20%, that schools can control. And in my mind, coming from sort of my place of Whiteness and privilege. You know I thought, well, 20%, that's nothing like that doesn't really matter that much. And then I think Matt's point that, if you're coming from a different place say, that makes a big difference. Like that, that's a lot.

Andrew:  It's not insignificant.

Ali McKay:  No, it's not insignificant. And then it ties in well to, you know, when we later discussed, well, maybe these numbers, these ratings should be seen and used differently by different folks. Right?

Andrew: Yeah, that what the ratings and the information call you to do is different based on your background and is different based on the sort of cushion of privilege that you have. Whether or not  the system is going to keep pushing you along or the system is going to try to hold you back really changes how relevant that 20% is.

And so what does that mean our obligations are when it comes to looking at these ratings? Because as problematic as some of the ratings are, we can't have no accountability for schools. Right?

Ali McKay:  Right. I think it, it highlights for me this need to keep zooming in and zooming out. You know, you might zoom and you might look at the test scores, you might look at a school rating and ask a lot of questions, like where do they come from? How are they used?

And then zoom out to the inequity forest, right? There's the trees of testing. There's the trees of ratings, and then we have to keep zooming out too, especially I think when we're White and/or privileged.

Andrew: Yeah. What is our obligation to directing our actions towards not just the trees, but the forest?

Ali McKay: Yeah. I don't know, Andrew, if you get this question when you talk about this topic, but I do. It's often sort of like, “Well, what should we do instead? Like, if I can't use a GreatSchools rating or test scores, like what am I supposed to do?”

Andrew: Right. You start with, here are the problems with the ratings, but it's like, it's like ripping the rug out from under somebody. They're like, “All I'm trying to do is get my kid into a good school, because that's what it means to be a good parent, is to give my kids a good chance at a good future, which I do by giving them a good school.”

And I think, both last episode with, with Jack Schneider and this episode sort of tells us, maybe those measurements are not telling you what you think they are, that the average test score of kids at a school isn't actually telling you that much about how your kid is going to do in that school and what things your kid is going to get from that. And it's kinda terrifying as a parent right?

Ali McKay:  Yeah.

Andrew: It's like, then well, so then what? Now what do I have to do? Because I think part of it is the weighing of all the various factors that go into a school decision, you know, some of it is convenience. Some of it is, I can't actually get my two kids to two different schools on opposite sides of the city at the same time.

So how much do I weigh that? There are all these various competing things and figuring out how to weigh them is really both challenging and sort of terrifying because it feels like the stakes are really high and it, and that I think pushes this drive for something like GreatSchools. And like Matt says, if you get rid of GreatSchools tomorrow, something else is going to take its place.  There is a desire for this reassurance that you are doing right by your kid and that means you're sending them to a “good school”.

Ali McKay: Yeah, and GreatSchools didn't create these systemic issues themselves. Like, the bias existed, the racism existed, the systemic problems have existed for a long time. So in a way, now that we have this data, we have something to start talking about. We can sort of shine a light on the data like Matt has done and say, why is it like this? You know, and I think then I myself find, I, you know, I can kind of get into the weeds on this stuff and I can geek out and talk about like, well, I guess we could use, not just proficiency, but growth. Right? Like that, that would be good. Maybe we should do that. And I started to get into a lot of these problem solving and solutions. And I think it's often a White way of thinking about problems and solving them. And I myself will start to hide in that kind of discussion.

And, and I've found that then I really have to bring myself back to sort of more that, that inequity forest, right? And also like just the personal stories I find when I'm telling people about our choice and the move that we made. And then now, like you and I, we both have kids at these 2 rated schools, by all accounts they should be terrible.  But I have a personal story to tell that is meaningful to me and I hope is meaningful to other people. And I can't solve all these problems myself. GreatSchools isn't gonna solve all of this itself but what, what am I going to do that's going to push back a little bit? Just change the narratives? Help lessen that fear when you see that number two? Like just, yeah, there's, there's reasons that it's a number two, but it's not the terrible place that you might think that it is. And, you know, it's, it's like the silver lining to a dark cloud is that we get to have these personal conversations. And I think conversations that Courtney was particularly good at having, right. And, and that we're all sort of trying to do now, and do better.

Andrew:  Right? Yeah. Yeah. I mean, the data is actually highlighting things that are an issue, right? Like the fact that there is some number of kids who can't read in third grade is a problem. 

So we need data, we need to hold schools accountable. And this is, you know, the, the sort of good intent behind things like No Child Left Behind, which was like: let's highlight these; let's have, have real insight into what's happening and it just sort of comes back to me that it's like, what is our role as White and privileged people in interacting with this data? If we can't use GreatSchools data to choose a school what should we do? How do we choose a school?

Ali McKay: Yeah. You see conversations, like, well, you should look at the growth and you should ask the principal how many teachers of color they have and it's not that those things aren't, aren't important. But aren’t they yet another way that those of us with power and privilege are going to hoard a resource? It, maybe it's not the same resource, but, I'm going to make a choice. Right? I have the power and privilege to make a choice.  What do I do with the information that I have in a way that considers the fact that there are many other kids besides mine whose parents care deeply about them, who deserve the best in life?  

I think though it is helpful to start to go to go into a building, that's what, that's how I got started, is we just went into the building and I think seeing the physical space, you know, somebody as you'd say, somebody showed up and opened the building that morning, and there were kids there and teachers there and they were really cute.

It was the place for me to start. It wasn't the place for me to end, but it was the place for me to start.

Andrew:  One of those things that I struggle with a bit is we have this history of testing that came out of eugenics that continues to show White kids being smarter, even as the tests have evolved and have changed and have been designed not explicitly to do that in the same ways.

And, and I think there are some issues with how tests function. But tests are also reflecting problems in the system, right? So if the tests are showing us that Black and Brown kids are not reading by third grade at the same level as White kids, some of that is telling us that maybe how we're testing is problematic.

But some of that is also telling that as a society, we are not serving Black and Brown kids as well. Our education system and our greater society as a whole, right. Like our, our early childhood education programs, our childcare programs our...

Ali McKay: Taxation systems. I mean we can keep going all the way up!

Andrew: Right , right? Yeah. Where, where grocery stores are located are, our history of redlining, poverty, criminal justice, all of these things that contribute to what is actually a fact that fewer Black and Brown kids are reading at third grade than White kids.

And without the data, we can't look at that head on and try to do something about it. But we also have to recognize that there are problems with the way the data is being collected and then used.

I think that there is real value in sharing data about what's going on in schools, particularly for communities that have not been served well by our school system. You know, what that data is and how it's shared, I think is relevant. And I think the, the real issue that I take with GreatSchools in the way that it currently functions is that it is part of the smog, that it is part of contributing to segregation, contributing to gentrification, contributing to changes in real estate values, that it doesn't, like NextDoor, take some steps to do something about that and, that feels problematic to me.

Ali McKay:  Yeah, we all have a duty to, to not just float along, I think with the, the river of racism, right? Like sometimes you'll be paddling across the current, sometimes you'll be paddling up current. But the river’s there. You didn't make the river. But especially if you say, “I want to serve the underprivileged” or “I want to serve Black and Brown folks, or marginalized or oppressed communities”, then, I think you need to be asking those tough questions, understanding some of that tough history too, about test scores. You know, and you've discussed with Jack Schneider, who's a history professor, right? And I think one of the reasons that's so valuable is because these things didn't just arise out of like whole cloth like suddenly we're here. We have hundreds of years of history.

And so we might not, we might not have the same motives that were had back when tests were created, but it doesn't mean that the tests don't still have some really problematic outcomes and then aren't used in problematic ways by, by White and privileged folks too.

Andrew: Coming back to, you know, what, what do we do with this information? GreatSchools sort of purports to give us an easy out to something that's just not, and shouldn't be, an easy decision.

Ali McKay: Exactly

Andrew: And I, I would love to see GreatSchools do something like NextDoor is doing to just try to take one step to maybe interrupt that bias process along the way.

Ali McKay: Yeah, I would too. Asking ourselves and others, including GreatSchools and Zillow and Redfin, like asking them at least, what do you know about these systems? How are you placed in these systems and how are you contributing or are pushing back? Like, I can't answer that for every single person. I can't answer it for GreatSchools or any other site, but you know, I'm asking myself those questions and I would hope that they would want to ask themselves as well, to try to make things better.

Andrew: Try to make things better. That is the goal. Lots more to talk about, to think about, to discuss. If you’d like to join us in that conversation, come on over to our Patreon page. Patreon.com/IntegratedSchools, there’s a link in the show notes. You can discuss on the forums, you can sign up for our podcast happy hours, a monthly Zoom call, where you get a chance to talk about the podcast or anything else on your mind. And you can help support this all volunteer effort. You may notice that we don’t try to sell you things and we’d really like to keep it that way but we can’t do that without your support. So, anything you can give we would be incredibly grateful.

Well, Ali, thank you so much for coming on, for joining me for this episode, for the conversation with Matt and for all you do for Integrated Schools.

Ali McKay: Thank you, Andrew. It was a pleasure to speak with you and Matt and be able to share my experience with you guys.

Andrew: If you enjoyed the episode, leave us a rating or review, share it with your friends. If you hated the episode, that's okay too, send us an email, let us know about it. Hello@integrated schools.org and be sure to check us out on social media. We're on Facebook, we're on Twitter. We've got a great Instagram page @IntegratedSchools, and as always, grateful to be in this with you as we try to know better and do better.

See you next time.