Degree Choices Hopes to Reduce US News’ Rankings Dominance

Higher Voltage is brought to you by eCity Interactive. For over 20 years, eCity Interactive has created websites and digital marketing strategies and solutions for colleges and universities, that deliver results and exceed expectations. Their latest offerings to higher ed clients include enrollment funnel diagnostics, and enrollment support services that efficiently attract and engage potential applicants with results you have to see for yourself. To learn more, visit ecityinteractive.com.

Hello and welcome to Higher Voltage, a podcast about higher education that explores what’s working, what’s not, and what needs to change in higher ed marketing and administration. I’m your host, Kevin Tyler.

Welcome to Higher Voltage. Today’s guest, David Levy of Degree Choices. David manages the content, product and data strategy for Degree Choices. He has recently been spending a lot of time developing and honing the economic valuation methodology used for college ranking and graduated from Kenyon College in Gambia, Ohio, with a degree in political science. David, I’m so happy to have you on higher voltage.

Thank you so much. I’m really happy to be here, Kevin.

So before we get into the work of Degree Choices, I want to talk to you about rankings and how it’s broken. There’s so much discussion right now about how the college ranking system is broken. But I’m curious if, in your perspective, if you think it was ever good, and if so, when and how did it go wrong?

I don’t think that the college ranking ecosystem historically… And I’m thinking specifically about US News here, has done really anything positive for higher education, just the opposite. I think that any ranking system is a subjective exercise by virtue of what you decide to include in your methodology and what you don’t. And I think their methodology has provided schools a very negative incentive structure. Basically in which providing elite education to the already elite gets you high marks. And it’s not like I’m saying anything groundbreaking here, I’m borrowing from other people’s arguments like Colin, whose podcast here I really enjoyed, and Malcolm Gladwell’s who had an excellent, broad side against the ranking industry. And there’s tons of literature on the subject. So no, I don’t think the college ranking system has done much good historically.

Now I think that’s starting to change for a number of reasons. Politically, I think that an increasing taxpayer funded higher ed granted loan balance, a lot of which is going to be written off, means that there’s more focus on which schools are providing value to students and taxpayers, and which schools aren’t. And I think also, not just from the political realm, there are a number of polls that are showing that students and Americans more widely have lost faith to some extent in the value of higher education.

So I think the ranking landscape is beginning to change to account for these facts. And I think it’s made possible largely by access to better cost and earnings data, thanks to College Scorecard. And much of leadership in this is being done by NGOs in the space. We’ve been very influenced specifically by Postsecondary value, which is a Bill and Melinda Gates product, and Third Way and their ex-education director Michael Itzkowitz, who actually consulted with us on the first iteration of the rankings. So I don’t think the stuff is breaking through entirely like the new methodologies yet. But I think the criticism of US News in the start of colleges refusing to participate provides an opportunity for new voices and different methods of rankings.

I really appreciate your statement around the value of college, which has been called into question millions of times recently around the value of college and what the return on investment is. How do you start to assess value in the absence of something like rankings? What kinds of things would you look at?

I think that from our perspective specifically, we are looking at one very specific part of a school’s value. And that is strictly and specifically the economic value. There are a number of things that go into providing a valuable experience for a student and different things will be more or less important for a student based on what they’re looking for. I don’t think any ranking system will be able to synthesize all those different attributes into one system and assign a number. I actually thought that Colin was exactly right in the podcast where he said there need to be specialized ranking systems that each account for different angles and approaching the schools, and I think students should be looking at multiple different ranking systems to determine what’s the right school for them. So value is in the eye of the beholder. For us, we fulfill one very specific value proposition, and that is basically weighing what a reasonable expected return on your educational investment can be.

Right. Okay, so if you could develop the most accurate and penetrating ranking system possible, with no barriers to the kinds of information that you could collect, what data points would you use for that? Or some of the data points?

From our perspective, and this is something that can be done, is that the data provided by the Department of Education should be disaggregated to the student level. And by the way, this is something that was requested in an open letter from the Bill and Melinda Gates Foundation with a bunch of different people signing onto it. So what does disaggregated to the student level means? It means that earnings and cost should be shown at the student level along with things like ethnicity, race, gender, income level, SAT scores, graduation status, time it took, what program, did you go on to get a master’s, et cetera, et cetera. And so why is this important?

I think a great example of why it’s important is the difficulty we have in ranking HBCUs, historically Black colleges and universities. We do a really horrible job really in assigning value to HBCUs. And the reason is, all we can compare between different schools is the median cost and earnings of the graduates at a program and a school level. HBCUs are 80% Black and they have a much higher rate of low income students. And it is well documented that Black students as a whole earn substantially less than white graduates.

So how can we adequately compare earnings between two different schools, one in which 80% of the student population is Black like in an HBCU’s, your average HBCU, and the other in which 80% of the student population is white. To really understand that the value that a school is providing its Black students, or really any student demographic, you need to be able to measure those specific students in different schools and make those direct comparisons. So to bring it back to what data and why and how I would change our rankings, and I have to explain now what we’re doing a little bit. So bear with me for another… Sorry if this answer gets long-winded, but I got another maybe three minutes on this one.

So let me explain what we’re doing now and I can explain what I would change. So we rank colleges according to economic value based on two different metrics. The first is the relationship between the cost of an education and the reasonably expected earnings after you graduate, after you leave school. And we get this by basically calculating the total cost, which we look at the annual cost to title four recipients, and then we look at an average completion time. And then we divide that total amount by the marginal earnings, which we can attribute to the school itself. And we get that by looking at median school earnings and subtracting what the average high schooler could expect to earn.

And that margin is what we divide then total cost by. And the output is basically how long it would take you with the earnings attributable to your college education to pay back the total cost. Now, this by the way, is heavily influenced by Third Way and Mike Itzkowitz. They call it the price to earnings premium. We make minor adjustments, but that’s the first part of it. Now, cost to earnings though isn’t really enough. And this is where it comes back to the disaggregation, isn’t really enough to sign economic value of a school or program because it doesn’t compare the relative earnings. So for instance, there can be one student who earns 30 grand and at a cost of 10 grand, we would all rather be in the shoes of a student who earns 45 grand and pays 15 grand because he is earning more. But according to cost of earnings, the relationship is exactly the same. It’s three to one.

So to compare the relative earnings of a thousand different schools, we create earnings benchmarks. At the simplest level, the benchmark basically means looking at the average earnings of all graduates in one cohort, one year, and taking that weighted average and then comparing it to the median earnings of a particular school. And the difference there between that weighted average benchmark in the school is how much more or less than average students at that school are earning. Now, this doesn’t account for very important differences in the school, and this is where I’m getting back to the HBCU example. Right now we’re looking at program ecology when we calculate this.

So imagine for instance, a school is 80% engineering majors. An engineering major will earn much more on average than the average student when they graduate. Okay? Now imagine if you were to compare that school to another school that has a myriad of different majors and students majoring in all different subjects. That second school could have engineering majors that actually perform better than the students at the engineering school. But because we’re aggregating everything together into one earning sample, you would think that the first school is actually outperforming that second school. So what we do is we basically calculate the relative earnings at each school of every major, and then we aggregate that up to the school level based on the proportion of that major of the total school. So it basically adjusts earnings based on program ecology.

Okay.

Okay. So back to my point, if we had disaggregated student info, then I could account for… I don’t know exactly what, but I could account for for instance, race and income at the program level and then adjust the school’s economic performance accordingly. So if a school has 80% rich engineering students, I would be looking at that compared to other rich engineering students or low income engineering students and making the adjustments that way. And I think that has a number of ramifications on how it would impact the ranking sort of ecology. I know that is a lot, but-

Listen-

Good question.

I’ll be first to confess, I was not a math major and that was a word problem. But what you are saying, I think I understand, and I think I’m curious though if we got to that type of comparison, how students and their families would navigate a space like that. It feels like a lot of more work. I mean, one of the things that US News purports to do is make it a one-stop shop for college assessments, which we all know is a broken way of assessing institutions, but that notwithstanding, it is a place, a single place that people can go now to learn something about higher ed and what colleges they want to attend. Because of how mystifying we have made the college search process for students and their families. How do we get people to do the extra work to compare apples to apples as opposed to apples to broccoli?

Right. I think one of the challenges and where we sit is between those NGOs doing this sort of work in the mass market and we try to make this information digestible to the mass market. All this data, we pride ourself on transparency. We’re only not transparent to the extent that I just can’t explain us ourselves well on the website. But your average Excel stud could probably do this stuff with available data. What we do is we show the end of our work and we show basically how schools perform economically. If students want, they can go in and really start digging in. But what we want to do is be able to provide that assessment so that students can easily do that comparison. So what does it mean exactly? So right now if I’m looking at two schools, we’re taking the overall earnings and we’re saying, look, this school performs 10% better than that school.

What it would end up being is we would look at different categories of students in the school and then adjust that percentage accordingly. And then the students would at the end see, okay, this school is outperforming that school. Whether or not the earnings of lower income students are 5% higher than average or higher income students are 5% higher than average, using this sort of system, it will still be 5% higher than average. Whereas before, it’s sort of emphasizing that top end of the student body, if that makes sense.

It does make sense. I appreciate some of the topics that Degree Choices covers on its blog. I was reading a couple of those pieces preparing for this conversation, one of them being titled The Challenge of Ranking HBCUs, which you’ve touched on in your previous points. These Are The Worst 42 Colleges in America. Another piece that is another ranking, but it is done under the following. “We ranked the lowest schools in America using US governmental data. These colleges have outrageously low earnings after graduation alumni who are in debt for most of their lives and disturbingly low graduation rates.”

It feels like those inputs for a list like that, which is, this is obviously an unfortunate list to be named to, but those three inputs feel like some of the most important conversation points or data points that families and students are looking for, earnings, debt, and graduation rates. And so is there an inverse of that, that might make you feel better about the ranking structure as it is? Can we just have here these are the 42 best colleges in the country based on outrageously high earnings or alumni who are not in debt and disturbingly high graduation rates? Does that work that way or is there something else that needs to take place?

So that’s the bread and butter of the site. This worst college was just one blog piece, which was taking it from the opposite perspective. But that’s what we do. That’s how our rankings work. We’re looking based on any different type of segment, any school type, any major you can think of, different degree levels. And we’re comparing the schools based on that economic return. So for us, that’s what we do. The inverse is the economic value provided to students who are looking for particular types of schools. But again, this is just one way to skin the cat.

One of the schools on that list is Berkeley School of Music. Now, couldn’t really take it off because where do you stop? Right? But someone who’s going to Berkeley School of Music wants to be a concert pianist or whatever it is, and they’re considering a whole lot of different considerations than just the economic return on their initial investment. So again, to have that single perspective view of rankings is necessarily problematic. Because even our rankings, which are transparent and single issue and we use public data, they still by virtue of the fact that we’re only looking at economic return, means that they are subjective. Because they’re not looking at other things that people want to be looking at. So for us, the inverse is how well is the economic return based on cost to earnings and based on relative earnings.

Gotcha. And the other part about rankings that I am always so challenged by, you touch on this in the piece around the challenges of ranking HBCUs, is the parts of the institution that many people don’t see and that is that HBCUs are historically underfunded. And so what you are doing with the money that you have to influence a person’s career trajectory when you have less to begin with, the gains that that person can make are generally going to be less. However, HBCUs have a long history of providing the kinds of support and programs that African American, Black students need in order to thrive in those communities, in HBCU communities.

And so the parts of a ranking structure like the one we have now in US News and World Report and some of the others, that praise or reward already wealthy institutions who are just obviously educating really smart and likely other wealthy students, that whole story is not being told. And so it then becomes a disservice, as you mentioned in your email, a disservice to African American and Black students about what an institution can do for them in their lives because they’re not being ranked in the way that we typically rank institutions. How do we start to tell the more full or complete story around the higher ed landscape, in a way that informs the general public about the inputs that affect the way that the rankings shake loose, if that makes sense?

Well, first what you said about wealthy versus non wealthy schools, I think it’s very important also for the way that we’re doing the rankings. Because private PWIs, predominantly white institutions have much higher endowments, and that means that they’re able to subsidize to a much greater level the cost to students. Now, the best or the highest earning HBCUs have compared to the average school, very good earnings. However their endowments, I think Howard has the highest endowment, it’s like 500 million, that’s like one… What is it, hundredth of Harvard’s? So they just simply cannot compete on making that cost side of the equation lower, and we’re not able to measure that. In terms of what we can do to paint a better picture, I am not qualified to get into those other things because all I can do is look at the data that’s available to me and then make those models for it.

Sure.

I think that there is a risk in trying to do that. I think that’s something that US News probably started out trying to do. They do all of these surveys and they get reputational score stuff and they try and figure out a way to quantify stuff that isn’t really quantifiable and it can be dangerous. It’s something that needs to happen. I mean, like Colin was saying, we should be looking at the quality of teacher quality and the ability to prepare students to get master’s degrees. These sorts of things I think are super important. That’s not the angle we’re taking, because we’re not really set up to do it at the moment, we’re just a startup. I think to a certain extent, the stuff that we do, can also serve as a proxy for doing some of those other things really well. Not always, but I think if you can see that there is a value in what students are getting out of their education and they’re graduating at a high level, then I think that shows that there’s a lot of other stuff that’s going right.

Right on. Okay, so I’d like to learn more about Degree Choices and how it advocates for equity in higher ed. Can you talk a little bit about that?

Well, we’re not probably advocating as much as I would like. I think that anything that chips away at US News’ dominance in the market is a good thing. I think our rankings are actually very much appreciated by institutions that are generally underappreciated by other more serious ranking systems. And these schools are usually more representative by the way of the country as a whole. So CUNY schools, I think City Colleges, is our top-rated school. As a matter of fact, Cal State, some U Texas campuses, public universities are very highly represented on our ranking list, and that is because they have exceptional cost-to-earnings ratios. So they are providing students great value, and that value is undercounted in other ranking systems.

Now, if we were to utilize that disaggregated student info, then I think theoretically we could be doing a lot for equity in higher ed. Because I think, and this is what I was trying to talk about a little bit earlier, I think comparing the KPIs of specific student demographics reshuffles the deck, and it sort of minimizes that negative incentive structure catering to the elite. Because again, what’s important is the degree to which you’re able to provide better value absolutely to different subsections of students and compare them against their relevant subsection.

Lower income students. If a lower income student is earning 15% more than his low income demographic on average, I think that is just as good as if a higher income student is earning 15% more than his demographic, considering all of the built-in advantages that that higher income student has. So I don’t know that it’s any more laudable to take elite rich students and make them a bit richer. I don’t think that’s any more difficult or more laudable than doing that with more at risks or lower income or populations.

I hear you on that. So we talked a little bit about how Degree Choices looks at ranking institutions and a little bit about how it differentiates from more mainstream ones. There’s a piece on your blog about how you’re updating your rankings system and methodologies, and I’m wondering if you could talk a little bit more about that and if there are other moves that could make your rankings a bit more equitable and representative of the institutions that are ranked?

So the first change we made was that we take into account the program ecology of the schools, which we discussed earlier. Another thing that we did was we started comparing the in-state out-of-state student percentages as well. Because what we were finding was that sort of local economic, or labor dynamics can have a large impact on earnings. And so for instance, it can have two effects. One, if there is a national school that is renowned, that is a target university that is located in a less affluent state. On average, the students from that school are competing in the same labor markets as schools that are located in higher earning states. And what we were originally doing was we were comparing their earnings to the state earnings only.

Now basically we take the out-of-state student percentage and we compare their earnings to the national earnings benchmarks. And we do the same for in-state students. So if a public university services primarily in-state students, then their earning benchmark will be primarily determined by the in-state earnings averages. Whereas schools that are pulling from across the country will be against the national. So that’s the other change that we made. I think it probably bolsters a bit some of the in-state schools, some of the national publics that are drawing more local student populations.

Awesome. This is my last question. I like to ask this of every guest. In your opinion, what do you think higher ed will look like in five 10 or even 20 years from now? Or what would you like it to look like?

Yeah, I was thinking about this after, I figured you would ask me this after the Colin. Let me actually go off what he said, which I thought was very interesting, and he has much more insight than I do. He presented a really pessimistic answer to the question in which he saw mass consolidation, closure of institutions of higher educations due to students seeking out vocational or skill-based training. So I think two things on that. I’m a bit more optimistic. I don’t know if it’s well placed or not. I think first of all, this trend is somewhat of a reversion to the mean because the percentage of jobs now that require four year degrees has risen to a very high percentage. So even compared to the relatively recent past, the need to get a four year degree, despite governors saying we’re going to earmark certain jobs as non-college educated requirements, I still think it’s above where we were before, that’s the first thing.

The second thing is I think a lot of the refocus is just based on student shell shock when they see that sort of sticker price, which is so astronomical. And I think that there’s a disbelief that they’ll be able to get value out of that. I mean, there was a recent poll that I think for the first time, a majority of Americans see college is not worth it. So I think that’s a big reason for it. So we’re sort of approaching a reckoning. More emphasis is going to be placed on value. I think it’s something that’s from policymakers, from the top and the bottom, there’s pressure. And it’s going to lead to more accountability, and hopefully, this process will lead to a restoration of faith in the system. Additionally, by the way, and for my very specific, if US News gets knocked off its perch, I think slowly we’ll see greater income and ethnic diversity across a larger number of schools. And I think this could lead to a better allocation of endowment funds, which will really help I think the ecology of the higher education landscape.

Yeah, I agree with you, especially on that last point. I think as a person who works in marketing and branding in higher ed, when the chief rankings infrastructure has a rubric that forces everyone to look the same, it makes it very, very difficult to, A, differentiate yourself from each other. But B, makes it very difficult for students and their families to pick the right place for them based on what they need and not what everyone else expects them to want. And I think that once we can dissolve what we have now as a way to assess quality, because these rankings are a proxy for quality, then we can start to see institutions really be who they are supposed to be. And if you’re a place that is for people who learn differently, you can say that, shout that from the mountaintops without being dinged by this national periodical that assesses on much different factors.

So, I’m hopeful that that is part of the future of higher education. I’m also hopeful that institutions are able to adjust how they deliver education and to whom. As the population shifts, higher ed will have to shift as well. And so we hear a lot about this enrollment cliff or demographic cliff. I mean yes, we know that the numbers of human bodies are fewer of the traditionally aged, and I use that in quotes. But if you prepare an institution to educate more kinds of people who are of different ages, who have different needs, who need more flexibility, that is one of the ways to maintain your status as an institution as opposed to you saying there’s not enough freshmen, we have to close.

So this adult learner population, that some-college no-degree population are going to be really important for a lot of the institutions who are figuring out how to make ends meet. And so broadening who you’re educating is going to need to be part of the plan, there as far as my perspective is. So I hope that we can get to a place where higher ed can be more flexible and meet the needs of a changing demographic, but it will take a lot of work, of course.

Preach on, absolutely. No, I think you nailed it. I mean listen, it’s going to need to evolve to a changing tapestry in America, right? No longer is everyone a full-time, first-time student. No longer is everybody full-time. There’s students that need to work and there are students that need to go part time, and there are students that are going back to school, and schools can also start specializing in that sort of thing as well. So I think, yeah, I think you’re absolutely right. I think the changes that are being made that College Scorecard started, I think will only do good in that respect.

David Levy, thank you for joining us today on Higher Voltage. This was a great conversation, I appreciate you taking the time and look forward to seeing how Degree Choices continues this conversation around college assessment and rankings. And if you are not familiar with Degree Choices, please check out the link on the episode page. We will have a couple of pieces linked to their blogs. And thank you so much for your time today, David.

I really enjoyed it, Kevin. Thanks a lot for having me.

That’s it for this week’s episode of Higher Voltage. We’ll be back soon with a new episode, and until then, you can find us on Twitter @volthighered. And you can find me, Kevin Tyler on Twitter @KevinCTyler2.