Michael Livermore 0:11
Welcome to the free range podcast. I’m your host, Mike Livermore. This episode is sponsored by the program on law communities and the environment at the University of Virginia School of Law. With me today is Lisa Robinson, a Senior Research Scientist and the deputy director at the Center for Health decision science at the Harvard School of Public Health. Lisa is a leading expert in the use of cost benefit analysis to evaluate public policy, which is what we’ll be talking about today. Hi, Lisa, thanks for joining me today. Hey, Mike. So So you’ve spent a good portion of your career working on and thinking about formal cost benefit analysis using formal cost benefit analysis to evaluate lots of different kinds of public policies? Just to give a sense, what do we mean by public policy here like things like whether we should require a new safety feature on automobiles, or whether we should cut certain kinds of air pollution or whether we want to ban or control a food additive that has some negative health risk? Right, that’s the kind of stuff we might do a cost benefit analysis on. And as you well know, as an expert in this area, not everyone likes this kind of cost benefit analysis, there’s, you know, it’s very well entrenched in our public policy system, but there have been critics for as long as it’s been around. So I maybe I thought we would just start with your kind of 92nd elevator pitch for why we should be doing this say you, you know, had a senator in the Senate office building in the elevator, you had 90 seconds since she asked you, you know, knowing who you are, you know, why are we doing this cost benefit stuff? Why is this valuable?
Lisa Robinson 1:41
I have to tell you, I don’t need to wait for some sunder in the elevator, I get these sorts of questions all the time. It’s definitely so um, the, in the scholarly literature, a lot of times you see, almost all the time you see people describe benefit, cost away analysis as a way to see what policy is most economically efficient in the sense that the benefits exceed the cost by a significant amount, or I don’t think that’s why we do benefit cost. However, usually, there is enough that we have difficulty quantifying and there’s enough uncertainty in things that we can’t quantify that it doesn’t give us a definitive answer. What it does do is give us a well established systematic framework for investigating policy impacts. I think this is just forcing us to poke at the problem. I’ve done a lot of analysis over the years that were actually never completed, because somewhere along the way, we discovered something that nobody knew that made the policy just not make sense, not feasible. For example, I worked on one regulation, we’re in the very early stage that would have released some materials that were potentially hazardous. And we found very early in the analysis that the cost of testing to see whether or not materials were hazardous were so high that it didn’t make sense to even get started on doing some sort of regulation or policy. I found that over and over again, for people who are really deeply into these frameworks, there’s all kinds of controversies that I’d be happy to talk about. But I think for the general public, there’s really just two things. And one is that we made the unfortunate mistake many, many decades ago, I think of labeling the way that we value mortality risk reductions as the value per statistical life. If I could get a My Little time machine go back to whenever it was that somebody first coined that term, I would tell them to please please, please come up with something else. People think that we are placing a value on their life and they find that very offensive. But that’s not really what we’re doing. What we’re doing is asking the question, how much would you be willing to pay for a very small change in your risk of dying this year, because most policies that we’re evaluating only change your risk of dying by a very small amount. But it’s very hard to get that across. The second is, I think, a little bit more subtle. And I think we as analysts need to do a better job of communicating, but benefit cost analysis is intentionally not paternalistic. What we’re trying to get at is the preferences of people for spending on different goods or services that will benefit them somehow. So we’re trying to compare their own evaluation of the cost to their own evaluation of the benefits that they accrue. There are many other forms of economic evaluation, including the type of work that’s done for health and medicine, whether they’re using qualities or dollies that is not that is more paternalistic, because there’s this idea that health is what economists often call American So you should get lots and lots of health improvements. Regardless of your preferences, it doesn’t matter if you’d rather spend your money on other things. And then if a cost analysis, at least is conventionally conventionally applied, isn’t paternalistic and attempts to reflect the preferences of the people who are who were affected?
Michael Livermore 5:22
Good, okay, great. So so the, just to re re capsulate that a little bit, so on the benefits, and so to speak of cost benefit analysis, or the value that that you think is most worth emphasizing is this kind of structured analysis, right? The idea here isn’t to come up with some magical number that’s going to tell decision maker whether a policy is a good idea or not, it’s but it does force analysts and ultimately, policymakers to, in a systematic fashion go through and try to understand the consequences of their of their actions. And so that’s kind of on the pro side. And then you mentioned a couple of controversies. So one is this notion of, quote, unquote, placing a value on life, and then questions around kind of paternalism versus non paternalism and, and how that might generate controversy. So I think we can we can return to all these things, but maybe just focusing on the mortality risk one, it’s been discussed a lot, it’s something that you’ve done a lot of work on the value of statistical life, continuing to use that phrase, it’s just so current, it’d be so hard to change the words there. But maybe we could just unpack a little bit, as you said, what that is often about is a as a small change in your in your mortality in a given year, this kind of the fundamental unit that we’re often dealing with. So maybe let’s just, again, just to, you know, unpack all of this, like, what is a context where, like a concrete context, either you’ve worked on or, you know, we can even talk hypothetically, where the value of statistical life or the value of mortality risk reduction is kind of an important part of a benefit cost analysis, like, when does this arise?
Lisa Robinson 7:07
You know, that’s a more complicated question than it might seem to be on the surface. But let me start with the simple answer, and then give you the more complicated one. Most policies have some impact on health, it’s been interesting to see, you know, turn like climate change, air pollution, transportation, food safety, Homeland Security, all of those policies have some effect on mortality risk. And in fact, interventions that we often think of as being entirely outside of the health sector, like education often have fairly significant effects on mortality. But I think that to some extent, that’s a little bit misleading, because if you think about it, health includes how long you survived, it also includes the likelihood that you’re going to get sick, or you’re going to get injured. And we do not have enough research on non fatal health effects injuries and illnesses, to have a very good idea of their value. So I think a lot in a lot of the work that we do mortality risk reductions dominate the benefits estimate, partially because we care a lot about living longer. The partially because we’re not very good at assessing the value of other effects, you might have a health effect that has a relatively small value compared to the value of saving a life, I shouldn’t use that language, should I even I get tripped up the value of reducing mortality risks. But there’s lots and lots and lots of cases. I mean, I needed to think about COVID. There. I’ve forgotten the numbers now. But there are so many people who’ve had non fatal cases, that even if the value is a lot lower than the value of avoiding the risk of dying, it could add up to something very large.
Michael Livermore 9:03
Yeah, that’s an interesting point where a lot of the rules that we’re talking about a lot of the policies have lots of different kinds of effects, right? It can have effects on health, just your quality of life, right? So here’s what we’re kind of talking about. And then there’s effects on your how long you’re gonna live. Basically, those are the two general things that we’re talking about. And then, you know, there could be other kinds of effects what your consumption is, or whatever else. Just to, you know, we’ve been kind of ragging on the value of statistical life and the notion of saving life, but I think it is worth maybe just offering, if not a defense, a little kind of explanation of how we get to that kind of language. We talked about saving a life a rule saving life, and certainly agencies talk that way, right? They don’t say, at least sometimes they’ll say, Yeah, we’re gonna reduce the mortality risk, but over a given population, we will be saving lives. That’s often the idea. So if you have a population of 200 million people, and you eliminate a one in 100 Word 1000 risk for everybody in that, when I say 200 million population, you can be expected to save 2000 lives. Right? That’s the idea. Right. And so that’s so it there’s a there’s a peculiar kind of future of policymaking in a large country like the US, which is that small things add up, right small reductions in, in risks of different sorts, we’re talking about life, you know, risk to life and limb, small reductions add up. And so, you know, the risk that any child individual child will die by drowning is quite low, but over every single year, I believe hundreds, if not a couple of 1000 You know, kids die in drowning accidents. And you know, that interested in the individual will slip in the tub, and, you know, and, and have a fatality is low, but it but it happens every year in the US. So, yeah, so there is this, there’s almost like a peculiarity is, I think, a moral peculiarity, something that we’re not used to, in our evolutionary history or given our cultural history, you know, we think in terms of, oh, you know, a small risk to me, right? Or that, that very likely will not get realized, but a small risk to many people over big population, essentially, it’s inevitable that it’s going to be realized,
Lisa Robinson 11:32
well, it is going to add up. But I think, you know, a lot of this is Sneem, to be very careful about our language. And I’ve, you know, I’ve played around with different ways of explaining this, in both what I’ve written and what I’m teaching, and teaching over the years. And it’s challenging, but I think the key and what you said is, we expect deaths to decrease by someone that we don’t know, who would die, if we hadn’t implemented the policy, we don’t know whose deaths are being averted. We also don’t, we don’t all we’re doing this Delaine those deaths, we’re all gonna die sometime, perhaps unfortunately,
Michael Livermore 12:12
better or worse, I don’t know if the alternative of living forever would be all that wonderful either. So something we have to accept, we’re all gonna die.
Lisa Robinson 12:20
Maybe leave that one off the living forever, one off the table issue this conversation, because, but let’s unpack the term for a second, because I think it relates exactly to what you were just saying, you know, it’s got two pieces. Well, three pieces, maybe value is the dollar value. Because we’re using, we’re using money as our medium of exchange, because we could use apples or oranges or pick your favorite thing. But money is the easiest thing to use, because we’re used to spending money on different things in the marketplace. The second is statistical. And that, to me is really important. Because what that’s talking about the probability of dying, it’s that this policy, for example, is decreasing each person in the populations likelihood of dying by one in 10,000 in the defined year. And once we sum that up across everybody in that population, we get to a number of expected deaths. So it’s a probability of dying, it’s not death with certainty. And then the final one final piece, of course, is just just life. And the idea there is the goes back to what I said earlier, the awkward delaying deaths, and sometimes we’re only delaying them a year, it might not be that long. We’re not, we’re not saving somebody’s life with certainty. It doesn’t. If we have a risk reducing policy that’s implemented this year, it doesn’t mean that you’re going to live to whatever average life expectancy is for somebody your age, it just means that you’re less likely to die in this particular year.
Michael Livermore 14:03
Right. Yeah. So so this is all true, right? So that’s, it’s, that’s what we’re talking about. The one is, is statistical, right? That’s an important characteristics of it, characteristic of it. And then the other is, you know, we’re only ever extending anybody’s life. That’s the only thing that we can do. And there’s interesting counting questions that come up, like, I think of let’s do a little philosophy, philosophical question. I’m curious what you what you think the answer to this is. So. So imagine, let’s take away the statistical thing for a second just so that we can make it a bit more clear. And let’s say I’m a crossing guard. And there’s, there’s someone, you know, just Person A, who is just terrible at crossing the street. They don’t look both ways before they cross the street. And so, you know, once a week, the crossing guard has to grab ahold of person A and block them from walking right in front of it. Bus, it’s just happens with, you know, it’s like clockwork. Now, what’s gone on every time that cross, you know, crosswalk guard has or the, you know, the crossing guard has saved this person’s life clearly, at least, let’s just say that every time what, but for the cross guard crossing guard, the person a would would have been flattened by this bus. Okay. And that, but the crossing guard has saved this person’s life, you know, let’s say 50 times a year, should that count as one life’s saved? Or 50 life saving events? And I think that’s, you know, that kind of is a it’s a peculiar question. But it’s something that actually kind of comes up in regulatory cost benefit analysis, because it might be that we have all these different interventions. But for any one of them, you know, the people might be dying. But it is the same population that we’re saving over and over and over again. So yeah, I’m just curious what you think, like, what is the right way to value something like that? Should we just have, you know, the value? Is this one person’s life? Or is it you know, not, you know, whatever, whatever value we place on that, or is it 50 lifesaving events?
Lisa Robinson 16:08
So I have to say, first of all, I’m not that fond of the analogy, but
Michael Livermore 16:12
just because of the statistics, you don’t like the statistical part of it?
Lisa Robinson 16:16
Well, no, because these these methods are not designed to deal with individuals. He’s not okay.
Michael Livermore 16:22
Okay, I get it. So you’re, you’re resisting the fact that I’m separating out the statistical from the, the repeat player kind of
Lisa Robinson 16:29
element? Well, no, it’s because you’re talking about one person.
Michael Livermore 16:32
That’s what I mean, that’s what I’m talking about.
Lisa Robinson 16:34
Yeah. I mean, I think the way that we talked about this in the literature identified versus
Michael Livermore 16:38
Okay, okay, fine. I’ll, you know, I’ll, I was trying to simplify it, but I will, I will deal with the population, if that makes you feel better. Yeah.
Lisa Robinson 16:47
So so we have a population of kids in the population of crossing guards from something like
Michael Livermore 16:51
that, but it’s but it’s, but it’s the same deal. Like you’re, you’ve got the same kids, you’ve stopped your, you know, there’s a group of them that will cross you know, you know, bad and then one of them would get hit every time and, you know, over a certain number of times, you know, the same kid is going to end up getting saved several times with the same group of kids is getting saved many times or whatever. Yeah, I
Lisa Robinson 17:12
think this is a so. So I think the question you’re asking is, so let’s say today, we implement this new crossing guard program that keeps those bad street crossers from getting hurt. And tomorrow, we implement the safe driving program, that also is going to keep kids who are crossing the crosswalk from getting hurt. So there’s some aggregated effect across those two policies.
Michael Livermore 17:40
Say that say there’s only 100 kids could possibly save, right? But you’re gonna save each of them 10 times?
Lisa Robinson 17:47
Well, no, you’re gonna decrease decrease the risk 10 times?
Michael Livermore 17:51
Well, there’s a crossing guard that’s going to actually reach out and stop a kid, like that kid will not have gotten hit. Right, just same way that like that, like, you know, with a particular matter reduction. I mean, I get the, the, and actually, I think I’m gonna resist the resistance to populate it to individuals at some point, some will, because if, like, ex ante, we cannot say and we might not even be able to say ex post that like air quality rule that reduced particulate matter exposure saved, anyone’s saved any individual’s life, like we can’t actually attribute at the level of an individual that, you know, something like this happen that anybody was even benefited, right. But if we think that reducing particulate matter pollution, well, if we think that particulate matter, pollution causes people to die, right, the exposure to particulate matter, pollution causes, you know, heart attacks. And we cut particulate matter exposure. And we could even say, let’s even imagine, which would be very nice for like, we have a nice clean experiment where we can say, Okay, in this population here, particulate matter exposure went down by X amount. And now instead of, you know, on average, what we’re seeing is 1000 heart attacks every year, which, of course, with some noise, and now we’re seeing 900 heart attacks every year in this population, or fatal heart attacks. So we can say there’s 100 people, and we can’t say what 100 People would have died. But we can say that there are people walking around, right there. And there really are there’s someone walking around, you know, in the in the counterfactual, where we were, we had the same level of pollution where we hadn’t cut the pollution and that counterfactual. There’s someone walking around. Sorry, in the real world, there’s someone walking around who in the counterfactual would not be walking around. I think we have to think that if we think that these policies are actually having effects in the world, we have to think that there’s a counterfactual in which someone who is now alive Is that?
Lisa Robinson 20:02
Well, or? Yeah, um, I think, um, you know, the problem was a lot of the stuff is the simplifying, it takes away some of the clarity about what the concepts are. Because first of all, I think like going back to the crossing guard for a moment, you’ve taken out the idea of risk. So let’s say we’ve the crossing guard, there’s a 90% chance each time the crossing guard. Yeah, that’s, or maybe there’s only a 20% saying, yeah, it is, right, there’s, you know, a 90%, or a 20% chance that that kid is actually going to do something dangerous. The, you know, the in the real world. These things happen in a very complicated way. The other thing is that the one key piece of the VSL definition that a lot of people miss, is that a similar defined time period. So you would fear crossing guard example, we would want to talk about the chance that that crossing guard or the maybe this policy, this crossing guard policy, reduces the risk of X number of deaths within a month within a year. And I think that those two things take care of some of your concern about about overlap or double counting.
Michael Livermore 21:20
Not even sure I’m concerned about it yet. Oh, I was curious. I’m not raising a concern. I was actually curious what I’m, I’m not sure whether that’s double counting. And I was curious whether you thought it was I think it’s an interesting question.
Lisa Robinson 21:32
I think I think it’s a question of being very been mean, this is I think one of the benefits of doing benefit cost analysis is it forces you to be clear about things and, you know, for your examples, both air pollution one and the crossing guard, one, we need to be, in order to do a good benefit cost analysis, we need to be very precise about what we think would have happened in the absence of the policy, the basic, we need to be very clear about what we think would happen. And I’m using the word what we think intentionally because all these are expectations are not things we know with certainty would happen with the policy. And we need to be very clear about the timeframe and the population over which we’re estimating those things. One issue that does come up, especially with large policies that affect lots of people. So this idea that, you know, maybe policy A is affecting these 100 people in the population and policy B is affecting a different 100 people in the population, assuming you have a large population is, is whether we need to revise the baseline. So you know, my risk of dying, I don’t want to think about the sense of getting older, but if my risk of dying, is, you know, two and 1000 This year, we can’t keep comparing each policy to a 200,000 baseline, we need to think about how that incremental effect of preceding policies have changed my risk, maybe my risk is now you know, it’s now one instead of two, because of everything that’s come before it. But I think that’s often is that not what you’re getting at? Or is that what you’re? Well,
Michael Livermore 23:16
it’s funny, because we both know a lot about cost benefit analysis. And I think a listener who is who’s unfamiliar with some of these ideas, would would be shocked that we would we would make the claim that these that these techniques have a way of clarifying the the questions that that policy analysts are trying to figure out, right, because it is I mean, it’s actually quite technical, it’s quite difficult requires very careful thought. And, and so, you know, maybe we could maybe, maybe we could kind of change gears a little bit. And just and and I would be curious, your your thoughts about the kind of this meta question about the degree to which cost benefit analysis can be quite difficult for a non expert to, to comprehend what’s going on? Because as soon as you know, you know, we started talking about what is really a fairly straightforward policy, this crossing guard policy, right? Where the mortality risks are kind of straightforward, like the causal pathways, at least are straightforward and, and so on, and actually quite quickly turns into a complex question involving, you know, what are the you know, what are the, the suite of policies that we’re talking about? What are the populations what is the timeframe? What are the base rate risks that we’re, you know, that we’re talking about? And, you know, that’s an those are all those are the right questions to ask. Absolutely. One has to ask those questions and any kind of cosmic even have a simple policy, even the almost like the simplest policy that you can think of still requires, you know, kind of a nuanced, really understanding of of these various features of the policy problem. And I think that can be part of what folks resist the leads folks to resist cost benefit analysis, because then they start to say, well, this is always looks like gobbledygook. This just looks like, you know, jargon, and I can’t understand what’s going on here. And it seems to, I think, what what some of the critics of cost benefit analysis will say is it actually obscures the stakes rather than clarifies them? So? So what’s your what’s, what’s your response to that kind of tick, because I actually think that’s fairly common is that people start to hear things like baseline and population risk, and etc, etc. And they, they just think, well, you’re over complexify, and something that we can we can think about in more straightforward ways?
Lisa Robinson 25:56
Well, I think so let me I feel like there’s several different issues there. And one goes back to what we were talking about at the very beginning, which is that my experience has been that if we don’t do some sort of analysis, people, people always have an opinion, or almost always have an opinion. But they don’t, it’s unexamined. I’ll never forget, I’m working in an environmental role in the US where somebody asked me, aren’t there benefits for women, and just environmental rule, this environment in the United States? Or I think it was actually more a statement that we’d left out the benefits to women. And, you know, and if I wasn’t, if we weren’t trying to do this benefit cost analysis, I don’t think that person would have never even asked the question, it would have just been implicit in their thinking. And after talking to this person, for a while, I realized that what they were thinking about was actually low and middle income countries, where people were women walked to get water. So having a cleaner water source nearby has labor savings for the women. But we weren’t talking about doing a policy in a low and middle income country. And the person I was talking to hadn’t really grappled with that in the US, we just got, most of us are fortunate enough to be able to just go and turn on our tap. The so I think it’s it’s, you know, it’s one of my colleagues, let’s talk about as an aid to thought. But I think it’s a way of just getting people to talk about what they think the policy is all about, and ask questions and try and reach a deeper understanding of it. terms of complexity, I think, I think it’s a real problem for us as a profession. What happens to me and I think happens to all of us is we get deeply into the details of our analysis, we write up the analysis, the way we think about it, and the way we did it, and nobody except us, understands it. And we, I think we need to get much better at communicating. And I think the other problem is the we ourselves need to be better trained. And some of that’s, there’s a lot of issues in terms of technical training, that would be really helpful. But I think, learning how to communicate what we’re doing is incredibly important. It’s striking to me that I’ve had the honor privilege to teach many, many, many incredibly smart students. And it’s not unusual for them to tell me that I’m over their head, even though I think I’m giving a simple explanation. I’ve had to think hard about that over the years and try and communicate better in a more basic way. But it’s a huge challenge for our profession. And one of the, I think, as you said, it gets us in big trouble.
Michael Livermore 28:52
Yeah, I mean, I think part of it is, you know, just as we’re thinking about this, I think part of it is that there’s, you know, it’s with any discipline, right, you start with some basic premises, and then you kind of start to build from there, and then you need to build from there and build from there. And what you end up with can be a fairly complex edifice that’s built out of fairly, that could be built out of simple parts. And that’s just the way it is. And it’s sometimes you know, to a certain extent, it might just be that this is irreducible, like in order to understand, you know, complex biology, you know, you need to know, the building blocks, right. And I think part of what is tricky about, about cost benefit analysis, is that we’re talking about public policy, right? So people don’t go into, you know, a lecture on, you know, recent findings in and in particle physics, or they don’t go open, you know, a scientific journal like Nature and Science and open to the genetic section, and expect to kind of understand what’s going on. Right, wherever As when we’re talking about, should we hire more crossing guards? Or even should we require more stringent air quality standards? Or should we require a rear facing camera in new automobiles? You know, these are questions in a democratic society. We think people, you know, people, people have opinions on, people feel that they ought to be able to have opinions on. And, you know, and maybe under some theories of democracy, we what ultimately, we do, as a government should reflect what people think about, you know, whether these are good ideas or not. So there’s this really big tension, because, you know, potentially be curious what you think about this. On the one hand, the fact is, we’re dealing with public policy questions in a democracy where people have often strongly held views. On the other hand, these questions can be approached with an edifice, basically, that is this cost benefit analysis that we’re talking about, that is a little bit more like a sophisticated scientific enterprise. And I think we need to be careful about that right to not, you know, there’s a lot of values questions as a scientific element to it. But it’s also a sophisticated moral language, whether we’d say that’s a highly articulated language about trade offs, and how you make those trade offs and so on. And so whatever it is, it’s a highly developed discourse, let’s call it that. I don’t want to imply that it’s a purely scientific discourse and make too many analogies to fields like biology, but it’s a highly articulated discourse. It’s a highly developed discourse. That not that it’s going to be difficult for most people to access without, you know, a pretty substantial amount of, of investment in of time to understand it. So so we have this dilemma. It’s almost like a dilemma. On the one hand, the highly articulated discourse that can be difficult for the uninitiated, on the other hand, we’re talking about public policy in a democracy, where folks are going to have opinions. Yeah, so what do we do about that? Like, what do you what? You know, you’ve confronted this issue many times over the years? How do you? How do you kind of think about this problem of negotiating these two? These two realities?
Lisa Robinson 32:16
Several points I’m starting with. I’ll get to your core question in a minute. But let me let me step back through a few other things first. So any form of research, I don’t care whether it’s particle physics, chemistry, or BAFTA cost analysis has a lot of uncertainties, all you need to do is look at the news reports about scientific findings. And you’ll see they get reversed, they get amended or changed in different ways. Uncertainty is a fact of life. But it’s also something that people don’t like very much, I would much rather have somebody say to us, if you don’t want to get COVID and do this, you definitely won’t get COVID Want to hear all these messages about? Well, you know, you could decrease your risk if you do this. Now. I’m saved from the question whether, you know, there’s lots, I probably shouldn’t be using COVID as an example, so
Michael Livermore 33:11
complicated. So it’s a complicated case, but whatever, you know, people don’t like uncertainty. That’s clear, right?
Lisa Robinson 33:17
Yeah. People don’t don’t like uncertainty, yet. There’s uncertainty everywhere. And that’s a challenge for those of us who who recognize that. The second is we always talk about benefit cost analysis as a way of informing decisions. I don’t think there’s anybody who’s a practitioner who thinks that benefit cost analysis should be used to make the decision, because there’s a lot of normative value judgments that underlie the framework. Um, it shouldn’t be, it shouldn’t be the only source of information because of those value judgments. But there’s also a number of things that it doesn’t address very well, it doesn’t address legal and political issues that doesn’t do a very good job of looking at the distribution of effects across people are advantaged and disadvantaged. So I think the one, one thing that we need to be making more clear to a general audience is, hey, we’re providing you with some information. But there’s, you know, lots of other things you should be thinking about, including your own moral beliefs and other things. I also think that that we really need to think carefully about audience because, you know, I know when I’m talking to a scholarly audience, there’s a lot of misperceptions about benefit cost analysis. But there’s a it’s a very different type of discourse than when I’m talking to, to the general public, one of the most effective things I’ve seen as I was working on the radiation protection standard years and years ago, and because you can’t see or feel radiation unless it gets really, really high. We were having real trouble communicating the amount of risk reduction that you would get, um, from this, from this regulation, and when other people in the project came up with this idea of having an illustration of how many hours you’d have to spend, it wasn’t even hours, I think minutes, you have to spend an airplane flying at 30,000 feet to get the same dose of radiation. And that’s what resume. And you know, I don’t think many people don’t really care about the things that we debate about. Academically, they just want to know, you know, how much is, you know, how bad is this going to be? How good is this going to be? And we need to figure out ways of explaining that, you know, that that that airplane exhibit could be reduced to a bunch of numbers and forming load. But the airplane worked a lot better in helping people understand what the benefits of this particular policy were. Does that make sense for us? I think that you and I live in an academic world where there’s lots of detailed technical debates about these things. But when I talk to people who don’t do this type of work, what they really want me to tell them is, is this a good thing? Is it a bad thing?
Michael Livermore 36:07
Well, that’s this is the tricky problem. This is exactly the tricky problem, right? Because that’s exactly what folks want to hear. And in some sense, I think this was kind of what your one of your early points was, which is cost benefit analysis, often, or analysts often don’t want to provide the answer, in part because they don’t they still don’t know. Right? There’s uncertainty, right? So there’s uncertainty about what regulatory consequences are going to be on the ground? How many, you know, what is the real risk reduction? There’s controversy about how to value those risk reductions. There’s other there’s potentially other values in play. Right, that that could they could conflict to this normative choices that go into cost benefit analysis. So like, kind of an All things considered, is this a good idea? Or is this a bad idea? Arguably, is something now I think that sometimes anyway, people say I think what you said was that analysts don’t want to provide that they don’t see that as the as what’s going on. It’s a it’s a, it’s a tool to inform analysis. But if the tool to inform analysis just says, Yeah, this is a good idea, or no, this isn’t a good idea, then, you know, it’s kind of it is it is ultimately making a quite a forceful recommendation.
Lisa Robinson 37:25
Well, I think there’s actually two two points there. The first is that we all those of us who are in the people who do this sort of work, or people who, like analysis, they’d like, you know, they’d like to get into sort through all these three things. And, you know, not everybody’s like us, but we get too much caught up in our own heads. So I think a lot of the oh, I can’t tell you anything hand waving is because we are so caught up in all the details of what we just did that we can’t step back and say, oh, wait a minute, you know, I can tell you that. Maybe Maybe my message is there’s so much uncertainty here that I don’t really know if the benefits of this policy are going to exceed the cost. So you mister missus decision maker, either need to figure out some other basis for the decision? Or maybe you decide not to do anything at all? Or are we I think most of the things that I work on, we can pretty clearly say that the benefits can exceed the costs, or the costs exceed the benefits, we just can’t tell you sort of right out to the decimal point with any certainty. So I think it’s a question of needing to get out of our own heads. And think about, you know, in simple terms, what’s the bottom line of what we’ve done? But also, I think when I say informed decisions, you know, the what I mean by that is, you know, I could say to you, Michael, this crossing guard policy that you’re thinking about, my benefit cost analysis says that the net benefits will be positive and could be significant. But the thing that I haven’t looked at carefully, or that I don’t know, that you Michael should take into account is the fact that most of these kids are getting hurt, or very low income kids. So, you know, we should add that into your decision making, along with the fact that I don’t know, maybe it’s illegal to grab kids in a crosswalk, you know. So I think the, we need to tell you tell a story about what’s in the analysis and what the analysis tells you. But we also need to be able to tell a story about other things that you might want to think about.
Michael Livermore 39:33
Well, the legal part, you know, that’s, you know, that that agencies always have to deal with their live their constraints and what they can and can’t do, right. So we can say, well, this policy would be great if, if you could do it, but you don’t have the power to do it. Right. In some sense. It’s fairly straightforward. The other issue maybe we could get into a little bit because this is a more of a I don’t know if to call it a cutting edge issue. It’s an issue that has been long discussed in policy circles, right. So Should we be? How should we accounting for the distribution of costs and benefits? And so the in the example you gave that was easy case, I’d be like, Look, there’s more precaution and benefits. And the benefits will mostly go to folks who are, you know, less well off? Well, that’s, that’s, that’s fantastic. The real issue comes up when there’s a conflict, right? So we say, Okay, well, we we do our cost benefit analysis, which still, at this point, even the simple, simple case, there’s something a little mysterious about it for the average person, like, we look at the populations, we look at years, we look at these risks that that today, we have this value of statistical life that we’re using, you know, we could explain where that comes from. But it’s complicated. Anyway, we do our analysts thing, and we come up with the bottom line, which says, the benefits, you know, are going to be greater than the cost. And we could even say that we looked at a suite of policies, and this is the one that has the highest net benefits will be, you know, adding crossing guards, you know, then that’s, that’s what’s going to make most sense, as opposed to changing around the routing for the kids or changing the school day, or putting in a cross a stoplight or whatever else. And so, so anyway, we looked at a suite of policies, this is the one that has the highest net benefits. But you know, it’s mostly going to help advantaged, you know, highly advantaged kids, the alternative policy that we looked at actually would have lower net benefits, who would cost a little bit more, but it would also change who’s going to be advantaged by the policy. And it’s actually the case that the kids who would be helped by the second policy, which let’s say, is rerouting traffic, the rerouting traffic policy, again, which is costlier, but it’s going to actually help more disadvantaged kids. So that’s where you have a conflict between what might be appealing on cost benefit grounds were the one that has the highest net benefits, versus one that might be appealing on distributional grounds, the one that helps the least well off. So this is a hard, a hard kind of problem. So. So one possibility is that we just kick that to the decision maker, and we just say, look, we’ve done the the kind of standard cost benefit analysis. We’ve also looked at these distributional questions. And so we can tell you kind of who benefits and who, who bears different kinds of costs. And and then you policymaker, you need to decide. Another alternative would be to incorporate the distributional analysis somehow into the cost benefit analysis, which there’s different technical ways of trying to do that. I’m just curious what you know, broadly at that, at that level, where you have a conflict between, you know, what we might think of as normatively desirable distributional, or undesirable distributional effects versus efficiency effects, you know, just the total costs and benefits. How do you think that should? You know, given the state of knowledge now what what should we be doing? Should we be trying to should we do just lay it off after the decision maker? And that person has to do the calculus? Or is this something different we should be doing?
Lisa Robinson 42:59
So this is an area that I’ve been working a lot on recently? And I don’t think you phrased that as what should we be doing? And I think we’re still trying to figure that one out. You and I both have a strong background and regulatory analysis. But I think it’s important for us both to keep in mind that many, many benefit cost analyses are not done through regulations. And a lot of the work that I’ve been doing lately has to do with social programs, or with policies that don’t require a regulation. They just require somebody to fund the policy or government to decide
Michael Livermore 43:34
the government spending program space.
Lisa Robinson 43:36
Yeah, yeah. Well, you know, could be a foundation. Or it could be something that, you know, a private firm could do. And I think that that changes the role of the legal issues quite a bit. It’s not just that we have requirements for what we need to do for regulatory analysis that you’re in different worlds, you know, if it’s a foundation, you don’t have Congress, and the courts breathing down.
Michael Livermore 44:02
Yeah, do whatever they want. Illegal, almost whatever
Lisa Robinson 44:05
they want. They still have some constraints. But I think that the, it’s, as you know, from having seen the stuff that we’ve written, people do not do a good job of distributional analysis, even if all they’re trying to do is describe the distribution.
Michael Livermore 44:20
Yeah, we just just just to sorry to intrude here, but you’ve done a lot of work on this very question of just how is distributional analysis done at the at the federal level amongst agencies? And just this practical question of what are we currently doing? And so yeah, as you were saying, some of the findings there is that we’re not doing all that good of a job.
Lisa Robinson 44:40
Yeah. I mean, I think this isn’t just true for the regulatory stuff. That’s also true. And, you know, I’ve been doing a lot of work in global health and development, same thing, every single guidance document, there’s probably some exception out there that if anybody listens to this, they’re going to tell me exists, but I don’t think I’ve ever read a guidance to argument that doesn’t say, along with estimate of net benefits, you need to tell us the distribution across disadvantaged, almost never see that? I have many theories, why not? But I think one piece of it is that it’s often fairly easy to think about, or at least estimate how benefits are distributed, you know, you know, you know what the population is who gets heart disease. So if you have a policy that reduces heart disease, you can just estimate that it’ll be distributed the same way those base cases are unless there’s something about your, your policy that makes one subgroup affected more than the other costs are very difficult, because often costs are incurred by a government agency or by private industry. And you need to think about how is that agency or that industry going to react to them position with additional costs? Are they going to absorb it by decreasing their profits? Are they going to lay people off, we just people salaries, increase the prices of the product, and we know very little about that. And if you can’t estimate, who’s going to be affected by the cost and the benefits, it’s very hard to do any of these sort of more sophisticated approaches that look at, for example, people’s preferences for for distribution. As I think, you know, there are approaches for waiting costs and benefits by when I say by the, by the marginal utility of income, which is a fancy way of saying the fact that saying that a poor person. An additional dollar to poor person means a lot more than an additional dollar to a millionaire. But they’re also variants on that theme, like, what gets called priority Arianism, that try to provide greater weight to people are disadvantaged across a number of different dimensions might be health, it might be something else. But we can’t apply it. So all those approaches are being developed, people are experimenting with them. But in order to apply them, we need to get much better at this sort of starting point of describing who bears the benefits. I mean, who bears the cost and who receives the benefits. I think there’s ways of doing that. But we need to think hard about that. And we need to start implementing the ways of doing it rather than just waving our hands around and saying, oh, distribution is important. And then not addressing it in our analysis. But I think those sticky questions, the sticky questions about, okay, you’ve got something that has net costs, but with advantage port would be really good for poor people, I think, no matter whether you do some sort of waiting inside the analysis itself, or whether you just display the results without doing anything to weight, the effects on the advantage versus a disadvantage, it still comes down to normative judgment that needs to be made by somebody. And
Michael Livermore 48:11
right, this is that’s interesting, right? Somebody needs to make this normative judgment.
Lisa Robinson 48:15
Yeah. And, you know, if it’s government policy, it’s the government agency, or maybe the legislature, if it’s foundation spending program, it’s whoever makes their decisions. And I don’t know, I don’t, I think that’s outside the benefit cost analysis. I don’t, my job is, fortunately, maybe not to make these decisions. My job is to make sure that the person who is making this decision has some good information in front of them.
Michael Livermore 48:41
Right, okay, good. And so but just to maybe clarify or reiterate the point you’re making there, which is just, it’s actually quite difficult. So there’s just to be quickly summarize how this debate has unfolded over the past, you know, 20, odd years, you know, cost benefit analysis tends, basically, in its kind of traditional form focuses on aggregate costs and benefits. So if we take our Crossing Guard example, we’re not looking at whether the kids are advantaged or disadvantaged, we’re not looking at, you know, who pays the cost, whether it’s the city or individual drivers, if it’s like, traffic rerouting is the policy, we’re not interested in that kind of question. We’re solely focused on, you know, just something, everything just gets summed up and you look at the aggregate number, and you say, Okay, we know what are the benefits and costs look like in total, and a consistent critique that has been offered over the years is, well, that is, there’s something wrong with that, that we ought to as a society be sensitive to who bears the costs and benefits benefits. And in particular, we should care about costs and benefits that are borne more by people who are less well off. Exactly for the reason that you describe at least Lisa, which is the diminishing marginal utility of consumption, which is just exactly that, you know, a cost that’s imposed on someone and who is doesn’t have a lot of money, that’s a much bigger hit to their well being than it costs than the same cost that’s imposed on someone who has a lot of dough. And so that’s a kind of a very straightforward normative idea. And, and cost benefit analysis, again, in its most traditional form, is not sensitive to that, that basic normative impulse that many people share. And so, so the the line then says, as you note, in most guidance documents, and most of the dictates that agencies follow for doing cost benefit analysis, they will say something like, and you should do distributional analysis to write. But the problem is, it’s really tough, it’s really tough. And I think maybe we can just spend a couple of minutes investigating the difficulties there. And again, you’ve you’ve really, you know, got the the experience with, you know, real world analyses, and you’ve gone through and you’ve looked at agencies analyses and what they’re doing. So an example that I use sometimes is a very classic one, I believe it’s in the a four circular actually, the existing government wide guidelines on cost benefit analysis is, say you improve air quality in low income neighborhoods, then you might say, well, that is a benefit that accrues to low income people. Well, not necessarily, if everyone there is a renter, and their rents go up as a consequence of their their where they’re living the property is now more, there’s more valuable, there’s more demand for that, for that real estate, as a as a rental unit. So the landlords charge more money, and they end up extracting some of that benefit. And so actually, the benefit goes to wealthy landowners. So you might think, in the first instance, that the benefit goes to low income people, but it actually there’s a kind of a transfer there that happens in the marketplace, where the land lords extract some of the benefit.
Lisa Robinson 51:59
Well, that’s even worse than that. If you clean up the air on that data, that neighborhood, well, here, people are gonna move in exactly the
Michael Livermore 52:07
dentist a gentrification issue, right? So then you’ve got over time, so even so there’s all these complex dynamics that affects so we might have a benefit or a cost that in the first instance is the benefit goes to low income people, right? Because they’re living where there’s the air quality, and the cost might be imposed on say, a local polluter, but then the cost can get shifted to consumers, maybe the effects the taxes that are paid, and the like, so So what do you think of that? I mean, you know, someone might say, Okay, this is just too complicated, this is impossible. And so we should really focus cost benefit analysis on aggregate effects, just looking at wherever the incident, the first instance is, and not trying to tease through all these distributional effects. Or, as your sense, you know, given that you’ve you know, really have gotten into this in a in a bunch of different contexts, that actually, it is worth the candle that we can do this and might be complicated, but we can do complicated things. And it’ll be imperfect and uncertain, but that what the information that we would get out of that would have at least enough value to justify the analysis. Make you
Lisa Robinson 53:14
just describe why it is that doing this sort of work is so fun, it’s funny talking. I’ve been working for long enough, a lot of the people I’m surrounded by are retiring, but because they feel like they’ve been doing the same thing over and over again for years. But there’s no danger of ever doing that if you do benefit cost analysis, because there’s so many interesting issues to explore. But um, I think that, you know, what is this really, really gets back to why what brings people like me to benefit cost analysis. I fundamentally believe that benefit cost analysis is an important contributor to good policy decisions and by good policy decisions and policies, policies that will increase the welfare of the people affected. When I maybe I can tell you a short story, when I first started working, I was I worked with maybe some Tolstoy, how many people says, I worked. I worked in someplace I worked in let me keep this big. I worked in a federal agency where we had the experience over and over again, of political appointees making what seemed like arbitrary decisions for legislative folks, you know, they’d say, Oh, I’m just going to I’m going to do X because my buddy is thinks that’s the right thing to do. Or I’m going to do X, because I feel like disagreeing with you. I mean, that literally was what I had somebody tell me once. No rationale. And this is a benefit cost analysis is a way of pushing back against that of saying, Here’s some information that I’m hoping you will take into account in your decision making, um, instead of making these arbitrary decisions, and So, but if, if my goal in terms of doing this type of work, which I think is not just my goal, I think it’s the goal of most of us who do this sort of analysis is to improve the welfare. The people are affected. Distribution is an important piece of that. And I really don’t think we can ignore it. So it’s, you know, you’re right. It’s hard. That’s what makes it fun and interesting. But it’s also important, and I think we need to, we need to work on it.
Michael Livermore 55:29
Now, are there cases that you’ve come across where you’ve seen that you think that an agency has done a good job at some kind of distributional analysis?
Lisa Robinson 55:39
It’s interesting, that’s interesting that you asked that because I think that recently, at least in the US, there’s been a lot of pressure to improve distributional analysis. And I have not gone through everything that federal agencies are doing, to try and see whether some of them have gotten far enough along to deal with the cost side, because that’s where that’s where the challenges are, I think there’s lots of people, lots of agencies and scholars who’ve been able to think about the distribution of benefits, mortality and morbidity, risk reductions and other types of benefits. The one place where people have done a lot of this is with macro economic modeling, where you have really large policies that you can feed into these. These big computer, more general equilibrium models, because they allow you to, in different ways, often model households, they have mechanisms that allow you to estimate the effects on on the rich and the poor, but even there, they’re making underneath all the complexities models are some very strong assumptions about things like whether costs are passed down as changes in prices. And is this not models tend to be useful only for very large policies, because there’s enough uncertainty in them that smaller policies sort of get lost in the error term, you don’t really end up with a good understanding of the effects. But I think that’s an interesting question. And one of your next shows you should pull out a bunch of the federal economists who are working on this sort of stuff and see if they’ve they’ve made some progress on this, because I really don’t know. And the cost side, because I really don’t know.
Michael Livermore 57:22
Yeah, it’s tough. I mean, it’s it’s, you know, it’s a tough issue. I think that, you know, maybe the kind of the final just note on all of this is, you know, it’s easy to talk in very general terms about, you know, well, we should look at cost and benefits, we should kind of examine the pros and cons of our policies, we should do things like account for the distribution of those costs and benefits. And I think if there’s, if there’s a lesson for for folks who, who aren’t deeply steeped in all of these questions, if there’s a lesson from our podcast today, it’s that, you know, it turns out that there’s just it’s actually, even though conceptually fairly easy to state, the goals of this research program is very, very complex to carry out in practice is conceptually complex. There’s enormous data requirements, there’s careful thought that’s required, there are assumptions that are going to be made and so on. And so So you’ve stuck with it, you know, that notwithstanding all of that complexity over the years, and so, yeah, I’m just curious if you have any kind of parting thoughts on, on your, on your general takeaway from the from the fact that, you know, that this is a very sensible thing to do, at least, you know, arguably, there’s a lot of a lot of smart people who think that it’s a sensible thing to do. But it is also a very difficult and complex, complex thing to do. I think,
Lisa Robinson 58:50
if there’s people out there listening to this, who are looking for something that’s incredibly interesting and challenging to do and who, and who are very interested in increasing the social welfare of their countries state region, the world, this is a great, great sort of thing to be working on. It’s, I think it’s really exciting. It’s really challenging. It keeps your brain keeps your brain Young. Can I say that now that I’m over 65. And I think it’s incredibly important. And there’s so much that we can do both to extend, extend the framework into different policy areas, but also to improve it.
Michael Livermore 59:29
Yeah. Great. Well, thanks so much for a really fun and interesting conversation today, Lisa. It’s always fun chatting with you, and thanks for all the great work that you’ve done on these issues over the years.
Lisa Robinson 59:40
Well, thank you for inviting me on it’s always great to talk to Mike
Michael Livermore 59:43
and listeners. If you enjoyed this episode, please let us know. You can give us a like rating subscribe to the podcast and follow us on social media. It’d be great to hear from you till next time.
Free Range Podcast 