MARY LOUISE KELLY, HOST:
Say you're a spy and you want to know who will win the upcoming presidential election in Brazil or what the spot price of Brent crude oil will be on a certain date a year from now. Well, you could turn to gadgets and gizmos and satellite intel, networks of agents, or you could outsource it to the wisdom of crowds. Here, I'll let him explain.
SETH GOLDSTEIN: My name is Seth Goldstein. I'm a program manager at the Intelligence Advanced Research Projects Activity.
KELLY: IARPA, as it's called.
GOLDSTEIN: That's right.
KELLY: IARPA being the office inside the intelligence community that is devoted to thinking outside the box, thinking of new ways to do stuff.
GOLDSTEIN: I think that's a fair description.
KELLY: And the challenge you've got on the horizon coming up next month is you are launching a crowdsourcing challenge. Explain.
GOLDSTEIN: That's right. We're launching a challenge called the Geopolitical Forecasting Challenge. What we're going to do here is allow anyone who's interested - citizen scientists, hobbyists - to develop forecasting methods to attempt to anticipate what events are going to occur sort of in real time.
KELLY: So I mentioned, you know, one example of the type of things you're asking people to forecast. You know, the price of Brent crude oil on a given day in the future. What other questions might you throw to people to see if they can forecast?
GOLDSTEIN: For instance, will the United Nations vote to impose new sanctions on some particular country, say, by the end of the year?
KELLY: OK.
GOLDSTEIN: We might ask that question about multiple countries. Or will any additional EU country vote to depart the EU by the end of the year?
KELLY: And there's prize money at stake here.
GOLDSTEIN: There's $200,000 in prize money at stake. We're going to be providing them with state-of-the-art data on crowdsourced human judgments. And we know it's the state of the art because we've validated this state of the art in a previous IARPA research program called ACE. I won't tell you what the acronym means simply because it's got a lot of big words in it...
KELLY: Not because it's classified.
GOLDSTEIN: Not because it's classified.
KELLY: (Laughter) OK.
GOLDSTEIN: This program...
KELLY: Oh, come on. Just tell me.
GOLDSTEIN: Aggregative Contingent Estimation.
KELLY: OK.
GOLDSTEIN: So this program was about improving on what was then a very limited state of the art in crowdsourced forecasting. We achieved that in that program. We beat the existing state of the art by greater than 50 percent. You all did a story on it.
KELLY: Yeah.
GOLDSTEIN: And taking that method as the state of the art, IARPA is launching this Geopolitical Forecasting Challenge by giving those who are interested access to a data stream based on the method that was so successful there.
KELLY: So here's my question - why does a crowd of individuals stand a better chance of forecasting something accurately than one really smart person sitting there with a classified security clearance who has access to all of the information that the intelligence community has?
GOLDSTEIN: The problem with expert judgment generally is that it's difficult to know in advance which expert is going to make a correct forecast on any particular event.
KELLY: Inevitably, somebody might get it right, somebody might get it wrong.
GOLDSTEIN: Somebody might get it one time and they might get it wrong another time. The idea behind crowdsourcing is that if you assemble a reasonably sized crowd, a large crowd of hundreds or even thousands of people making judgments, the idea of there being any particular directional bias in some aggregate of that judgment is reduced. It's not to say experts can't make accurate forecasts. It's to say if you had to choose a method, this might serve you better.
KELLY: It seems like there's a conundrum inherent in here for you. If crowdsourcing by ordinary civilians can out-forecast, you know, the smartest minds of the intelligence community, if this project succeeds, do you risk putting yourself out of business?
GOLDSTEIN: I don't think so.
KELLY: You hope not (laughter).
GOLDSTEIN: Yeah. I don't foresee that happening for a variety of reasons. Expert judgment is always going to be innately valuable. Experts provide critical context about causal understandings of current phenomena, past phenomena and so forth. So, no, I don't think that we're going to put sort of the intelligence analysts out of business.
KELLY: So, Seth Goldstein, I have to get you on record and ask, will another country depart the EU and go the Brexit route before the end of the year?
GOLDSTEIN: I'm going to leave that to our research teams and our prize challenge entrants to tell us what the right answer is.
KELLY: Outsource to the wisdom of the crowds. Thank you so much.
GOLDSTEIN: Thank you.
KELLY: Seth Goldstein of IARPA, Intelligence Advanced Research Projects Activity. Transcript provided by NPR, Copyright NPR.
