本ブログでも折に触れ取り上げてきたAutor-Dorn-Hanson論文(cf. ここ)にJonathan Rothwellが反論を寄せ、Autor-Dorn-Hanson再反論した。それを見たRuss Robertsが実証研究の正確性全般に疑問符を付ける表題のエッセイ(原題は「What Do Economists Actually Know?」)を書いたところ、ジョン・コクランDon Boudreauxが肯定的、Adam Ozimekノアピニオン氏が批判的な反応を示した。

Is Rothwell correct?
I have no idea. Here is what I do know. There is likely to no way of knowing which view is correct with anything close to reliability or certainty. ...This is about interpretation. Like most of the questions I listed above — questions about the impact of the minimum wage, or the cause of the financial crisis, there is no simple way to resolve differences in analysis done by professional economists.
...What usually happens is that very smart well-trained people on both sides of the issue argue. They argue over the structures of the models and assumptions that enable the empirical conclusions, or the quality of the data or whether a finding is generalizable or not. Eventually, sometimes a consensus emerges but that consensus can be reversed by further empirical analysis. This consensus is something like a market in ideas. It’s something like the two sides in a trial — one hopes the process yields truth more often than not.
But there is no way of knowing reliably if the consensus reflects the truth. It may rely instead on the underlying biases of the prosecutors and defendants in the intellectual trial of ideas. Or where they received their PhD degrees. Or the fashionability of certain positions over time as society changes. Unlike product markets where poorly made products are punished by low prices or fewer and fewer consumers, there are no clear feedback loops in the world of academic economics. You can say something that is wrong and the price you pay may be zero. In fact you may be rewarded.
And that is because of what does not happen. There is never a clean empirical test that ultimately settles these issues. There is no reliable scientific experiment where each side is forced to make a prediction and the results settle the matter.


One response to the questions I am raising is that we have new techniques that solve a lot of the problems I’m talking about. We’ve had what’s called (by the creators of the new techniques) the credibility revolution. These new empirical techniques have allowed us to run quasi-experiments that while not perfect, eliminate many of the problems of complexity and multi-causal reality. I remain a skeptic. But maybe the champions of the new empirical techniques will convince the skeptics. We’ll see.
Young economists are enthusiastic about these quasi-experiments. As one economist once told me — I don’t rely on theory, I just listen to what the data tells me. But numbers don’t speak on their own. There are too many of them. We need some kind of theory to help us decide which numbers too listen to. Inevitably, our biases and incentives influence which numbers we think speak the loudest.

この後Robertsは、ジンガレスの経済学の虜論(cf. ここ)を持ち出している。そして、以下のように述べている。

What I am saying in this essay is that while economic fundamentals like income or even changes in income over time are somewhat measurable with some precision, we are notoriously unreliable at the things the world really cares about and asks of our profession: why did income for this or that group go up by this or that amount? What will happen if this or that policy changes? Should the subsidy to college education be increased or decreased and if so, by how much? These much-demanded answers for precision and an understanding of the complex forces that shape the world around us are precisely the questions we are not very good at answering.
The fact that economists relentlessly and cheerfully do their best to answer them anyway might be because we are tenacious and optimistic that we are getting closer to the truth over time. A simpler answer uses the economics of the kind Luigi Zingales talks about: those who purport to “know” the answers to causal policy questions get attention, money, and influence. That doesn’t prove that they are wrong, of course. Sometimes incentives encourage good outcomes. But as I suggested earlier, I’m not convinced that the feedback loops of profit, loss, and competition in product markets are as effective in the market for ideas. Or as Brian Nosek, Jeffrey Spies and Matt Motyl put it:

Published and true are not synonyms.

このエッセイで私が述べているのは、所得といった経済の基礎指標は――あるいは時系列的な所得の変化さえも――ある程度の正確性を以って計測できる半面、世間が本当に問題として経済学者に問い掛ける事柄に関しては、我々は頼りないことで悪名高い、ということである。その事柄とは次のようなものである:なぜこれこれのグループの所得はこれこれだけ上昇したのか? これこれの政策が変更されたらどうなるのか? 大学教育への補助金は増やすべきか減らすべきか、してその増減額は? こうした正確性および我々の周りの世界を形成する複雑な力の理解が強く望まれる回答は、まさに我々がうまく回答できない質問である。
それでもとにかく経済学者が最善を尽くしてこれらの問題に粘り強く進んで回答しようとすることは、我々は段々と真実に近付いている、ということについて終始楽観的であるためかもしれない。もっと簡単な理由は、ルイジ・ジンガレスの唱える経済学を使うならば、政策の因果関係の問いへの回答を「知っている」と主張する人は注目と金と影響力を手にするから、ということになる。もちろん、そのことは彼らが間違っていることを意味するわけではない。インセンティブが良い結果をもたらすこともある。しかし先に述べたように、製造物市場における利益、損失、競争のフィードバックのループがアイディアの市場で同様に効果的であるとは私は確信してはいない。あるいは、Brian Nosek、Jeffrey Spies、Matt Motyl*2が言うように




What I am arguing here is that the combination of economics with statistics in a complex world promises a lot more than it delivers. We economists should be more humble and honest about the reliability and precision of statistical analysis.