Ian Ayres and John Donohue have a strongly worded letter to the editor that they have apparently sent to the Post-Dispatch. First is my letter to the editor:
St. Louis Post-Dispatch, July 21, 2003
Valid gun research
On March 9, and again on July 12, the Post-Dispatch has published editorials critically discussing my research regarding concealed-carry. These pieces have made it appear as though the academic debate is just between myself and a couple of critics. That is not the case.
Academics who have published refereed research in academic journals showing that right-to-carry laws reduce violent crime include: Carlisle Moody, David B. Mustard, John E. Whitley, David E. Olson, Florenz Plassmann, Nicolaus Tideman, Tomas B. Marvell, Mark A. Cohen, Stephen G. Bronars, and William A. Bartley. While some other studies claim the laws produce no change in violent crime rates, among all the national studies that have been done, there is not a single refereed academic publication concluding that these laws produce a significant increase in violent crime.
The Brookings Institution study that the Post-Dispatch cites was not published in a refereed journal. A version of that paper recently appeared in the Stanford Law Review, a student-edited, non-refereed journal. That research finds a temporary small increase in violent crime, followed by a drop in crime. Yet next to that article in the same publication appears another study by Plassmann and Whitley, who examine three additional years worth of data and find “annual reductions in murder rates between 1.5 and 2.3 percent for each additional year that a right-to-carry law is in effect. . . . the total benefit from reduced crimes usually ranges between about $2 billion and $3 billion per year.”
John R. Lott Jr.
Resident Scholar, American Enterprise Institute
LOAD-DATE: July 21, 2003
Here is their response that they e-mailed to Glenn Reynolds (August 20, 2003, posted at 12:40 PM):
On July 21, 2003, researcher John Lott wrote a letter to the editor in which he tried to shore up support for his now discredited theory that state adoption of laws allowing citizens to carry concealed handguns will lower crime. Although he refuses to acknowledge this fact, we showed in a recent article published in the Stanford Law Review that when the coding errors in his own data set are corrected, his own regressions show no such drop in crime. Lott tries to distract the readers by stating that our study illustrating his coding errors was only published in a law review, rather than in a peer-reviewed journal. But since Lott knows that merely correcting his errors did eliminate his finding, it is dishonest for him not to concede the fact. How can we assert that Lott knows that we correctly identified coding errors in his data? Because he had put his data set on his own web page before we found his errors, and he has now gone in and quietly corrected the errors that we identified. Lott should put an end to the charade and acknowledge that his own most recently published regressions, when corrected, offer no support for the more guns, less crime hypothesis. As the latest, peer-reviewed, article on this topic by researchers Tomislav Kovandzic and Thomas Marvell states: “we find no credible statistical evidence that increases in permit rate growth (and presumably more lawful gun carrying) leads to substantial reductions in violent crime, especially homicide. Similar to Ayres and Donohue (2003), we find that our best, albeit admittedly imperfect, statistical evidence indicates that increases in permit rate growth may actually lead to slight increases in crime.”
Professor Ian Ayres, Yale Law School Professor John Donohue, Stanford Law School
On substance there are several points:
1) The 1977 to 2000 county level data set has been up on my web site since February. To make the entire process easily accessible for others, the regression files are also available. Anyone who wants to critically examine the data set is thus completely free to do so and can compare it to the corrected output and to the paper file, both of which can be downloaded from the site. The data set was corrected, and the results labeled “corrected” figures and tables have been up since April. With well over 70,000 observations and over a hundred variables available in the data set, we are dealing with a few hundred data entries that contained mistakes. That amounts to thousandths of one percent of the data entries.
I have previously discussed these issues on this web site (6/7/03): “There are also papers by Plassmann and Whitley in the same issue of the Stanford Law Review as Ayres and Donohue’s piece (an earlier version of the Plassmann and Whitley paper with me as a coauthor is available here and see also the data and corrected results are available at www.johnlott.org ) . . . .”
I am not quite sure what to make of Ayres and Donohue’s claim: “How can we assert that Lott knows that we correctly identified coding errors in his data? Because he had put his data set on his own web page before we found his errors, and he has now gone in and quietly corrected the errors that we identified.” This is not what transpired. During a call that I made to Ian Ayres back in January, he asked me for the data, thus prior to me putting the data on the web, and I immediately sent it to him, along with all the regressions so that it would be very easy for them to check the research. This should be easy to verify through the editors at the Stanford Law Review and I should have documentation on fedexing him the data. They informed us of their points in their reply and the corrections were made as noted above on my web site.
As to the claim that “correcting his errors did eliminate his finding.” The data used in the Plassmann and Whitley paper can be downloaded at www.johnlott.org and one can readily see from the corrected tables and figures that while there are some changes in the results, this statement is false. Florenz Plassmann also has a detailed response here. For an additional discussion interested readers can also go here.
Unlike Ayres and Donohue, I have endeavored to make the data readily available in a timely manner and to explain how it was constructed.
Updated paragraph 8/22 (minor edit on 9/19/03) Additional statement with respect to point 1: The estimates do change somewhat, but the basic point is still clear. Whether one uses the types of statistical analysis that Ayres and Donohue use for all their county level regressions or whether one uses the type of methodology that Plassmann argues very strongly for in the Stanford Law Review piece with John Whitley, you still get a drop in crime. Clustering with the least squares esimates does affect some of the results, but Ayres and Donohue do not use clustering for any of their own results and do not discuss Plassmann’s statements about using Poisson regressions and the “major” problems attempting to employ a linear specification on nonlinear data. Despite the Poisson regressions being the primary estimates in the Plassman and Whitley paper they are ignored and they continue to show significant drops in violent crime rates after the law is passed. It is also not clear that applying STATA’s clustering command to all counties within a state provides an adequate solution to the problem of cross-correlation. First, clustering by state requires that the law has the same affect across all counties within the state and that is obviously not true with right-to-carry laws. The vast majority of states adopting right-to-carry laws had may issues laws to begin with and there was a large variation in the rate at which permits were issued in those counties (e.g., with more rural less densely populated counties issue permits at a much greater rate). In addition, while it is possible that error terms are correlated across jurisdictions within a state, the more important correlations may be among neighboring jurisdictions across state lines. (Ayres and Donohue’s reply where they first use clustering does not state when they are using clustering or not so I have shown what happens when clustering is not used.) Professors Eric Helland and Alex Tobarrok paper also provides an interesting approach that deals with this problem in an novel way and shows that the results are significant even after these correlations are accounted for.
2) Despite their continuing claims to the press, Ayres and Donohue’s own papers do NOT provide any statistically significant evidence that violent crimes increase. I suggest that interested people carefully read Plassmann and Whitley’s paper. But let me draw the readers to a couple of points:
a) The “Hybrid” Estimates imply an initial increase in crime
Plassmann and Whitley write (p. 1360, Section D):
The confusion apparently arises because Ayres and Donohue concentrate solely on the significance of the post-passage dummy itself. For example, take their hybrid estimates for murder using the 1977 to 1997 sample in row 6 of Table 10. The post-passage dummy for murder equals 6.9 percent and is statistically significant at the 5 percent level. But for the first year that the law is in effect, the net effect on the crime rate is the sum of the 6.9 percent dummy plus the -3.5 percent post-law crime trend. The net effect in the first year is 3.4 percent, but with a standard error of 2.9, which is only 1.2 standard deviations from zero. In fact, none of Ayres and Donohue’s hybrid estimates for murder, rape, or robbery in Tables 10 or 11 imply a net effect that is much more than one standard deviation away from zero. This is not particularly surprising since even the earliest non-linear results provided by Lott found this pattern for aggravated assaults.
b) State-by-state breakdown of effects
There are many problems with the way that these regressions are reported (again see Plassmann and Whitley), but the bottom line is that Ayres and Donohue fail to discuss the statistical significance of the overall effect. They simply note what way individual states go and discuss the weighted average for the effect, but there is no discussion of whether this average effect is statistically significant — and of course it is not. In statements from the LA Times to their letter to the Post-Dispatch, they claim that there is an increase, but they never offer any statistical evidence to support this claim.
3) Ayres and Donohue misstate my claims in their letter with respect to the Kovandzic and Marvell paper (I didn’t know it was accepted for publication) . However, my letter specifically mentioned that some papers did claim to find no changes in crime from right-to-carry laws. I have problems with the Kovandzic and Marvell piece, but the point is the Kovandzic and Marvell paper doesn’t contradict anything that I wrote in the letter.
4) Ayres and Donohue’s tone is extreme, especially in comparison to my language. Unfortunately, their piece is not an aberration. Donohue’s recent comment on Kovandzic and Marvell in Criminology and Public Policy uses a variety of outrageous language from “blood on Lott and Mustard’s hands,” “discredited,” mentions of “fraud,” to a “blight on democracy” and “harm to the democratic process.”
Furthermore, while I have freely shared my data, Ayres and Donohue have not. In a paper on Abortion and Crime, Donohue obtained results that John Whitley and I questioned. During the 19 months it took to receive any data, John and I reconstructed the data on our own but it did not match what they had used. Both John Donohue and his co-author refused to answer any of our questions on how the data was constructed. Ayres has also declined my requests for his data in the past, and my attempt to reconstruct what data is publicly available did not produce the same results that he claimed to obtain (e.g., p. 257, fn. 28).
Yet, I have never used the language that they used in describing those papers.
8/22/03 Some new interesting research on concealed handgun laws bears directly on this debate.
A new research paper has an important new approach towards estimating statistical significance. Professors Eric Helland and Alex Tobarrok conclude that:
“the cross equation restrictions implied by the Lott-Mustard theory are strongly supported.”
“Surprisingly, therefore, we conclude, that there is considerable support for the hypothesis that shall-issue laws cause criminals to substitute away from crimes against persons and towards crimes against property.”