Journal of Criminal Law and Criminology (Northwestern)
87 (1997): 1462.
Posted for Educational use only. The printed edition remains canonical. For citational use please visit the local law library or obtain a back issue.

A CALL FOR A TRUCE IN THE DGU WAR
Tom W. Smith *

Copyright © 1997 Northwestern University School of Law & Tom W. Smith

For almost a decade scholars have been debating about how many defensive gun uses (DGUs) occur annually. Gary Kleck and colleagues, [1] citing a series of polls culminating in the 1993 Kleck-Gertz survey, argue that at least 2.55 million people use a firearm for protection against criminals each year. Hemenway and others, [2] relying on the National Crime Victimization Surveys (NCVSs), contend that only about 55,000 to 80,000 victims use guns against offenders in a given year. The estimates are wide apart and their academic champions staunchly defend their respective figures as correct and accurate, while dismissing the opposing figures as invalid and implausible.

Neither side seems to be willing to give ground or see their opponents' point of view. This is unfortunate since there is good reason to believe that both sides are off-the-mark. Below the main shortcomings of the two approaches and some of the keys issues of contention are discussed.

First, it appears that the estimates of the NCVSs are too low. There are two chief reasons for this. First, only DGUs that are reported as part of a victim's response to a specified crime are potentially covered. While most major felonies are covered by the NCVSs, a number of crimes such as trespassing, vandalism, and malicious mischief are not. DGUs in response to these and other events beyond the scope of the NCVSs are missed.

Second, the NCVSs do not directly inquire about DGUs. After a covered crime has been reported, the victim is asked if he or she "did or tried to do [anything] about the incident while it was going on." Indirect questions that rely on a respondent volunteering a specific element as part of a broad and unfocused inquiry uniformly lead to [Page 1463] undercounts of the particular of interest. [3]

However, some other proposed reasons for under-reporting on the NCVSs are questionable. The claim that DGUs are under-reported because the NCVSs suffer from "the taint of being conducted by, and on behalf of employees of the federal government" [4] and that respondents see themselves in effect as "speaking to a law enforcement arm of the federal government" [5] is improbable. The survey literature does not indicate that Bureau of the Census surveys are held in special suspicion. [6] If anything, it indicates that cooperation is greater than usual in part because of the high quality of Census interviewers and because most people accord the Bureau of the Census more legitimacy than given to other surveys. [7]

Second, the estimates of the Kleck-Gertz study and other cross-sectional surveys using a direct question are too high. First, unlike the panel NCVS which uses bounded recall to minimize telescoping (i.e., the misreporting of past events as having occurred within a more recent specified time period), nothing in these surveys mitigates against such over-reporting. Kleck and Gertz (K-G) are correct to note that forgetting would tend to off-set errors from telescoping, [8] but these two cognitive errors are rarely balanced. While no definitive study of the relative telescoping versus forgetting rate for DGUs exists, given a one-year reference period and the saliency of DGUs, it is likely that telescoping is greater than forgetting. [9]

Second, there is a significant amount of sampling error around the direct DGU estimates. While K-G are correct that, broadly speaking, all of the direct estimates are compatible with one another (i.e., probably mostly within the confidence intervals), [10] this is in part because the sampling variation is often rather large. To say that the [Page 1464] high-end K-G estimate is not statistically implausible, given results from other similar surveys, is not to say that it is correct. What is needed is a meta-analysis that takes comparable estimates from similar surveys and produces the best overall estimate. Since the K-G estimate is near the high end of the range of estimates based on national samples covering specific reference periods, this means that the best estimate is lower. Giving equal weight to all estimates, [11] the composite annual estimate based on one-year recall would be 1.81- 2.01 million, the annual estimate based on five-year recall period would be about 1.34-1.38, using the K-G multiple occurrence adjustment, and around 0.9- 1.0 without that revision (Table 1). [12]

Table 1 Number of Adults with DGUs Per Annum (millions)
Study	Variant	Based on One Year Recall	Based on Five Year Recall
K-G 1993[13]	A	2.55	1.88
K-G 1993	B	2.16	1.68
K-G (Hart, 1981)[14]	--	--	1.80
K-G (Mauser, 1990)	--	--	1.49
K-G (Tarrance, 1994)	--	--	0.76
NSPOF[15]	1	1.46	0.65
NSPOF[16]	2	1.46	0.97

[Page 1465]

Third, as Hemenway (H) points out, the K-G estimate is likely to suffer from false positives, [17] although the situation is not nearly as clear as H asserts. As K-G note in their response to H's critique, the basic medical misreport model assumes that the errors are random. [18] As such, the rarer the event the greater the over-reports because there are many more true negatives that can be "accidently" misclassified as false positives than there are true positives that could by chance be misreported as false negatives. In medicine, this problem is addressed by a definitive and independent follow-up test to confirm or refute the more error-prone, screening test. K-G in effect argue that they apply such a test, by asking up to nineteen follow-up questions to verify that the reported positive is a true rather than a false positive. [19] To their credit, they use these follow-ups to eliminate both probable and some possible false positives.

But there are two serious limitations to this procedure. First, the confirming test is not an independent test. Just the opposite. There is likely to be correlated error. If a person misreports a DGU on the screener, he or she is likely to misreport it in the follow-ups. For example, if respondents misreport real DGUs as occurring within the referenced time period due to telescoping, then in the follow-up questions they would report the details of the misreported incidents.

Second, neither the screener nor the follow-up questions are non-reactive. While people cannot consciously affect typical medical tests, they can consciously or unconsciously distort survey responses. For example, if respondents intentionally misreport a DGU, they could merely continue the deception in the follow-up items.

The follow-up questions are essential for collecting useful information of the characteristics of DGUs and can help to weed out some misreports. But they will only reveal a limited number of errors such as if the interviewer misheard or miscoded the initial response as a "Yes" or when the respondent misunderstood the screening question and reversed the initial response in light of the follow-up questions that made clear the meaning of the initial question. Many other misreports will be undetected.

The key aspect of the argument over how much distortion occurs and in what direction it leans concerns respondents' motivation to report accurately. H contends that there is a strong social desirability effect that would lead people to over-report DGUs (either via exaggeration, increased telescoping, or outright fabrication). [20] He believes [Page 1466] people see DGUs as heroic acts that reflect well on themselves. [21] K-G however in their rebuttal contend that the social desirability effect works in the reverse. [22] They repeatedly argue: (1) that "most of the reported DGUs involved illegal behavior on the part of Rs." [23] and "DGUs typically involve criminal behavior" [24]; and (2) as a result, that "people are far more likely to fail to report illegal behavior in which they have engaged than they are to falsely report illegal behaviors in which they have not engaged." [25] Both authors are correct about the impact that social desirability would have if respondents on balance saw DGUs as respectively heroic or criminal.

Unfortunately, there is no empirical evidence as to how people view DGUs. To establish social desirability effects two types of evidence are needed: (1) validation studies that document that people consistently over or underreport the true level of a behavior; and (2) studies that indicate that self- presentation (rather than cognitive error such as misunderstanding the question or some imbalance of forgetting and telescoping) is the source of the misreports. No such studies apparently exist for DGUs.

However, one part of K-G's argument about possible social desirability effects is questionable. [26] Neither their study (at least as reported), nor any other study really demonstrates that "most" DGUs involve illegal actions or "typically" or "usually" involve criminal behavior on the victim's part. K-G's data are too imprecise to ascertain this in general. [27]

Finally, as K-G acknowledge, various statistics from their survey are wrong, questionable, or severely limited: (1) DGUs by household members other than R are under-reported; [28] (2) DGUs estimates based on five-year recall are inconsistent with and lower than the one-[Page 1467] year estimates; [29] (3) the wounding rate is too high; [30] (4) the incident hit rate is too high; [31] and (5) in general, figures based on just the DGU cases are subject to considerable sampling variation and the less accurate five-year reports and household incidents have to be relied upon in order to get enough observations for any analysis of the characteristics of DGUs. [32]

Another questionable finding, and one echoed by the NSPOF, [33] is that a high proportion of DGUs are carried out by women. K-G find that 46.3% of defenders are women [34] and the NSPOF finds that 41.2% are women. [35] Given that the best estimate is that only 20-21% of gun owners are women, [36] this means that women are twice as likely to use a gun defensively as one would expect. When one looks at the small number of DGUs that results in justifiable homicides, only 13.8% are committed by women. [37] If women are 21% of gun owners and approximately 14% of those who lawfully kill someone with a gun, it seems improbable that they would make up 41-46% of all DGUs.

In addition, further concerns come from figures relating to the level of gun usage in crimes, the absence of guns in some DGU households, and how many lives are thought to have been saved by DGUs. Likewise, in the similar NSPOF internal confusion and contradictions are noted. [38]

K-G are right to note that the details of DGUs are based on only 222 cases (or even less when sub-group analysis is performed) and that these estimates are much more subject to sampling variation than are the estimates of the level of DGUs based on 4,977 cases. [39] But the large number of doubtful figures about the particulars of DGUs do raise concerns about the reliability and accuracy of such reports. If many of the details about DGUs are suspect, this suggests that recall is unreliable and that the counts themselves may also be inaccurate. [Page 1468]

However, various other arguments against the K-G estimates do not shed much light on the situation. Figures on UFOs and the mentally ill are not particularly relevant or useful.

If we factor in some of the probable over- and underestimates affecting the NCVS and K-G 1993 survey, the widely divergent figures on DGUs draw much closer together. The latest figures from the NCVS indicate 108,000 DGUs per annum. [40] If this is adjusted for a 50% under-reporting due to not directly asking for DGUs, this increases the estimate to 216,000. Next, research by Cook and Ludwig suggests that perhaps 16-42% of DGUs involve crimes not covered by the NCVS. [41] Adding in these would raise DGUs to 256,500-373,000.

Similarly, using the average of the K-G one-year lower (B) estimate and the NSPOF figure gives a starting estimate of 1,810,000. Assuming a net cognitive over-reporting (telescoping-forgetting) of 50%, [42] reduces the figure to 1,210,000. [43] These estimates should draw even closer together if other measurement errors could be factored in. [44] But even as they stand, the gulf has been narrowed from 30+:1 to 3.2-5.6:1.

Of course, the above calculations are based on reasonable, but mostly unconfirmed, estimates of various error parameters. What is needed is less argumentation and speculation and more and better data. First, some additional information can be gained by refined analysis of the existing surveys (K-G 1993, NSPOF, NCVS, etc.). Each [Page 1469] of the surveys should be fully documented and archived at the Roper Center, University of Connecticut, and the Interuniversity Consortium for Political and Social Research, University of Michigan, for any researcher to use. [45]

Second, more studies are needed. These should include: (1) validation studies specifically designed to ascertain whether a social desirability bias exists for DGUs; (2) studies that experimentally vary factors that are believed to inflate or deflate DGUs reports to see (a) how robust reports are and (b) whether they are affected by the hypothesized factors; (3) taped descriptions of reported DGUs with detailed probes so that one can determine exactly what transpired, including such issues as whether (a) a criminal threat existed, its nature, and seriousness, (b) the DGU was probably legal, and (c) accounts are accurate and truthful; (4) trying alternative methods for measuring DGUs that might lessen both any self-presentation bias and cognitive error. One possibility would be to ask people whether they had handled or fired a gun in the last year and then ask about for what purpose it was used (e.g., hunting, target shooting, self-defense, etc.); and (5) the use of refined, direct experience questions on a large, high quality, panel survey with explicit corrections for telescoping. Adding a few questions to the NCVS would be the easiest way to achieve this. Only by such further careful, empirical research will the errors in measuring DGUs be understood and the true level of DGUs ascertained.

* National Opinion Research Center, University of Chicago

[1]. Gary Kleck & Marc Gertz, The Illegitimacy of One-Sided Speculation: Getting the Defensive Gun Use Estimate Down, 87 J. Crim. L. & Criminology 1446 (1997); Gary Kleck & Marc Gertz, Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun, 86 J. Crim. L. & Criminology 150 (1995).

[2]. David Hemenway, Survey Research and Self Defense Gun Use: An Explanation of Extreme Overestimates, 87 J. Crim. L. & Criminology 1430 (1997).

[3]. Seymour Sudman & Norman Bradburn, Asking Questions: A Practical Guide to Questionnaire Design, 36-45 (1982); Tom W. Smith, Trends in Voluntary Group Membership, 34 Am. J. Pol. Sci. 646, 647-52 (1990).

[4]. Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 166.

[5]. Id. at 155.

[6]. See, e.g., Richard Kulka et al., Self-Reports of Time Pressures, Concerns for Privacy, and Participation in the 1990 Mail Census, Paper presented to the Annual Research Conference, U.S. Bureau of the Census, Arlington, Virginia (1991); Panel on Privacy and Confidentiality as Factors in Survey Response, Privacy and Confidentiality as Factors in Survey Response (1979).

[7]. See Kulka, supra note 6; Privacy and Confidentiality as Factors in Survey Response, supra note 6.

[8]. Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 163.

[9]. See generally Ronald Andersen et. al., Total Survey Error: Applications to Improve Health Surveys (1979); William Foddy, Constructing Questions for Interviews and Questionnaires (1993); J. Neter & J. Waksberg, A Study of Response Errors in Expenditures Data from Household Interviews, 59 J. Amer. Stat. Ass'n 18 (1964).

[10]. Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 166.

[11]. A more statistically sophisticated meta-analysis could use varying weights for the studies to adjust for sample size and other factors.

[12]. K-G take the five-year rate, divide it by 5 and increase this result by about 50% to account for multiple occurrences during the five years which might have occurred in different years. See Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 166.

[13]. "K-G 1993" is K-G's own study and estimates. See id. at tbl.2 (A & B estimates as defined therein).

[14]. "K-G" (Hart, Mauser, and Tarrance) are K-G's 1993 estimates from other studies. See id. at tbl.1.

[15]. Philip J. Cook & Jens Ludwig, The Private American Arsenal: Results of a Comprehensive National Survey on Firearms Ownership and Use (1996) (unpublished report on file with author). Cook and Ludwig analyzed the National Survey of Private Ownership of Firearms (NSPOF). Estimate 1 takes their five year estimate and divides by five to get an annual estimate.

[16]. Id. Estimate 2 takes estimate 1 and increases by 50% to account for possible multiple DGUs that might have occurred in different years. This is approximately the adjustment that K-G used to get their annual estimates from five-year reports. See Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 166.

[17]. Hemenway, supra note 2, at 1435-37.

[18]. Kleck & Gertz, The Illegitimacy of One-Sided Speculation, supra note 1, at 1456-57.

[19]. Id. at 1449-50.

[20]. See Hemenway, supra note 2, at 1438-40.

[21]. Id. at 1438.

[22]. The issue is raised in a much more limited fashion in the initial article. See Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 156, 172-74. The authors do not state it so strongly, do not claim that DGUs are in general illegal acts, and focus on the impact of willingness to report on the NCVS, not on their own survey.

[23]. Kleck & Gertz, The Illegitimacy of One-Sided Speculation, supra note 1, at 1455.

[24]. Id. at 1452.

[25]. Id. at 1459.

[26]. See Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 163.

[27]. K-G state: "We made no efforts to assess either the lawfulness or morality of the R's defensive actions." Id. But elsewhere K-G infer that some of the used guns are illegal or the victim is not legally entitled to use or possess the weapons, that other guns were illegally carried prior to use, and that in other cases the victim was actually the aggressor. See id. at 156. There is no real evidence on the first and last points and the evidence on the middle point is inconclusive, but probably points to most DGUs not involving illegal carrying. K-G's estimate that 36-64% involved illegal gun carrying is on the high side. See id. at 174.

[28]. Id. at 165.

[29]. Id.

[30]. Id. at 173.

[31]. Id.

[32]. Id. at 172-73.

[33]. Cook & Ludwig, supra note 15, at 104.

[34]. Kleck & Gertz, Armed Resistance to Crime, supra note 1, at tbl.4.

[35]. Cook & Ludwig, supra note 15, at 104.

[36]. Tom W. Smith & Robert J. Smith, Changes in Firearm Ownership Among Women, 1980-1994, 86 J. Crim L. & Criminology 133, 143 (1995).

[37]. Based on the analysis of the 1976-1990 Supplementary Homicide Reports of the Federal Bureau of Investigation (FBI) reported in Abraham N. Tennenbaum, Justifiable Homicides by Civilians in the U.S., 1976-1990: An Exploratory Analysis, tbl.11 (1993) (unpublished Ph.D. dissertation, University of Maryland) (on file with author). Unpublished figures from special data run by the FBI for 1991-1995 for the author show similar results.

[38]. Cook & Ludwig, supra note 15, at 91.

[39]. Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 162.

[40]. Philip J. Cook et al., The Gun Debate's New Mythical Number: How Many Defensive Uses Per Year?, 16 J. Pol'y Analysis & Mgmt. 463, 468 (1997).

[41]. Cook & Ludwig, supra note 15. Based on a very small sample, they found that 16% of DGUs involved trespassing and 47% involved no threat, attack, or injury. Id. at 109. K-G seem to indicate that fewer DGUs involve non-NCVS crimes (3.7% only trespassing and 9.5% "other crimes"). Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 185-86. They also state that "a large share of the incidents covered by our survey are probably outside the scope of incidents that realistically are likely to be reported to the NCVS ...." Id. at 167-68.

[42]. K-G suggest that telescoping might lead to an overestimate of 21%. Kleck & Gertz, Armed Resistance to Crime, supra note 1, at 171. While no definitive figure exists, work on the NCVS and other surveys suggest the rate is more likely to be around 50%. See Ronald Andersen et al., supra note 9, at 19-20, 32-37, 42-50, 52-74; David Cantor, Substantive Implications of Longitudinal Design Features: The National Crime Survey As a Case Study, in Panel Surveys (Daniel Kaspryz et al. eds. 1989); Neter & Waksberg, supra note 9.

[43]. Or to an even lower figure if annual estimates based on the five- year recall questions are used.

[44]. For example, K-G acknowledge that up to 10% of DGUs might really represent instances of "mutual combat." Kleck & Gertz, The Illegitimacy of One- Sided Speculation, supra note 1, at 1450. Also, if a positive social desirability bias exists (as H argues) the K-G number would come down much further. See Hemenway, supra note 2, at 1430-31. But if K-G are correct about there being a negative social desirability bias, then the above adjusted estimates would already be too low and both their and the NCVS estimates would rise.

[45]. The NCVS is in the public domain already.