全球智库信息集成服务系统

• Simulation results reveal that many commonly used modeling approaches in gun policy research have quite poor type 1 error rates.
• Several models have type 1 error rates ten times greater than the nominal α = 0.05 that was intended.
• Huber and cluster adjustments often do not fix these problems, and Huber adjustments can sometimes make them worse.
• The models also had surprisingly low statistical power to detect an effect-sized equivalent to a change of 1,000 deaths per year if a law were implemented nationally.
• Most models could correctly reject the null hypothesis only about 10 percent of the time with this true effect. With power this low, a large fraction of effects that are statistically significant will be found to be in the opposite direction as the true effect, and all significant effects will greatly exaggerate the magnitude of the true effect.
• One model was identified as having the best performance across all assessed criteria. This model is a negative binomial model of firearm deaths that includes time-fixed effects, an autoregressive effect, and change coding for the law effect. The preferred specification includes no state-fixed effects or standard error adjustment.

The RAND Corporation launched its Gun Policy in America initiative with the goal of creating objective, factual resources for policymakers and the public on the effects of gun laws. As a part of this project, RAND researchers conducted a systematic literature review and evaluation of scientific studies on the effects of 13 classes of policies. One of the findings of the review was that the effects of policies estimated in the literature appeared to be sensitive to the specific statistical methods that were employed. This suggests the importance of identifying the most-appropriate statistical methods to use on these data.

In this report, the authors use simulations to assess the performance of a wide range of statistical models commonly used in the gun policy literature to estimate the effects of state-level gun policies on firearm deaths. The study aimed to identify the most-appropriate statistical modeling and analysis methods for estimating the effect of these policies on firearm deaths, which may help in the evaluation of whether estimates from prior research can be considered to be accurate. The results suggest substantial statistical problems with many of the methods used. The authors also identify the best method among those assessed.

This report should be of interest to researchers familiar with statistical methods for estimating causal effects in longitudinal time series data, those who are trying to understand the effects of gun policies as revealed in the existing literature, or those who are planning new studies that use statistical models to investigate these effects.

• Chapter One

  Introduction

• Chapter Two

  Methods

• Chapter Three

  Results

• Chapter Four

  Discussion

• Appendix A

  Technical Description of Evaluated Models

• Appendix B

  Standard Error Correction Factors