Data and Code for: Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Principal Investigator(s):  View help for Principal Investigator(s) Sebastian Kranz, Ulm University; Peter Pütz, Bielefeld University

Version:  View help for Version V1

Name File Type Size Last Modified
  Data 03/21/2022 06:27:AM
  Docker 05/31/2022 01:06:AM
  R-packages 05/31/2022 01:09:AM
  Results 03/21/2022 06:15:AM
  Scripts 03/21/2022 06:15:AM
  Tex_templates 03/21/2022 06:15:AM
LICENSE.txt text/plain 18.2 KB 05/30/2022 09:03:PM
README.md text/x-web-markdown 12.7 KB 05/30/2022 08:31:PM
README.pdf application/pdf 142.5 KB 05/30/2022 08:38:PM
mm_replication.Rproj text/plain 218 bytes 03/21/2022 02:14:AM

Project Citation: 

Kranz, Sebastian, and Pütz, Peter. Data and Code for: Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment. Nashville, TN: American Economic Association [publisher], 2022. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2022-08-23. https://doi.org/10.3886/E159221V1

Project Description

Summary:  View help for Summary Brodeur, Cook, and Heyes (2020) study hypothesis tests from economic articles and fi nd evidence for p-hacking and publication bias, in particular for IV and DID studies. When adjusting for rounding errors (introducing a novel method), statistical evidence for p-hacking from randomization tests and caliper tests at the 5% signi ficance threshold vanishes for DID studies but remains for IV studies. Results at the 1% and 10% significance thresholds remain largely similar. In addition, Brodeur, Cook, and Heyes derive latent distributions of z-statistics absent publication bias using two di fferent approaches. We establish for each approach a result that challenges its applicability.


Scope of Project

Subject Terms:  View help for Subject Terms p-hacking; publication bias ; research methods ; causal inference; rounding; replication
JEL Classification:  View help for JEL Classification
      A14 Sociology of Economics
      B41 Economic Methodology
      C12 Hypothesis Testing: General
      C40 Econometric and Statistical Methods: Special Topics: General
      C52 Model Evaluation, Validation, and Selection
Time Period(s):  View help for Time Period(s) 1/1/2015 – 12/31/2018 (2015 and 2018)


Related Publications

Download this project

Published Versions

V1 [2022-08-23]

Export Metadata

Dublin Core

DDI 2.5

Report a Problem

Found a serious problem with the data, such as disclosure risk or copyrighted content? Let us know.

This material is distributed exactly as received from the data depositor. As of April 2026, depositors are required to submit study materials in accessible formats. ICPSR has not reviewed, checked, or processed this material. For additional information about the study, please contact the investigator(s) directly. If you have questions about the accessibility of materials distributed by ICPSR or require further assistance, please visit ICPSR's Accessibility Center.