ECIN Replication Package for "Selective reporting of placebo tests in top economics journals"

Principal Investigator(s):  View help for Principal Investigator(s) Anna Dreber, Stockholm School of Economics; Magnus Johannesson, Stockholm School of Economics; Yifan Yang, Stockholm School of Economics

Version:  View help for Version V3

Name File Type Size Last Modified
  Placebo_Replication_File 02/27/2024 12:26:PM

Project Citation: 

Dreber, Anna, Johannesson, Magnus, and Yang, Yifan. ECIN Replication Package for “Selective reporting of placebo tests in top economics journals.” Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2024-02-27. https://doi.org/10.3886/E197963V3

Project Description

Summary:  View help for Summary This is data and code accompanying the article "Selective reporting of placebo tests in top economics journals". The abstract for that paper is the following: Placebo tests provide incentives to underreport statistically significant tests, a form of reversed p-hacking. We test for such underreporting in 11 top economics journals between 2009-2021 based on a pre-registered analysis plan. If the null hypothesis is true in all tests, 2.5% of them should be significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% ((95% CI [2.2, 4.0]).


Funding Sources:  View help for Funding Sources Knut and Alice Wallenberg Foundation; Marianne and Marcus Wallenberg Foundation; Riksbankens Jubileumsfond; Jan Wallander and Tom Hedelius Foundation

Scope of Project

Subject Terms:  View help for Subject Terms placebo tests; underreporting; economics; metascience
JEL Classification:  View help for JEL Classification
      C10 Econometric and Statistical Methods and Methodology: General
      C12 Hypothesis Testing: General
      C18 Methodological Issues: General
Manuscript Number:  View help for Manuscript Number ECIN-Nov-2023-0499
Time Period(s):  View help for Time Period(s) 1/1/2009 – 12/31/2021
Universe:  View help for Universe
Papers published in 12 top journals in economics 2009-2021 that reported at least one placebo test.
Data Type(s):  View help for Data Type(s) observational data
Collection Notes:  View help for Collection Notes Please see the paper for more information.

Methodology

Sampling:  View help for Sampling We collected data on placebo tests for papers published in 12 top journals in economics that reported at least one placebo test One of these 12 journals (Economic Journal) was included in a pilot study conducted prior to posting a pre-analysis plan, and papers from this journal are therefore not included in any hypotheses tests.
Data Source:  View help for Data Source 12 journals: American Economic Journal: Applied Economics; American Economic Journal: Economic Policy; American Economic Review7; Econometrica; Economic Journal; Journal of Development Economics; Journal of Labor Economics; Journal of Political Economy; Journal of the European Economic Association; Review of Economics and Statistics; Review of Economic Studies; and Quarterly Journal of Economics.
Collection Mode(s):  View help for Collection Mode(s) other
Unit(s) of Observation:  View help for Unit(s) of Observation Paper
Geographic Unit:  View help for Geographic Unit NA

Related Publications

Download this project

Published Versions

V3 [2024-02-27]

V2 [2024-02-23]

V1 [2024-01-26]

Export Metadata

Dublin Core

DDI 2.5

Report a Problem

Found a serious problem with the data, such as disclosure risk or copyrighted content? Let us know.

This material is distributed exactly as it arrived from the data depositor. ICPSR has not checked or processed this material. Users should consult the investigator(s) if further information is desired.