Integrity: the practices that produced the replication crisis

Psychology’s replication problem was not produced by fraud. It was produced by a set of ordinary practices that each seemed defensible in isolation, were taught implicitly, and were rewarded by journals. Knowing their names is the first defence, because they are much easier to spot in a manuscript than in your own analysis.

HARKing

Hypothesising After the Results are Known — presenting a post hoc explanation as an a priori prediction. It converts an exploratory finding into a confirmatory one on paper, and it is invisible to the reader.

p-hacking

Trying analytic choices until something crosses .05: adding participants and re-testing, dropping outliers by different rules, choosing among several outcome measures, adding or removing covariates. Each decision is defensible; the sequence is not.

Garden of forking paths

Gelman’s point that p-hacking need not be deliberate. A researcher making reasonable data-dependent choices, once, without trying alternatives, still inflates the false positive rate — because the choices were contingent on the data.

Outcome switching

Reporting a secondary outcome as primary because the pre-specified primary was null. Comparison of registered protocols with published papers finds this is common.

Publication bias / the file drawer

Null results are less likely to be submitted and less likely to be accepted, so the published literature over-estimates effects. Funnel plots and trim-and-fill detect it imperfectly.

Salami slicing

Splitting one study across several papers to increase count. It fragments the evidence and can produce double-counting in meta-analyses.

Citation without reading

Repeating a characterisation of a source from a secondary citation. Errors propagate through literatures for decades this way. Read what you cite.

What actually protects against this

1. 1Preregistration. State the hypotheses, the primary outcome, the sample size and the analysis plan before you look at the data — OSF, AsPredicted or ClinicalTrials.gov. Deviations are permitted; they must be declared.
2. 2Registered Reports. The design is peer-reviewed and accepted before data collection, so publication does not depend on the result. The best structural fix available.
3. 3Distinguish confirmatory from exploratory, explicitly, in the paper. Exploratory analysis is valuable and honest; it is only a problem when disguised.
4. 4Report everything you measured, or say where the rest is. "All measures, manipulations and exclusions are reported" — the 21-word solution.
5. 5Share data, code and materials where consent and confidentiality permit, and say why where they do not.
6. 6Power the study, or say plainly that it is underpowered and treat the result as provisional.

Authorship, and the arguments about it

The ICMJE criteria require all four of: substantial contribution to conception or data; drafting or critically revising; final approval; and accountability for the work. Providing funding, supplying data, or heading the department does not qualify. Gift authorship and ghost authorship are both misconduct, and both are common.

Conflicts of interest

In psychotherapy research the most consequential conflict is usually not financial but allegiance: the investigator developed or is committed to the treatment being tested. Allegiance effects are comparable in size to the treatment differences under study. Declare it, and where possible design against it — independent assessors, comparator experts with their own allegiance, or an independent replication.