How to read a psychotherapy trial

1. 1What was the comparator? This changes the effect size more than the treatment does. Waitlist controls inflate effects substantially — patients waiting for treatment do worse than patients receiving nothing, because expectancy is suspended. Treatment-as-usual is better but varies wildly. Another bona fide therapy, delivered by people who believe in it, is the demanding test.
2. 2Who ran it? Researcher allegiance is among the strongest predictors of outcome in comparative trials, with effects comparable in size to the treatment differences being studied. It operates through therapist selection, training quality, protocol fidelity and outcome measure choice, not through dishonesty.
3. 3What was blinded? Participants and therapists cannot be blinded in psychotherapy. Only the assessor can be, and only for observer-rated outcomes. Self-report outcomes in an unblinded trial carry a known and unquantifiable expectancy component.
4. 4Who was excluded? Comorbidity, suicidality, substance use and low literacy are common exclusions, and they describe a large share of clinic populations.
5. 5What was the attrition, and how was it analysed? Completer analyses inflate effects. Intention-to-treat with sensible imputation is the standard.
6. 6Was the change clinically significant? Reliable change tells you the movement exceeded measurement error. Clinical significance tells you the person crossed from the clinical into the functional range. A statistically significant three-point change on a symptom scale may be neither.

Effect sizes, honestly

Meta-analysis: what to check

• Heterogeneity — a pooled estimate across wildly different studies may describe none of them
• Publication bias — funnel plots, trim-and-fill, and whether the authors looked
• Study quality weighting, and whether excluding weak studies changes the conclusion
• Whether the included comparisons are like for like
• Who conducted it, and what they had previously published