“Evidence-based” appears on essentially every treatment website in America. It is unregulated, uncheckable as written, and by itself tells you nothing.
Here is how to make it mean something.
Question one: evidence for which outcome?
This is the question that does the most work, and almost nobody asks it.
A programme can be genuinely evidence-based for completion rates and have no evidence at all for what happens a year later. It can improve quality of life without changing drinking. It can reduce drinks per drinking day while the number of drinking days is unchanged.
All of those are real, measurable outcomes. They are not the same thing, and a facility gets to choose which one to talk about.
There is an entire research literature on this problem. Falk and colleagues (2010) examined whether “percentage of subjects with no heavy drinking days” works as a meaningful measure of whether a treatment succeeded — because the field itself has been arguing about what counts as success. When researchers need papers to settle what a good outcome is, a marketing page claiming “proven results” is not describing anything in particular.
Ask: what specific outcome, measured how, at what point after treatment ends?
Question two: compared to what?
“Our clients improve significantly” is almost meaningless, because people entering treatment tend to improve. They are usually at their worst on the day they arrive.
The useful comparison is against another treatment, or a control group — not against the person’s own worst week.
Project MATCH is instructive here. It compared three well-designed treatments head to head and found little difference in outcome by type of treatment — while all three produced significant, sustained improvement over the following year.
So: three different approaches, all with genuine improvement, none clearly better. Any one of them could truthfully advertise “significant improvement.” None could honestly claim superiority over the others.
Ask: compared to what, and did anyone check?
Question three: how large is the effect?
“Statistically significant” means unlikely to be chance. It says nothing about size.
The most honest way to express a treatment effect is number needed to treat — how many people you would have to treat for one additional person to benefit. Jonas and colleagues (2014), pooling 122 trials and 22,803 people, reported:
- Acamprosate: NNT 12 to prevent a return to any drinking.
- Oral naltrexone: NNT 20 for any drinking; NNT 12 for heavy drinking.
An NNT of 12 means eleven of twelve people show no detectable benefit on that measure. That is a real, worthwhile effect for a cheap and safe medication — and it is nothing like the impression left by “clinically proven.”
Ask: how many people does this help, out of how many treated?
Question four: who was studied, and were they like this person?
A result in stably-housed volunteers with no psychiatric comorbidity may not transfer to someone with a decade of use and untreated depression.
Project MATCH found several client characteristics predicted how people fared regardless of which treatment they received: motivational readiness, whether their social network supported drinking, degree of alcohol involvement, gender, psychiatric severity. Those mattered more than the choice of approach.
Ask: does the evidence come from people in a comparable situation?
Question five: who did the measuring, and who did they lose?
The quietest problem in outcome claims is not dishonesty. It is who is missing from the count.
If a facility follows up its graduates and reports that 70% are doing well, the question is what proportion they managed to reach. People who are doing badly are systematically harder to contact — phones change, they move, they avoid the call. A study that reaches 40% of its sample and reports on those people is describing the reachable, not the treated.
This is exactly why serious trials report their follow-up rates. Project MATCH interviewed over 90% of living participants at the one-year assessment, and says so. That figure is not incidental detail; it is what makes the rest of the numbers meaningful.
Ask: what proportion of people who started were you able to follow up, and are the ones you couldn’t reach counted?
A facility that has never measured this is not lying to you. It simply does not know, and neither will you.
The one thing “evidence-based” usually does tell you
Despite all of the above, the phrase is not entirely empty. A facility using it has at least positioned itself as accountable to research, which makes the four questions above legitimate to ask.
And the answers are diagnostic in themselves. A programme that can tell you which outcome, over what period, compared to what, is telling the truth. A programme that responds to those questions with testimonials and a photograph of the grounds has answered you.
What we do with this
Working out whether a specific facility suits a specific person is most of what placement and case management is. Not because the brochures are all lying, but because they are all optimised, and comparing optimised documents is not the same as comparing programmes.
The questions above are the ones we ask on your behalf. They are also yours to ask, and a good facility will not mind at all.
References
- Falk D, Wang XQ, Liu L, et al. (2010). Percentage of Subjects With No Heavy Drinking Days: Evaluation as an Efficacy Endpoint. Alcoholism: Clinical and Experimental Research. 10.1111/j.1530-0277.2010.01290.x
- Jonas DE, Amick HR, Feltner C, et al. (2014). Pharmacotherapy for adults with alcohol use disorders in outpatient settings. JAMA, 311, 1889-1900. 10.1001/jama.2014.3628
- Project MATCH Research Group (1997). Matching Alcoholism Treatments to Client Heterogeneity. Journal of Studies on Alcohol, 58, 7-29. 10.15288/jsa.1997.58.7
Read more about Rehab Placement.
