by: Clarke Pollard
Alzheimer’s Support Network
“Significant” is a term in statistics that means the results were likely not due to chance. When we hear “significant” in a research study, we need to translate how we normally use the word “significant” to its use in research studies.
Let’s say when you come out of the grocery store and you go to put your bags in your trunk, you notice a little scratch on your back bumper. You have to lean down and look really close. But, there it is. Looks like some white paint rubbed off on your black bumper.
You would never march back into the store and demand to talk with the manager because your car had “significant” damage. You wouldn’t say you had “significant” evidence that a white car backed into you. But this is how the word “significant” is used in the world of statistics and research. Is there evidence that someone in a white car backed into your black car? The little white mark on your bumper is evidence that someone slightly bumped into you when backing up. It is more likely that happened rather than the white mark appearing by chance. When we hear “significant effect,” we think, I’m going to come out to my car and I’m going to see a huge dent in my bumper. But in statistics and in drug trials, “significant effect” means the result occurred more than by chance. It means there is a scratch on your bumper.
Does statistics have a name for your bumper having a 3-inch dent that you could see 20 feet away from your car? Now we are getting closer to the idea of “clinically meaningful.” When there is a clinically meaningful effect, this means it is noticeable in the real world. It has profound results. For the Statistically Significant damage (the scratch on my bumper), I had to get right up to my car. And even then, I wasn’t sure there was actually any damage. I had to lean down and look close to see the white scratch mark. That is a statistically significant effect. Clinically Meaningful damage is the 3-inch dent that I can see while walking towards my car. The data presented to the FDA showed Lecanemab had a benefit on cognition that was statistically significant. If you lean down and look closely, it is likely this slowing of decline didn’t occur by chance. The data, however, did not show a clinically meaningful effect. How do we judge that?
When you do a drug trial, you need to pick beforehand a measurement tool that can tell you if your drug worked or not. Researchers will pick a “primary end point.” This is the main result that researchers pick to determine if the drug works. The evaluation tool that was chosen for Lecanemab was the Clinical Dementia Rating – Sum of Boxes. On this primary end point, Lecanemab (Leqembi) showed a 0.45 difference versus placebo (Van Dyck et al., 2023). For the Clinical Dementia Rating – Sum of Boxes tool, 0.45 is statistically significant (William et al., 2012). That is, if you bend down and look closely, you can see the difference. The problem is the Lecanemab results were not clinically meaningful (a difference you can see plainly and a difference that makes a real-world impact). For the Clinical Dementia Rating – Sum of Boxes scale, clinically meaningful results for people with mild cognitive impairment is 1.0. And for people with mild Alzheimer’s the meaningful threshold is 2.0 (Isaacson, 2021). Lecanemab showed a 0.45 difference. Does Lecanemab slow cognitive decline? After all, if you are going to go every two weeks to get an infusion and risk brain bleeds and brain swelling, ultimately, you want to know if this drug will keep you sharper for longer. Does it do that?
The answer: yes, if you bend down and look very closely (0.45 = statistically significant). But the results presented do not show clinically meaningful results (0.45 is less than half the meaningful threshold of 1.0 for MCI and less than 1/4 of the meaningful threshold of 2.0 for mild Alzheimer’s disease). If you have been diagnosed with Alzheimer’s, for this drug to slow down your cognitive decline in a meaningful way, you should look for a change of 2.0 or greater on the Clinical Dementia Rating – Sum of Boxes over a period of a year. Lecanemab showed a 0.45 change after a year and a half. In newspaper stories about Lecanemab, you will see claims that the drug slowed cognitive decline by 27% (Moore, 2022; Robbins and Belluck, 2022). You should know that the 27% decline is equal to 0.45 points on the Clinical Dementia Rating – Sum of Boxes. And this point is made even in some of the same newspaper stories. For example, the New York Times included the 27% figure but went on to say: “Some experts said the drug’s ability to slow cognitive decline — by 0.45 on an 18-point scale — was modest at best and might not be a difference that patients in the mild early stages of the disease would notice” (Robbins and Belluck, 2022).
The same point of caution has been expressed by the editorial staff of highly respected journals, such as The Lancet which warned the 0.45 effect might not be clinically meaningful (The Lancet, 2022). The journal Nature also published an article saying: “It’s unclear what impact that 27% effect will have on the lives of people with Alzheimer’s — or whether the effects will persist after 18 months” (Reardon, 2023). The real percentage that matters is: did Lecanemab hit 100% of the meaningful change threshold? The answer there is no. Lecanemab reached 22.5% of the meaningful change threshold for mild Alzheimer’s. In other words, for Lecanemab to have a meaningful effect on cognitive decline, future versions of the drug need to improve by 77.5% Is the current version of Lecanemab worth it? You need to decide for yourself.
