Workplace Wellness Programs - Effective or Not? My Review of the Fleming Study
My review of the study that’s been making the news. Spoiler alert: The results are not as black-and-white as they seem.
I’m sure by now most of you have come across articles and discussions about the recent study by Dr. William Fleming analyzing the efficacy (or lack thereof) of workplace wellness benefits for improving mental health outcomes. I’ve finally had the chance to review the study with a critical eye and want to give my two cents, for what they’re worth.
I’m not going to reiterate the results, as you can read those for yourself, and most of the strengths of this study are already presented. For example, it’s a national study conducted with thousands of participants, giving it ample statistical power. The statistical methods are also strong. Overall, I think this study gives us a lot of important things to consider.
There are also some major limitations. Several of these limitations and biases are discussed directly by Fleming. Some of those limitations are discussed in the New York Times article by Dr. Larissa Bartlett. Some are not. For the sake of space, I’ll combine. These are not in any particular order.
The study is conducted only in UK employees and organizations. You probably don’t need me to tell you that mental health care access and affordability differ greatly between countries. Being from the US and with the majority of workplace wellness programs being based in the US, I immediately think about the differences between the two countries that could contribute to disparate results.
Without getting too far into the details, the US is greatly lacking in healthcare affordability and accessibility of compared to other developed countries. The majority of Americans rely on their employer for insurance. Even so, they are reluctant to utilize healthcare services due to high deductibles and out-of-pocket costs. As such, Americans tend to only seek care when their symptoms have become severe. That’s not to say the UK’s NHS doesn’t also have room to improve; but I think their systems are different enough that we can’t assume that the results from a UK study will replicate 1:1 with the US.
The binary yes/no method of identifying “people who participated” leaves a lot of ambiguity and range on the table. In this study, participants were asked whether they participated in a program in the past 12 months. The question is subjective and could lump together people who have used a program one time with people who are daily users.
How many of you have downloaded an app, used it once for 10 minutes, and then left it to silently die as a permanent fixture of your phone’s wallpaper, never to be used again? Some of us might say we participated, others may not. But if I were asked if I had used the app regularly, with a clear definition of what “regular” means, my answers would be very different. Not understanding these data in a more objective and nuanced way presents a major limitation to interpreting the results.
The study is cross-sectional, meaning it represents a single point in time. It doesn’t take into account how people may have improved or declined in their mental health over time. It also doesn’t account for the self-selection effects as well as ceiling effects that may have taken place.
People with the lowest well-being are in greatest need of help, but it is often the hardest for them to initiate and follow through on programs. If they do, though, they’re have the most room for improvement. The lack of nuanced data in this study makes this impossible to parse out.
People with the greatest well-being may be the types of people to invest heavily in their health and they may opt into these workplace wellness programs at a higher rate than others. However, they don’t have much room to improve, because they already have a high level of well-being. These are called “ceiling effects”.
Fleming did try to account for individual differences in health and well-being in the study by adding workplace stress into the model. Using the question “During the last 12 months have you felt unwell as a result of work‐related stress?”, with potential answers being “no”, “yes, somewhat”, and “yes, definitely”, Fleming examined whether results differed by response. Results remained for the people in the “no” and “yes, somewhat” groups, but it appears that results were more supportive of workplace wellness programs for people that had reported more workplace stress (the group that answered “yes, definitely”.
Specifically, Fleming says, “results for ‘yes, definitely’ are less clear, and while credible intervals do overlap, participant predictions appear slightly higher, suggesting that those definitely suffering from work–stress who participate have slightly higher mental well-being. This may suggest some positive for stressed workers, but it may also indicate further selection effects within this group.” Overall, this indicates that there is likely more nuance involved in the results than can be readily examined in this study.
Another important point on this topic is that Fleming was only able to assess this with work-related stress, specifically. We all know that stress can come from various areas of our lives, and this item doesn’t reflect overall well-being. It makes me wonder what we could learn about how workplace wellness programs affect high-stress individuals with more data.
This study lumps together various types of workplace wellness programs to examine overall effects. I understand why Fleming did it from a theoretical perspective, but in the end it combines very disparate types of programs which can muddy results—imagine financial literacy courses being combined with mindfulness.
These data are from 2018. Girlies, we were in our pre-pandemic, pre-work from home era. Yes, it’s only 6 years ago. But in the tech world, that’s a lifetime. We’ve learned a lot about workplace well-being over the past half-decade, and a lot has changed. The number of new programs that have come to life since 2020—including ones that certainly understood the assignment—are not included here.
I want to be clear that I think Dr. Fleming did a great job with this study. I honestly think he did the best he possibly could with secondary data, with items and data collection methods out of his control. The reason I feel compelled to write this is to highlight the limitations (most of which Fleming said himself in the scientific article) and nuances that are not portrayed in the discussions that I’ve seen.
TL;DR: The study—while insightful and important—has some major limitations that should give us pause about generalizing the results. We need a great deal more information before we can outright dismiss workplace wellness programs as being ineffective. A nuanced perspective and a lot more data are needed to understand what is effective and for whom.
👋 I'm Lydia Roos, health psychologist and Founder of EvolveWell Research Partners
📌 EvolveWell offers specializes in conducting research for mental and physical health, wellness, and fitness companies on a project, fractional, or advisory basis, including companies that offer employer-sponsored mental health benefits. [evolvewellresearch.com]