Home » Posts tagged 'Wellness ROI Calculator'
Tag Archives: Wellness ROI Calculator
We are getting very frustrated with the failure of wellness advocates to show even the slightest net savings using a legitimate methodology. Therefore we are offering a million-dollar reward for the first person who does. To win this reward, there are specific rules that must be followed regarding data sources and the selection of panelist judges, all listed below.
Recently a group calling themselves the Global Wellness Institute Roundtable put out a press release and report criticizing us for “mud-slinging on ROI.” We are not familiar with this group. Their headliner seems to be a Dr. Michael Roizen, head of the Cleveland Clinic’s much-vilified wellness program. If that name sounds familiar, it’s because he used to work with Dr. Oz, though to Dr. Roizen’s credit, he avoided the Congressional investigation of Dr. Oz.
The report says we “impose a standard of evidence that doesn’t exist for any other workplace investment.” Um, like it needs to break even? Wouldn’t a company go bankrupt pretty quickly if it didn’t insist that its investments should break even?
Also, there are three very specific reasons why wellness needs a high “standard of evidence.” If Dr. Roizen doesn’t understand these reasons, he can get a smart person to explain them to him.
We get a little frustrated when we prove something, and then members of the wellness industry dress themselves up with words like “Global” and “Institute” and “Roundtable” and then say things like: “critics are misusing ROI science to castigate…workplace health efforts.” Then they cite articles that inadvertently undermine their own arguments and support the critics.
They also say things like: “93% of the workplace wellness return in the first year is in productivity gains, not reduced cost.” This is squirrelly even by the lax standards of wellness math. No company can measure its productivity gains with that precision. Still — assuming you exclude time wasted in filling out forms, being screened, and getting unnecessary checkups — maybe they’re right. After all, nothing focuses the mind on work-related issues like being told you’re sick.
And yet casual observers assume there are two sides to this “debate.” It doesn’t help that journalists need to print opposing quotes. However, pry, poke, and prod” wellness loses money, period–unless you count the money forfeited by employees who don’t participate, or don’t lose enough weight to earn their payment. Those payments are not counted for the purposes of this reward since they are transfers, not savings.
Leaving out employee forfeitures, the country as a whole has not even remotely approached breaking even on wellness spending vs. claims costs.
But don’t take our word for this. We are offering a million-dollar reward for anyone who can show that it is more likely than not that “pry, poke, and prod” wellness breaks even through healthcare claims savings.
Not” Show a 2-to-1 ROI” or “Defend the famous 3.27-to-1 ROI.” Just: “Show a breakeven.” Not “wellness is a success.” Just “wellness is not a epic fail.”
Specifically, you just need to show, using publicly available databases (not private “case studies” or vendor reports), that:
- it is mathematically possible that the country’s employers can reduce their medical claims costs enough to cover the wellness industry’s $8 billion in annual billings by enough to offset internal costs and consulting fees (you can estimate those); and
- during this millennium the wellness industry has reduced costs (by avoiding wellness-sensitive medical events, which is the methodology HERO and us agree on) by enough to break even according to the first calculation;
- The so-called “best programs” in the country — Koop Award winners the last 3 years — are, if not exemplary, then at least good performers that earned the savings they claimed to save, by significantly reducing risk factors.
Here are the rules. This is a binding legal contract. We can’t offer something like this and then say we had our fingers crossed.
- You need to start out with a lie detector test, to be performed by the Boston police, as I will. The questions to be asked are: “Are you telling the truth?” and “Is the opposing party either deliberately lying, and/or has no clue how to measure outcomes?” Either side may submit either or both sides’ results, as they prefer.
- You must provide your view of the C. Everett Koop Award to Wellsteps and Boise (600 words, 6 hyperlink maximum), and rebut the other view (600 words, 6 hyperlink maximum). We will also both be asked: “Did the Wellsteps program reduce costs in Boise by roughly a third, as they claimed?” in the lie detector test.
- In order to facilitate your quest, you can use include the wellness industry’s own HERO Outcomes Guidelines, which represents the “consensus” (their word) of 39 “subject matter experts” who support wellness. (None who oppose wellness were invited to participate.) US Census Bureau information, Kaiser Family Foundation, and the AHRQ’s HCUP database can also be used. Other databases will be allowed in at the discretion of the panelist judges if they deem their probative value to be very strong.
- You may cite/quote any peer-reviewed article in rebuttal of the opposing view; however articles in support of the position in question must be sourced from one of the ten medical journals with the highest reported “impact factors,” and have been published within the last five years;
- Once an article is brought into the discussion, the opposing party may also cite it in cross-examination.
- We give you a lien on $1,000,000 as soon as you escrow $100,000 to cover the costs of the program (honoraria for panelists, venue etc). The loser pays this (meaning that if you win, you get it back). If the costs are less than $100,000, the winner keeps the difference.
- We each pick four panelists from Peter Grant’s “A-List” of the leading 260 health economists and policy experts (this is an invitation-only email list in which health policy and health economics concerns are addressed and debated) that are unaffiliated with the wellness industry or with Quizzify . Each party can veto two of the other’s picks. Together, the remaining four pick a 5th.
- Each side submits up to 2000 words and 5 graphs, supported by no more than 20 links;
- The material linked must predate the claim for the award by 6 months, in order to discourage either side from creating linked material specifically for this contest.
- Each party may separately cite previous invalidating mistakes made by the other party that might speak to the credibility of the other party.
- Either side may cite an unlimited number of “declarations against interest,” made within the last 4 years–meaning comments made by the other party so prejudicial to their own position that the other party would only have said them if they were true. Example: if I said, “Wellness definitely saves money” (except when I said it as an April Fool’s gag), you could cite that. There is no word limit on these.
- Each party can then rebut the other party with up to 2000 words and 5 graphs, and 20 links.
- The parties will be convened, in Boston (or another agreed-upon city, if the other party is willing to pay my travel expenses), for a two-hour finalist presentation in which the panelists (whose travel expenses and professional charges are paid for out of the entry fee) can ask questions of either party, and both parties can cross-examine the other for up to 40 minutes, with followup questions and no limitations on subject matter. Each party can make a 10-minute opening and 10-minute closing statement. Up to 20 slides are allowed.
- During their 40 minutes, each party that has not achieved Certification in Critical Outcomes Report Analysis by the Validation Institute must explain why they haven’t, and each party that has not signed the Employee Health and Wellness Code of Conduct must explain why they haven’t.
We invite the wellness industry leaders — the “Global Wellness Initiative Roundtable,” the Koop Award Committee, and the Business Roundtable (BRT) and of course the Health Enhancement Research Organization — to collect their million dollars. Or shut up.
Our most widely read blog post offers a free Wellness ROI calculator. It’s been getting hit after hit in the weeks we’ve had it up. Further, no one has found any mistakes in it, which is unique in the ROI calculator field. (The Brand X ROI calculator — we won’t mention any names but it rhymes with Wellsteps — is nothing but a mistake, since if you zero out inflation, their model always shows $1359/employee in savings by 2020.)
If you missed it (meaning ours), this is a good opportunity to download it and use it yourselves.
Here is the link directly to the calculator. You still need to scroll down, though, to the June 30 posting.
We are pleased to present a free wellness ROI estimation model, as we promised about 3 months ago. This is the only tool of its kind in the industry. (Wellsteps has one, but let’s just say the good news is that NASA employees don’t have to worry about job security, because these people aren’t rocket scientists. If you zero out inflation, no matter what other variables you enter, the Wellsteps model always shows savings of $1359.)
You can also use this to compare two wellness programs, to determine whether your vendor is lying (they are — and we are happy to help you get your money back from them), and to pressure-test Quizzify.
The HERO Report concludes that wellness loses money. We agree. We also think it loses much more money than they will admit to, but the news here is not about us. The news is that more than 3 dozen self-described experts and industry leaders representing more than 2 dozen companies have reached consensus that their industry loses money.
Together, the HERO findings — and our broad consensus with those findings — have serious Affordable Care Act policy implications. The entire basis for the ACA “Safeway Amendment” allowing large fines for (among other things) failure to lose weight is that the cost savings from skinnier employees merits invading their privacy, dignity and automony through medicalizing the workplace (“companies playing doctor” as some have called it). Senate committee hearings, proposed new legislation, and EEOC lawsuits around this provision have all been based on the assumption that wellness saves money. The Senate committee never even lobbed a softball question about that assumption, and even the more hostile witnesses didn’t challenge it.
Recently there was even an eyeball-to-eyeball encounter between the Business Roundtable’s (BRT) Gary Loveman and President Obama. Even though his company (Caesar’s) went bankrupt while embracing wellness as essential to their profitability, Mr. Loveman argued that corporations should be allowed to fine workers who don’t lose weight because the benefit to corporate bottom lines would trump both privacy concerns and the substantial health hazards of these programs.
Apparently, though, Mr. Loveman’s company went bankrupt slightly faster because of wellness. Yes, along with employees, employers would be better off without forced (highly penalized or incentivized) workplace medicalization. If you fire your wellness vendor, everyone benefits.
Everyone, that is, except the wellness industry denizens who make their money off this. That’s why we think HERO spoke the truth unintentionally. Very few people (I was one of them, having switched sides in 2007 when I saw that data failed to support wellness/disease management) willingly undermine their own incomes for integrity’s sake. So this posting will proceed on the basis that is was a gaffe on their part.
Curiously, this is the second time in recent months wellness industry leaders have accidentally admitted wellness loses money, and the third time they’ve accidentally told the truth and had to walk it back.
Equally curiously, wellness economics information disseminates very slowly if at all — testament in large part to the absolutely brilliant and flawlessly executed strategy by the Wellness Ignorati of ensuring that facts get ignored (hence their name). So even as the vendors are admitting that wellness loses money, benefits consultants and HR executives have once again pushed participation incentives/penalties to new highs, a whopping $693/employee/year, according to a new report.
As for the figures themselves, we are also attaching a spreadsheet so that you—as an employer—can figure this out on your own in your own population, rather than just take HERO’s word for it that wellness loses money.
The costs, according to the HERO report’s own screenshots
First, review the screenshot from the first installment, showing the costs of wellness. The list of cost elements is fairly exhaustive –down to the level of a space allocation for a health fair — though the Committee conveniently left out consulting fees. No surprise there, given that Mercer consultants sit on the committee.
Then, compare the list of costs in that screenshot to costs in this second screenshot, from Page 15 of the HERO Report. That comparison won’t take long because only one program cost is listed: “$1.50 — Cost of EHM [Employee Health Management] PMPM fees.”
The two lists of costs are totally inconsistent. Suddenly, when it comes time to measure ROI on page 15, most of the costs on Page 10 have disappeared…
The reason for that? The savings from wellness – in the HERO committee’s own words below – are so trivial that in order for wellness to produce savings, the second screenshot has to ignore most of the costs listed on the first one. Whereas the first screenshot listed three categories of costs covering 12 different line items (13 if you count the AWOL consulting expenses), the second screenshot says you should only count one item: vendor fees.
And by the way, the vendor fees themselves self-invalidate. At about $40 per employee per year, biometric screening fees alone cost more than the stated $1.50 per person per month, or $18/year. Yet $18/year is the total they list for all fees combined, including the $40 screenings.
Rather than point out the many cost elements on the first screenshot missing from the second, we’ll invite you to use our spreadsheet and enter your own data instead of theirs. Simply fill in your own direct costs of wellness.
Whatever number you get will dramatically understate your true costs because there are three elements of cost that we aren’t counting on this spreadsheet:
- What their spreadsheet call the “indirect” costs, which we have listed as “$0”,
- What their spreadsheet calls the “tangential” costs of damaged reputations and employee morale—ask Honeywell whether they brag about their wellness fines and lawsuit in their recruiting (and, ironically, I just returned from a consult for Penn State itself, where the adverse morale impact still overhangs employee relations);
- The massive costs of overscreening, overdiagnosis, and overtreatment generated by biometric screens – all of which are conducted far more often than the USPSTF recommends and most of which (as in the examples we occasionally post on this site) include screens that no one other than a wellness vendor or consultant would ever propose.
The financial benefits
Against those costs are the benefits. Page 15 lists some alleged benefits of wellness that leave us scratching our heads.
Generic substitution? How does that have anything to do with wellness? Quite the contrary, obsessing with wellness might take your eye off the generic substitution ball, and cause you to miss some tiering opportunities. (The company that is best at tiering its pharmacy benefit, Procter & Gamble, is also known for its current employee-friendly wellness program, sort of the anti-Honeywell.) And has anyone ever seen one health risk assessment (HRA) or participated in one health screen that even mentioned generic substitution?
Outpatient procedures? Try to find one person in your organization whose outpatient procedure could have been prevented by eating more broccoli.
ER visits? Maybe they decline. But maybe they increase, due to sports injuries sustained by newly activated employees. And someone who really is eating more broccoli might slice their finger chopping the crowns off the stalks. (Anybody who voluntarily eats the crowns with the stalks still attached doesn’t need a wellness program.)
And then the catch-all: savings through “overall wiser use of healthcare.” Come again? This is an industry that — as well documented by their own words captured on this website — makes its living telling employees to do exactly the opposite: go get checkups you don’t need and won’t benefit from, submit to screens far in excess of USPSTF guidelines so that vendors can brag about how many sick people they find, yo-yo diet for “biggest loser contests” and weigh-ins, like ShapeUp’s get-thin-quick 8-week crash-diet programs, and avoid eating fat and cholesterol and load up on carbs instead.
Perhaps what the HERO committee intends is that since employees largely don’t trust their employers, they will do the opposite of the recommendations.
The savings from wellness
We are going to leave out respiratory savings. To capture those, charge a smoking differential and make smoking cessation available. Done. You don’t need an intrusive and expensive wellness program for that. (We are big believers in a “smoking differential” for employee-paid premiums. It makes sense for all the reasons weight loss and other wellness programs don’t.)
Instead let’s focus on people who have cardiometabolic issues. In order to lose weight and reduce their risk, they need to switch to a low-fat, low-cholesterol diet.
Oh, my bad! That is sooo 2014! While most of us not in the wellness business already knew the dangers of eating too many simple carbohydrates long before now, even the most ardent card-carrying member of the Wellness Ignorati learned in March that all their dietary advice has been wrong — to go along with their incorrect screening and checkup advice. Yet recommending exactly the wrong things hasn’t stopped most vendors from claiming massive savings. See “On the (Even) Lighter Side” and The Smoking Guns for examples.
Now let’s look at all the hospitalizations that can be avoided through wellness – heart attacks, angina, hypertension, and…um, hmm…did we mention heart attacks? You’re thinking: “What about diabetes events?” OK, we’ll add diabetes, only because the HERO report lists it and we want to be true to the report. But diabetes complications admissions (like CHF, which they also list) are a disease management issue, not a wellness issue — you can’t prevent or manage diabetic neuropathy or left-ventricular heart failure by eating more broccoli. The $1.50 PMPM price would not be high enough to also include disease management, and in any event what one does in disease management for complex cases is much different from a typical “pry, poke, prod and punish” wellness program.
And “straight” diabetes admissions are usually the result of diabetic employees pushing their blood sugar too low by over-medicating themselves—often in a good-faith effort to hit Hba1c “targets” that your wellness program set, no doubt on the advice of your consultants. Low blood sugar won’t do much for productivity either. Without the advice of a company specializing in diabetes, you’re likely to get this result. (And if this is the first you are hearing about the likely causes of “straight” diabetes ER visits and admissions, you should consider such an option.)
So we are now adding all ischemic and hypertensive heart events and diabetes as what they call “potentially preventable hospitalizations.” How many of your hospitalizations are for those items? Simply run the primary codes for those events, being careful not to double-count professional fees, to see how many you had. Here’s what happens when you do it for the United States as a whole.
Next, divide the relevant figure (Private insurance, 432,065) by the total number of privately insured discharges in the US (7,360,684)
So—using the HERO Committee’s own acknowledgment of the undeniable fact that wellness can only impact wellness-sensitive medical events (WSMEs) and using the diseases that the report says to use — less than 6% of admissions are WSMEs. If your non-birth-event admit rate is, as the report says, 45 per 1000, then you have 2.6 admissions per 1000 in non-smoking-related WSMEs. Once again, don’t take our word for this. Run this analysis on your own admissions. You won’t be surprised by how few there are. Do you even know anyone admitted to the hospital for these things, especially where the admissions could have been prevented with a few more screens, HRA and servings of broccoli?
Shameless plug: We are happy to do this WSME analysis for you. We do these all the time. It’s $4000. We can also tell you your savings, ROI, trend, comparison to others over time, and more. We also adjust for the major secular decline in cardiac events that has been taking place anyway for decades that the Committee seems to be unaware of, sort of surprising given their alleged expertise in cardiac risk reduction.
Let’s say you run this analysis with or without our help, and a rate/1000 similar to the US average pops up. The HERO report says you need to reduce this rate by “only 1 or 1.25 admissions.” But that’s almost half of your total 2.6/1000 WSMEs! And in any event, you’ve probably seen by now – if you downloaded the spreadsheet – that Page 15 seriously underestimates your wellness program expenses, meaning your breakeven reduction needs to be much higher than “only 1 or 1.25.” It’s probably higher than the number of admissions you have available to be reduced.
You can enter both your admissions per 1000 and the reduction in that figure you achieved directly into the spreadsheet.
But for now let’s very generously assume their expenses are right, and you only need to reduce admissions by 1 to succeed. How hard would it be to go from 2.6 to 1.6 WSMEs per 1000, a reduction of 39%? Here are five things to keep in mind:
- Your true engagement rate itself is probably much lower than that aforementioned 39%, not including people who simply participate for the money, and the people who are engaged generally aren’t the ones who would crash anyway;
- A big chunk of all heart attacks can’t be predicted at all, and certainly not now that law prohibits asking about family history;
- Even events that can be generally predicted can’t necessarily be prevented (we all know people who are “walking heart attacks” and have been ignoring advice for years);
- “Straight” diabetes admissions are more likely to be for over-control than under-control;
- In 7 years of measuring this, we have never seen a reduction in WSMEs remotely approaching 39% after adjusting for secular declines in cardiac events that take place even without a wellness program (which the report overlooks)
See The Million Dollar Workplace Wellness Heart Attack Screen in Health Affairs for a more in-depth view of the math. But the entire committee writing this HERO report insists wellness saves money, right? So, it’s us against them, right? A he said-she said? Wrong. Here’s the denouement. On Page 23, the report’s own example shows that wellness only saves $0.99 PMPM! That figure, by the way, is grossly overstated for reasons we will get to when we deconstruct Page 23. But for the time being, here it is.
So even their own comparison of their own overstated savings estimates to their own understated cost estimates reveal: wellness is a loser financially. They have already admitted it is a loser for employee relations. Funny — if we had made these two arguments, they would attack us. But they are making these two arguments themselves.
Once again, the Surviving Workplace Wellness mantra applies: “In wellness, you don’t have to challenge the data to invalidate it. You merely have to read the data. It will invalidate itself.”
Where does this leave us?
To summarize, pages 10, 15 and 23 combined tell us:
- Even before adding page 10’s cost categories back to page 15, costs are $1.50 PMPM;
- Savings are only $0.99 PMPM, meaning wellness loses $0.51 PMPM;
- The first two points are not our estimates — they’re their estimates and are far more optimistic than ours;
- Adding back the cost elements on page 10 to page 15, and then on Page 23 removing the respiratory savings, adjusting for secular decline in WSMEs, and adding in all the extra doctor visits would create a much larger loss from wellness;
- And they have already admitted that “pry, poke, prod and punish” programs are bad for morale.
Now you see why RAND’s PepsiCo study showed a negative ROI from wellness: It’s because there is a negative ROI from wellness, which no one disputes any more.
And you see the reason we asked the question in the last installment: Why would any company “do wellness” if the biggest proponents of wellness – people who make their living off it – admit that it’s a waste of money that adversely impacts morale?
Likewise, now you see why wellness vendors and consultants get “outed” all the time on this site, advocate savings methodologies designed to obfuscate rather than enlighten, and try to prevent you from learning that we exist. We are not saying they are sociopaths. Sociopaths lie for no reason. Conversely, wellness vendors and consultants are just trying to keep their jobs. Bleeding customers or clients dry is only a good job security plan if indeed the customers or clients never find out about it.
But now customers know how their own vendors and consultants really feel. And we can all work together to dismantle these programs and start doing wellness for employees instead of to them.
Poll: Cue the Wellness Industry Response…
We have a little dispute with RAND’s Soeren Mattke. He says the wellness industry modus operandi is, whenever one claim is disproven, to switch to another claim.
We say the reason they are known as the Wellness Ignorati is, their strategy is to ignore facts, including ones they admit, and they will simply just ignore this posting so as not to create a news cycle, rather than switch claims.
There is also the chance that they admit that their own financial model is accurate. This would demonstrate integrity, a quality historically in short supply in this field.
So vote early (but not often)…
While we aren’t deconstructing this as a sales tool for Quizzify. But as it happens, Quizzify is literally the only wellness program that does pay for itself. Don’t take our word for it. Quizzify is 100% guaranteed to save money and improve morale/engagement–exactly the opposite of what the HERO report says usually happens. No other wellness program is either, let alone both.
By Al and Vik
Harvard Professor Katherine Baicker is arguably the most acclaimed health policy researcher at arguably the most acclaimed (and not even arguably, the best-endowed) school of public health in the country. Her seminal account of the effect of Medicaid coverage on utilization and health status is a classic. As luck would have it, in 2008 Oregon used a lottery to ration available Medicaid slots. A lottery controls for motivation and as such eliminates participant-non-participant bias, since everyone who enters the lottery wants to participate. That meant only one major variable was in play, which was enrollment in Medicaid or not.
Chance favors the well-prepared, and Professor Baicker jumped on this research windfall. She found that providing Medicaid–and thereby facilitating access to basic preventive medical care–for the previously uninsured did not improve physical health status, but did increase diagnoses and utilization. Because of the soundness of the methodology, the conclusion were unassailable – more access to medical care does not improve outcomes or optimize utilization, which is a proxy for spending. (We ourselves reached a similar conclusion based on a similar analysis on North Carolina Medicaid’s medical home model.)
Yet Professor Baicker herself used exactly the opposite methodology to reach the exact opposite conclusion for workplace wellness. And that’s where the identity crisis begins.
She and two colleagues published a meta-analysis in 2010 of participant-vs-non-participant workplace wellness programs. Somehow—despite her affinity for Oregon’s lottery control—she found this opposite methodology to be acceptable. She concluded that workplace wellness generated a very specific two significant-digit 3.27-to-1 ROI from health care claims reduction alone, with another 2.37-to-1 from absenteeism reduction. The title of the article–now celebrating its fifth anniversary as the only work by a well-credentialed author in a prestigious journal ever published in support of wellness ROI—was equally unambiguous: Workplace Wellness Can Generate Savings.
This article wasn’t just an academic exercise. It gave the Obama administration academic cover for what has proven to be the most unpopular, dishonest, and even hazardous component of the Affordable Care Act: allowing employers to financially and clinically punish employees with coercive directives to lose weight, get unnecessary checkups, and answer intrusive, distasteful, and counterproductive questions about (for example) checking their testicles.
Professor Baicker did not question her too-good-to-be-true conclusion. Yet the Law of Diminishing Returns clearly contradicts her finding. Compelling privately insured people to get more healthcare is very unlikely to improve health status and reduce healthcare expense if provision of basic insurance to a medically needy population doesn’t noticeably improve health status while increasing healthcare expense.
Instead, she reveled in the limelight, receiving 307 citations, vs. 18 and 9 for two other Health Affairs articles on wellness that didn’t support more spending on vendors and consultants. (Even 307 citations aren’t enough to satisfy one of the leaders of the wellness movement, Larry Chapman, who says this study should be cited much more frequently since it’s basically the one that supports the entire industry.) However, at some point in 2013, overwhelming evidence totally invalidated her findings. At that point – like Dee Edington and Al Lewis, both of whom had previously reversed positions when the data didn’t support their previous positions—she could have acknowledged that her initial findings had been wrong and moved on.
Instead, she neither defended her position nor clearly refuted it, choosing instead a yin and yang middle ground that shifted with every interview. The metamorphosis from Queen of Significant Digits into the Queen of Significant Doubt started in July 2013, when she announced on NPR’s Marketplace that “it’s too early to tell” if wellness saves money, and that employers need to “experiment” with these programs to “see what happens to participants’ weight and blood pressure.” Right there, she invalidated herself. First, by then she certainly knew that a participants-vs-non-participants methodology was invalid since the key “smoking gun” slide in our Health Affairs posting was already widely circulated and her own opposing Oregon methodology was being widely praised. Second, even if she is right, the financial payoff for the modest “weight and blood pressure” improvements that the best programs might generate is 10-20 years in the future — and even then only if the improvements are sustained.
But then came another personality change.
In February 2014, she blamed readers for focusing on her attention-grabbing headline, the certainty of her two significant digits, and the gist of the conclusion…while ignoring the fine print, such as a caution about publication bias. Publication bias? You think? Start with the standard publication bias that negative articles rarely get published because they don’t get cited and hence reduce the all-important “impact factor” – recall the difference in Health Affairs citations in her own wellness article vs. the others.
Add to that a publication bias specific to those journals: most of the articles comprising her meta-analysis were published in third-tier journals. Among them, these journals have exactly once published an article critical of wellness (twice if you include a book review by the esteemed Norton Hadler, whom a third-tier journal is thrilled to publish regardless of what he says, and three times if you include publication of an article by a graduate student at the University of Tasmania that accidentally undercut the true believers’ own storyline, that they are now having to explain away).
Weeks later, a totally different personality emerged: she told the editor of Insurance Thought Leadership that she no longer focused on wellness and consequently has no opinions to share. Leaving aside the irony that the wellness true believers continue to cite as gospel someone who says she has no interest in what they are citing her for, this spin further invalidates her next comment, delivered in December 2014 – when suddenly, as a result of yet another personality change, she has opinions again. She told All Things Considered: “It could be that when the full set of evidence comes in, [wellness] will have huge returns on investment.”
Oops. First, she has just admitted she doesn’t follow wellness, so why speculate on future studies she has no knowledge of in a field she’s not involved in? Second, there is a rule of thumb in epidemiology: the bigger the impact, the smaller the sample size needed to discern it. An example would be smoking and lung cancer, a previously very rare disease whose cause was discernable from a handful of cases. A sample of only hundreds of veterans was needed to prove that very high blood pressure causes strokes, and studies of exercise almost always show either a physical or emotional benefit, even in small groups of people with significant disease. On the other hand, there have been probably close to a half-billion employee-years of wellness with nothing to show for themselves except results going the other way and a bunch of self-invalidating vendor lies.
So we are going to make a radical proposal to the true believers: you can continue to cite Katherine Baicker but must also note that she herself no longer supports the study you are citing — until and unless she says she does. In exchange for this disclosure, when do you cite her, we will acknowledge that you are telling the truth for a change.
Short Summary of Intervention:
“At Wellsteps, we’ve created a series of research-based ROI calculators to help you estimate the effect of well-designed wellness programs on health care costs, absenteeism, and presenteeism. Each of the three ROI calculators will examine a different employee expense and will help you determine whether investing in wellness strategies makes sense for your company. A well-designed wellness program is one that changes the health behaviors of employees, spouses, and dependents, and lowers health risks, reduces chronic disease, and helps worksites create a culture of health. The design of the WellSteps turnkey wellness solution was based partly on this body of evidence.”
Materials Being Reviewed:
Wellsteps ROI Calculator . You input your number of employees, health spending, and goals for obesity and smoking cessation. The calculator will tell you how much money you can save through the Wellsteps program.
Summary of key figures and outcomes:
Questions for Wellsteps:
In the first example above, your model calculated massive savings even with no change in obesity and smoking. In the second example, your model calculated the same massive savings even with a huge increase in obesity and smoking. It seems that no matter what smoking and obesity data we enter once we factor out inflation itself, your ROI calculator reduces healthcare costs to a level below zero by 2019. How is this possible?
ANS: Refused to answer
Shouldn’t a spike in smoking and obesity rates from 0% to 99% increase healthcare spending rather than reduce it?
ANS: Refused to answer
Your May 2014 email blast, sent out a few days after The Health Care Blog exposed your ROI model as being invalid, says your model is supported by “every wellness ROI study ever published” (a step up from being “research-based” on a “body of evidence” as your website says). We recognize that asking you to list “every wellness ROI study ever published” would be burdensome, but could you direct us to just one study that says increasing smoking and obesity can improve workforce health and/or reduce healthcare costs to below zero?
ANS: Refused to answer
Does “every wellness ROI study ever published” include the RAND studies in Health Affairs that have found negative ROIs?
ANS: Refused to answer
How are you able to “guarantee” this ROI, since it is impossible to reduce spending to a negative number?
ANS: Refused to answer
Since you’ve known that the Wellsteps ROI Calculator is invalid since this fact was pointed out to you in October 2013 and you have updated your model twice since then, how come you have elected to continue to overstate savings by a mathematically impossible figure?
ANS: Refused to answer
You lead your marketing blast by saying that “11,000 brokers and consultants” have used this ROI model? Are we the first of those 11,000 people to observe negative savings?
ANS: Refused to answer
Do you see any irony in publicly accusing one of the principals of this website, Al Lewis, of acting like a “tobacco executive lying to Congress” when even tobacco executives wouldn’t claim that smoking reduces healthcare costs like your model says it does?
ANS: Refused to answer
Where in your “ROI Calculator” can I find the ROI?
ANS: Their ROI Calculator doesn’t calculate an ROI so there was no point in even asking them to answer this question. The good news about Wellsteps is that NASA employees don’t have to worry about job security because these people are not rocket scientists.
Update: July 16, 2014
Addendum: Wellsteps accusation that I “entered false data” into the ROI calculator was posted on the “Wellness Is a Business Strategy” Linkedin Group
“It has come to our attention that an outspoken critic has entered false data into these calculators in order to make a point. We certainly support free speech; however, we wonder how valid the point can be when it is based on false data?”
“Use valid estimates for the percent of the current obese and smokers in an employee population. We provide the estimated default numbers based on national data of 33% and 20% respectively in all three calculators. In our combined 50 years of academic and consulting work, we cannot think of one employer with a 0% obesity AND 0% smoking rate. Again, valid estimates work best.”
Actually, we ran every combination of data from a reduction of 99% in smoking and obesity rates to an increase of 99% in smoking and obesity rates. Some of that data might have been “false” (whatever that means), but the result was always the same: $1358.85 in savings/employee by 2019. Here are two more examples, this time using the default numbers they recommended. The first is @$5000/employee in annual costs, with no change in smoking or obesity. The second is @50,000/employee. The answer is still the same.
How come, regardless of what assumptions get entered (and we have now entered many vastly different combinations of cost and success), the answer is always that by 2019, you save $1,358.85 per employee once you zero out inflation?
ANS: Refused to answer
Would you now admit that entering $5000/employee in spending and 33% obesity and 20% smoking (the national averages) constitute, in your words, “valid estimates that work best” ?
ANS: Refused to answer
Following my exposé, your model no longer allows a user to enter increases in smoking and obesity. Is this to prevent users from figuring out that even if the rates of smoking and obesity increase, the math underlying your model based on “every ROI study ever published” will still show a reduction of $1,358.85/employee in 2019?
ANS: Refused to answer
How come the model shows that very same $1358.85 (now finally “rounded” to $1359) potential savings from reducing obesity and smoking even if I start out with no smokers or obese employees?
Update–May 1, 2015: Wellsteps Doubles Down on Dishonesty
Two people forwarded us this, a sequel to their email that their ROI model is “based on every ROI study ever published”:
Update–September 10, 2015: Wellsteps Triples Down on Dishonesty
Somehow they reported costs going up and down at the same time. Even wellness industry math doesn’t allow that.