They Said What?

Home » Posts tagged 'Livongo'

Tag Archives: Livongo

Was Livongo’s “peer-reviewed journal article” really just an ad?

Here’s how an ad gets published. It’s a two-step process. I will lay it out so that even the dumbest member of the media who somehow missed this the first time when they swooned over Livongo’s outcomes can understand it now:

  1. Your employees and their colleagues write it.
  2. You pay to have it published.

Now, let’s look at what Livongo just published, touting their own outcomes, to see how, if at all, it differs from an ad.

  1. Their employees and colleagues wrote it.
  2. They paid to have it published.
    • We missed this the first time around. Our excuse is, so did quite literally everyone else who covered the story. And “covering stories” isn’t our Day Job. We aren’t journalists. We don’t even play them on TV.  We’ve never even watched journalist shows on TV, unless you include Superman reruns. Livongo seems to have a lot in common with that show, transparency being their kryptonite.

The journal is called the Journal of Medical Economics. Sounds really prestigious, so points for that. Yet virtually no other journal article cites articles in this journal, giving it an Impact Factor south of 2. (New England Journal of Medicine gets a 70.) Turns out there’s a reason no one cites it. Here’s how you get published in it. You pay them money.

They would say, yes, but we got it peer-reviewed. To which I say, apparently you didn’t in any meaningful sense. A real peer reviewer would have found and questioned all the fallacies in their article, rather than rubber-stamp some very sketchy “findings,” which for convenience’ sake are all catalogued in one place.


There is nothing wrong with advertising your outcomes, as long as your ad is labeled as an ad. You often see airline magazines with entire sections advertising various cities, using articles and pictures. But they are always labeled as ads. If you don’t do this, there is always the slight possibility, however remote, that someone doesn’t do the research to figure out that in fact this publication was pay-to-play. If that were to happen, you might see a headline like this:

Whereas a more accurate headline might read: “Livongo Pays for an Article to Claim Its Product Works.”


Update January 3: Someone contacted me to say that the correct term for paid highly information advertising is “Sponsored content.” This term would apply perfectly to Livongo’s self-generated, self-published study. They should relabel it as such.

Are Livongo’s outcomes real?

Breaking news first…

Sign up for the Validation Institute webinar and learn why Yale got sued for wellness because they made a series of rookie mistakes…and how you can avoid them.

Attention, Pennsylvanians: register now for the Pittsburgh Business Group on Health Symposium 9/12…and play in our trivia contest.


And, now, for the rest of the story.

Kudos to employers who have resisted the entreaties of their carriers to jump on the Livongo bandwagon. Resistance is not easy — carriers will pester employers endlessly because they get a nice commission every time an employer bites.

Instead, I would simply recommend waiting until Livongo answers these five questions and gets their results validated by the Validation Institute. One would think that if they are losing $30-million/year anyway, spending an extra $10,000 to see if their offering actually works shouldn’t break the bank.

Here are the five questions that Livongo apparently can’t answer. Or they can, but choose not to. Not sure which inspires less confidence.

The references below mention a “study.”,  By way of background, this study was conducted by Livongo’s employees, along with employees of its partnered diabetes supply company (Eli Lilly), which also funded the study. So there couldn’t possibly have been any conflict of interest, right? Right?

It was published in something called the Journal of Medical Economics (JME). And no, I hadn’t heard of this publication either. Turns out it’s an “open-access” journal offering “accelerated publication,” with an Impact Factor of 1.9.

Not familiar with the concept of Impact Factors? Those measure the influence of a publication. For instance, the New England Journal of Medicine tallies a 70.8. How hard is it to only get a 1.9? Even the American Journal of Health Promotion (AJHP), which recently proposed charging employees for insurance by the pound, scratches out a 2.6.  Possibly, that’s because AJHP does sprinkle more humor into its content than JME, like:

I guess that means 10 states both prohibit it and allow it.


Question #1 for Livongo

Even obscure journals need to do a little peer review (which itself is increasingly considered to be a joke), and this peer review was pretty basic: the authors were asked to disclose that the study couldn’t draw any causal relationship between the Livongo intervention and the results. The title of the article itself says that reduced medical spending is “associated with” their product. Later the authors say the results “imply” the product works.

So if the study showed only a correlation and not causation, why does Livongo’s press release announce:

“The findings showed that by using its remote digital health platform, the Livongo for Diabetes program delivered an $88 per member monthly reduction” ?


Question #2 for Livongo

Since these alleged findings are the opposite of Livongo’s initial claim below, featured in a recent Valid Points posting about diabetes vendors snookering purchasers, the second question is, how did the original 59% reduction claim (which basically requires eliminating every hospitalization not connected with births, trauma or cancer) get replaced by the opposite claim that large reductions in physician visits generate all the savings, while admissions increased?

A while back, I actually looked into the likelihood that tighter glycemic control, which itself is rather controversial, (“there is good evidence that intensive control of blood glucose increases patients’ relative risk of severe hypoglycemia by 30%”) could reduce hospitalizations. I used very optimistic assumptions (since I was consulting for a company making a glucometer with remote capabilities, not unlike the one Livongo pitches). Here’s what I came up with: savings of $27 per diabetic per year in inpatient admissions, as opposed to Livongo’s $88 per diabetic per month.

In all fairness to Livongo, they don’t promote that inpatient admissions result anymore (I may have missed the apology for fudging that outcome in the first place), focusing instead on getting doctors to do fewer visits by managing patients remotely.

Seems curious that physicians would be doing more work – more notifications from remote monitoring devices about blood sugar, more titrating dosages – and be perfectly fine making 26% less money. Plus, every other wellness vendor brags about how many more physician visits they generate, not how many fewer.


Question #3 for Livongo

If I’m seeking a vendor to reverse diabetes in my population (assuming that is even possible on a broad scale), I would look for weight loss (without which diabetes reversal is basically impossible, but then again sustained weight loss itself is pretty close to impossible) as the leading indicator.

As an intermediate indicator, I would measure units of insulin use across the entire population. That should decline if people are eating much better an exercising much more and losing weight.

The end-point indicator would be a decrease in admissions for diabetes. Of course, since diabetes admissions generally increase following retirement, it isn’t exactly easy to save money in decreased admissions for diabetes, for the simple reason that there hardly are any.

Specifically, for the last year in which a full set is available (2014),158 million commercially insured <65 people generated 126,710 admissions for diabetes. Meaning that in the commercially insured population, the admission rate is so low that a 10% reduction in admissions (which has never been achieved in any population health program) would mean that an employer with 10,000 employees would avoid — get ready — 1 admission.

We suspect Livongo knows this because they listed the diabetes diagnosis codes in their appendix, and it takes about 5 minutes to tally the US admission rate for those codes using the federal database designed for that purpose. And if they don’t know it, they should. Any health services researcher should be aware of this database.

So why didn’t Livongo measure any of those three outcomes? Or did they measure them and decide not to report them?

I’m not sure which answer is “right”: While I would be very concerned if they were suppressing data, I think I would be even more concerned if they didn’t know enough about diabetes to measure the key outcomes.


Question 4 for Livongo

Why did Livongo measure participants against non-participants, when that study design is known to be completely invalid?  Benchmarking a par-vs-non-par result has been done five times, including three times by wellness promoters hoping to prove validity of the design  The conclusion in each case: 100% of difference in outcomes between the two groups is attributable to the study design, and 0% to the intervention. The design is 100% invalid.

There was actually a case in which the two groups were separated and “match-controlled” in 2004, but the program didn’t start until 2006. During the 2-year period following separation but before the program became available to participate in, the would-be participants nonetheless dramatically outperformed the non-participants…despite not having a program to participate in. (This result caused quite a ruckus in the wellness industry once they realized what they had done.)

Further, more than 15% of the initial Livongo participants dropped out. Assuming dropouts largely failed, aren’t the authors of the Journal article overstating the outcomes for the participant group as a whole by not counting dropouts? Isn’t that like on-time performance statistics not accounting for planes that crashed?


Question 5 for Livongo (really multiple questions)

Livongo offers “free unlimited glucose test strips” to try to reach a “target level of glucose control of Hb A1c <6.5%.”

It is not quite clear that either unlimited free strips or a 6.5% Hb A1c target are good for diabetics. Why, one might ask, does the American College of Physicians propose 7% to 8% instead of 6.5%? While it is true that the American Diabetes Association is sticking with its much lower blood sugar goal, isn’t it also the case that the ADA is largely funded by companies that make products to help lower blood sugar?

And what is the rationale for encouraging more use of glucose strips while Choosing Wisely recommends less testing for many Type 2 diabetics (in JAMA Internal Medicineimpact factor 20)?  Exact words:

This recommendation is based on robust evidence, including a Cochrane review of 12 randomized clinical trials with more than 3000 patients, showing no statistical difference between patients who do not self-monitor their blood glucose multiple times per day and those who do self-monitor their blood glucose multiple times per day in glycemic control, nor evidence of effects on health-related quality of life, patient satisfaction, or decreased number of hypoglycemic episodes.


So it could be that this is just me, since their revenues are doubling every year. Maybe, like in the rest of wellness, answering questions about efficacy is beside the point. Maybe the point in HR is just to do something, regardless of whether it works, in case someone in the C-Suite asks. Or in the immortal words of the great philosopher Yogi Berra: “We don’t know where we’re going, but we’re making good time.”

Questions to ask Livongo before signing with them

In 2014, the Validation Institute, then a joint property of Intel and GE, tasked me, based on Why Nobody Believes the Numbers, to develop the most sophisticated outcomes measurement course ever devised. That was not a heavy lift, because at the time no other such course existed. They named it Advanced Critical Outcomes Report Analysis, or “Advanced CORA” for short.

Five years later, the course, now called CORA Pro, remains the Gold Standard in outcomes measurement expertise. CORA Pros — and there are only 8 of us — have developed a talent not just for sniffing out questionable claims (anyone can “challenge the data”), but more importantly for identifying contradictory, implausible and even impossible claims that somehow hadn’t been noticed before. Even a simple press release can become a case study in impossibility.

CORA certification itself is very useful, and takes you halfway there. But there’s something special about CORA Pro. Once you are certified in CORA Pro, you never look at a vendor outcomes report the same way again. For instance, consider Livongo’s recent glowing press release about its outcomes. Non-CORA Pros, like whoever these people are, basically just parrot a vendor’s results uncritically with a glowing headline. They sound like Flounder.


By contrast, a CORA Pro would review Livongo’s claims thoughtfully, and ask probing questions, first about the outcomes:

  • If your goal is to reduce admissions for diabetes and reduce insulin usage across the population, why didn’t you measure admissions for diabetes, or insulin usage?
  • Why does this report show the opposite of your report from 2018? This report showed huge reductions in outpatient and no significant change in inpatient, whereas the 2018 report showed huge reductions in inpatient with no significant change in outpatient?


 

Then a CORA Pro would ask whether Livongo may be harming employees:

  • Why is your goal Hb A1c of 6.5% in conflict with the American College of Physicians and other expert bodies not funded by the diabetes industry, which advocate 7.0% to 8.0%? Perhaps they mean reaching 6.5% with diet-and-exercise, but if insulin use is going up, which is likely because they didn’t claim it was declining, clearly some employees aren’t getting the subtle nuance that they shouldn’t aim for 6.5% with medication.
  • Why is Choosing Wisely advocating fewer glucose checks if the right answer is to check multiple times a day?

Just askin’…  Because that’s what CORA Pros do.


The full list of questions can be found here.

To learn about CORA certification, click here.

And to see how we predicted many of the ways Livongo would measure outcomes using vendor-friendly metrics, click here.

 

%d bloggers like this: