Apple Watch’s information ‘black box’ poses analysis issues

A Harvard biostatistician is rethinking plans to make use of Apple Watches as a part of a analysis research after discovering inconsistencies within the coronary heart fee variability information collected by the units. Because Apple tweaks the watch’s algorithms as wanted, the info from the identical time interval can change with out warning.

“These algorithms are what we would call black boxes — they’re not transparent. So it’s impossible to know what’s in them,” JP Onnela, affiliate professor of biostatistics on the Harvard T.H. Chan School of Public Health and developer of the open-source information platform Beiwe, instructed The Verge.

Onnela doesn’t often embody industrial wearable units just like the Apple Watch in analysis research. For probably the most half, his groups use research-grade units which might be designed to gather information for scientific research. As a part of a collaboration with the division of neurosurgery at Brigham and Women’s Hospital, although, he was within the commercially out there merchandise. He knew that there have been typically information points with these merchandise, and his workforce needed to test how extreme they had been earlier than getting began.

So, they checked in on coronary heart fee information his collaborator Hassan Dawood, a analysis fellow at Brigham and Women’s Hospital, exported from his Apple Watch. Dawood exported his each day coronary heart fee variability information twice: as soon as on September fifth, 2020 and a second time on April fifteenth, 2021. For the experiment, they checked out information collected over the identical stretch of time — from early December 2018 to September 2020.

Because the 2 exported datasets included information from the identical time interval, the info from each units ought to theoretically be similar. Onnela says he was anticipating some variations. The “black box” of wearable algorithms is a constant problem for researchers. Rather than exhibiting the uncooked information collected by a tool, the merchandise often solely let researchers export info after it has been analyzed and filtered by way of an algorithm of some variety.

Companies change their algorithms usually and with out warning, so the September 2020 export might have included information analyzed utilizing a unique algorithm than the April 2021 export. “What was surprising was how different they were,” he says. “This is probably the cleanest example that I have seen of this phenomenon.” He printed the info in a blog post final week.

Apple didn’t reply to a request for remark.

It was putting to see the variations laid out so clearly, says Olivia Walch, a sleep researcher who works with wearable and app information on the University of Michigan. Walch has lengthy advocated for researchers to make use of uncooked information — information pulled immediately from a tool’s sensors, as a substitute of filtered by way of its software program. “It’s validating, because I get on my little soapbox about the raw data, and it’s nice to have a concrete example where it would really matter,” she says.

Constantly altering algorithms makes it nearly prohibitively tough to make use of industrial wearables for sleep analysis, Walch says. Sleep research are already costly. “Are you going to be able to strap four FitBits on someone, each running a different version of the software, and then compare them? Probably not.”

Companies have incentives to alter their algorithms to make their merchandise higher. “They’re not super incentivized to tell us how they’re changing things,” she says.

That’s an issue for analysis. Onnela in contrast it to monitoring physique weight. “If I wanted to jump on a scale every week, I should be using the same scale every time,” he says. If that scale was tweaked with out him realizing about it, the day-to-day modifications in weight wouldn’t be dependable. For somebody who has only a informal curiosity in monitoring their well being, which may be tremendous — the variations aren’t going to be main. But in analysis, consistency issues. “That’s the concern,” he says.

Someone might, for instance, run a research utilizing a wearable and are available to a conclusion about how individuals’s sleep patterns modified based mostly on changes of their surroundings. But that conclusion may solely be true with that exact model of the wearable’s software program. “Maybe you would have a completely different result if you just been using a different model,” Walch says.

Dawood’s Apple Watch information isn’t from a research and is only one casual instance. But it exhibits the significance of being cautious with industrial units that don’t permit entry to uncooked information, Onnela says. It was sufficient to make his workforce again away from plans to make use of the units in research. He thinks industrial wearables ought to solely be used if uncooked information is offered, or — at minimal — if researchers are in a position to get a heads-up when an algorithm goes to alter.

There could be some conditions the place wearable information might nonetheless be helpful. The coronary heart fee variability info confirmed related tendencies at each time factors — the info went up and down on the identical instances. “If you’re caring about stuff on that macro scale, then you can make the call that you’d keep using the device,” Walch says. But if the precise coronary heart fee variability calculated on every day issues for a research, the Apple Watch could also be riskier to depend on, she says. “It should give people pause about using certain wearables, if the rug runs the risk of being ripped out underneath their feet.”