Apple is purportedly poised to announce a brand new instrument that can assist determine baby abuse in pictures on a consumer’s iPhone. The instrument would supposedly use a “neural matching function” to detect if photographs on a consumer’s machine match recognized baby sexual abuse materials (CSAM) fingerprints. While it seems that Apple has taken consumer privateness into consideration, there are additionally considerations that the tech might open the door to unintended misuse—notably in relation to surveillance.

The information comes by way of well-known safety professional Matthew Green, an affiliate professor at Johns Hopkins Information Security Institute. The information has but to be confirmed by Apple, nonetheless. That mentioned, Green is a reputable supply who’s written extensively about Apple’s privateness strategies through the years. Notably, he’s labored with Apple previously to patch a security flaw in iMessage.

“I’ve had independent confirmation from multiple people that Apple is releasing a client-side tool for CSAM scanning tomorrow. This is a really bad idea,” Green tweeted in a thread late final night time. “These tools will allow Apple to scan your iPhone photos for photos that match a specific perceptual hash, and report them to Apple servers if too many appear.”

The crux of the difficulty is that whereas numerous tech corporations, together with Apple, have added end-to-end encryption to their companies and merchandise, it’s been opposed by numerous governments. While end-t0-end encryption is a win for shopper privateness, the argument is it additionally makes it tough for legislation enforcement in makes an attempt to crack down on unlawful content material like baby pornography. According to Green, a “compromise” is to make use of these scanning applied sciences on the “client-side” or, in your telephone earlier than they’re despatched and encrypted on the cloud. Green additionally claims that Apple’s model wouldn’t initially be used on encrypted photographs—simply your iPhone’s photograph library if and provided that, you’ve gotten iCloud Backup enabled. In different phrases, it might solely scan pictures which are already on Apple’s servers. However, Green additionally questions why Apple would undergo the hassle of designing this kind of system if it didn’t have eventual plans to make use of it for end-to-end encrypted content material.

No one desires to go to bat for baby pornography, however Green factors out this tech, whereas nobly supposed, has far-reaching penalties and might probably be misused. For occasion, CSAM fingerprints are purposefully a little bit imprecise. That’s as a result of in the event that they have been too exacting, you might simply crop, resize or in any other case edit a picture to evade detection. However, it additionally means dangerous actors might make innocent photographs “match” problematic ones. One instance is political marketing campaign posters that might be tagged by authoritarian governments to suppress activists, and so forth.

The different concern is that Apple is setting a precedent, and as soon as that door is open, it’s that a lot tougher to shut it.

“Regardless of what Apple’s long term plans are, they’ve sent a very clear signal. In their (very influential) opinion, it is safe to build systems that scan users’ phones for prohibited content,” Green writes. “That’s the message they’re sending to governments, competing services, China, you.”