Apple unveiled plans to scan
U.S. iPhones for images of child sexual abuse, drawing applause
from child protection groups but raising concern among some security
researchers that the system could be misused by governments looking to surveil
their citizens.
Apple
said its messaging app will use on-device machine learning to warn about
sensitive content without making private communications readable by the
company. The tool Apple calls “neuralMatch” will detect known images of child
sexual abuse without decrypting people’s messages. If it finds a match, the
image will be reviewed by a human who can notify law enforcement if necessary.
But
researchers say the tool could be put to other purposes such as government
surveillance of dissidents or protesters.
Matthew
Green of Johns Hopkins, a top cryptography researcher, was concerned that it
could be used to frame innocent people by sending them harmless but malicious
images designed designed to appear as matches for child porn, fooling Apple’s
algorithm and alerting law enforcement -- essentially framing people.
“Researchers have been able to do this pretty easily,” he said.
Tech companies including
Microsoft, Google, Facebook and others have for years been sharing “hash lists”
of known images of child sexual abuse. Apple has also been scanning user files stored in its
iCloud service, which is not as securely encrypted as its messages,
for such images.
Some
say this technology could leave the company vulnerable to political pressure in
authoritarian states such as China. “What happens when the Chinese government
says, ‘Here is a list of files that we want you to scan for,’” Green said.
“Does Apple say no? I hope they say no, but their technology won’t say no.”
The
company has been under pressure from governments and law enforcement to allow
for surveillance of encrypted data. Coming up with the security measures
required Apple to perform a delicate balancing act between cracking down on the
exploitation of children while keeping its high-profile commitment to
protecting the privacy of its users.
Apple
believes it pulled off that feat with technology that it developed in
consultation with several prominent cryptographers, including Stanford
University professor Dan Boneh, whose work in the field has won a Turing Award,
often called technology’s version of the Nobel Prize.
The
computer scientist who more than a decade ago invented PhotoDNA, the technology
used by law enforcement to identify child pornography online, acknowledged the
potential for abuse of Apple’s system but said it was far outweighed by the
imperative of battling child sexual abuse.
“It
possible? Of course. But is it something that I’m concerned about? No,” said
Hany Farid, a researcher at the University of California at Berkeley, who
argues that plenty of other programs designed to secure devices from various
threats haven’t seen “this type of mission creep.” For example, WhatsApp
provides users with end-to-end encryption to protect their privacy, but employs
a system for detecting malware and warning users not to click on harmful links.
Apple
was one of the first major companies to embrace “end-to-end” encryption, in
which messages are scrambled so that only their senders and recipients can read
them. Law enforcement, however, has long pressured for access to that
information in order to investigate crimes such as terrorism or child sexual
exploitation.
“Apple’s
expanded protection for children is a game changer,” John Clark, the president
and CEO of the National Center for Missing and Exploited Children, said in a
statement. “With so many people using Apple products, these new safety measures
have lifesaving potential for children who are being enticed online and whose
horrific images are being circulated in child sexual abuse material.”
Julia
Cordua, the CEO of Thorn, said that Apple’s technology balances “the need for
privacy with digital safety for children.” Thorn, a nonprofit founded by Demi
Moore and Ashton Kutcher, uses technology to help protect children from sexual
abuse by identifying victims and working with tech platforms.
