Apple scans all images on iPhones – archived source #2

One Bad Apple

Sunday, 8 August 2021

My in-box has been flooded over the last few days about Apple’s CSAM announcement. Everyone seems to want my opinion since I’ve been deep into photo analysis technologies and the reporting of child exploitation materials. In this blog entry, I’m going to go over what Apple announced, existing technologies, and the impact to end users. Moreover, I’m going to call out some of Apple’s questionable claims.

Disclaimer: I’m not an attorney and this is not legal advice. This blog entry includes my non-attorney understanding of these laws.

The Announcement

In an announcement titled “Expanded Protections for Children“, Apple explains their focus on preventing child exploitation.

The article starts with Apple pointing out that the spread of Child Sexual Abuse Material (CSAM) is a problem. I agree, it is a problem. At my FotoForensics service, I typically submit a few CSAM reports (or “CP” — photo of child pornography) per day to the National Center for Missing and Exploited Children (NCMEC). (It’s actually written into Federal law: 18 U.S.C. § 2258A. Only NMCEC can receive CP reports, and 18 USC § 2258A(e) makes it a felony for a service provider to fail to report CP.) I don’t permit porn or nudity on my site because sites that permit that kind of content attract CP. By banning users and blocking content, I currently keep porn to about 2-3% of the uploaded content, and CP at less than 0.06%.

According to NCMEC, I submitted 608 reports to NCMEC in 2019, and 523 reports in 2020. In those same years, Apple submitted 205 and 265 reports (respectively). It isn’t that Apple doesn’t receive more picture than my service, or that they don’t have more CP than I receive. Rather, it’s that they don’t seem to notice and therefore, don’t report.

Apple’s devices rename pictures in a way that is very distinct. (Filename ballistics spots it really well.) Based on the number of reports that I’ve submitted to NCMEC, where the image appears to have touched Apple’s devices or services, I think that Apple has a very large CP/CSAM problem.

[Revised; thanks CW!] Apple’s iCloud service encrypts all data, but Apple has the decryption keys and can use them if there is a warrant. However, nothing in the iCloud terms of service grants Apple access to your pictures for use in research projects, such as developing a CSAM scanner. (Apple can deploy new beta features, but Apple cannot arbitrarily use your data.) In effect, they don’t have access to your content for testing their CSAM system.

If Apple wants to crack down on CSAM, then they have to do it on your Apple device. This is what Apple announced: Beginning with iOS 15, Apple will be deploying a CSAM scanner that will run on your device. If it encounters any CSAM content, it will send the file to Apple for confirmation and then they will report it to NCMEC. (Apple wrote in their announcement that their staff “manually reviews each report to confirm there is a match”. They cannot manually review it unless they have a copy.)

While I understand the reason for Apple’s proposed CSAM solution, there are some serious problems with their implementation.

Problem #1: Detection

There are different ways to detect CP: cryptographic, algorithmic/perceptual, AI/perceptual, and AI/interpretation. Even though there are lots of papers about how good these solutions are, none of these methods are foolproof.

The cryptographic hash solution

The cryptographic solution uses a checksum, like MD5 or SHA1, that matches a known image. If a new file has the exact same cryptographic checksum as a known file, then it is very likely byte-per-byte identical. If the known checksum is for known CP, then a match identifies CP without a human needing to review the match. (Anything that reduces the amount of these disturbing pictures that a human sees is a good thing.)

In 2014 and 2015, NCMEC stated that they would give MD5 hashes of known CP to service providers for detecting known-bad files. I repeatedly begged NCMEC for a hash set so I could try to automate detection. Eventually (about a year later) they provided me with about 20,000 MD5 hashes that match known CP. In addition, I had about 3 million SHA1 and MD5 hashes from other law enforcement sources. This might sound like a lot, but it really isn’t. A single bit change to a file will prevent a CP file from matching a known hash. If a picture is simple re-encoded, it will likely have a different checksum — even if the content is visually the same.

In the six years that I’ve been using these hashes at FotoForensics, I’ve only matched 5 of these 3 million MD5 hashes. (They really are not that useful.) In addition, one of them was definitely a false-positive. (The false-positive was a fully clothed man holding a monkey — I think it’s a rhesus macaque. No children, no nudity.) Based just on the 5 matches, I am able to theorize that 20% of the cryptographic hashes were likely incorrectly classified as CP. (If I ever give a talk at Defcon, I will make sure to include this picture in the media — just so CP scanners will incorrectly flag the Defcon DVD as a source for CP. [Sorry, Jeff!])

The perceptual hash solution

Perceptual hashes look for similar picture attributes. If two pictures have similar blobs in similar areas, then the pictures are similar. I have a few blog entries that detail how these algorithms work.

NCMEC uses a perceptual hash algorithm provided by Microsoft called PhotoDNA. NMCEC claims that they share this technology with service providers. However, the acquisition process is complicated:

Make a request to NCMEC for PhotoDNA.
If NCMEC approves the initial request, then they send you an NDA.
You fill out the NDA and return it to NCMEC.
NCMEC reviews it again, signs, and revert the fully-executed NDA to you.
NCMEC reviews your use model and process.
After the review is completed, you get the code and hashes.

(Update 2021-08-30: Today NCMEC informed me that Microsoft ended NCMEC’s ability to sublicense the code a few years ago. Now it must come directly from Microsoft.)

Because of FotoForensics, I have a legitimate use for this code. I want to detect CP during the upload process, immediately block the user, and automatically report them to NCMEC. However, after multiple requests (spanning years), I never got past the NDA step. Twice I was sent the NDA and signed it, but NCMEC never counter-signed it and stopped responding to my status requests. (It’s not like I’m a little nobody. If you sort NCMEC’s list of reporting providers by the number of submissions in 2020, then I come in at #40 out of 168. For 2019, I’m #31 out of 148.)

Since NCMEC was treating PhotoDNA as a trade secret, I decided to reverse engineer the algorithm using some papers published by Microsoft. (No single paper says how it works, but I cobbled together how it works from a bunch of their marketing blurbs and high-level slides.) I know that I have implemented it correctly because other providers who have the code were able to use my hashes to correctly match pictures.

Perhaps there is a reason that they don’t want really technical people looking at PhotoDNA. Microsoft says that the “PhotoDNA hash is not reversible”. That’s not true. PhotoDNA hashes can be projected into a 26×26 grayscale image that is only a little blurry. 26×26 is larger than most desktop icons; it’s enough detail to recognize people and objects. Reversing a PhotoDNA hash is no more complicated than solving a 26×26 Sudoku puzzle; a task well-suited for computers.

I have a whitepaper about PhotoDNA that I have privately circulated to NCMEC, ICMEC (NCMEC’s international counterpart), a few ICACs, a few tech vendors, and Microsoft. The few who provided feedback were very concerned about PhotoDNA’s limitations that the paper calls out. I have not made my whitepaper public because it describes how to reverse the algorithm (including pseudocode). If someone were to release code that reverses NCMEC hashes into pictures, then everyone in possession of NCMEC’s PhotoDNA hashes would be in possession of child pornography.

The AI perceptual hash solution

With perceptual hashes, the algorithm identifies known image attributes. The AI solution is similar, but rather than knowing the attributes a priori, an AI system is used to “learn” the attributes. For example, many years ago there was a Chinese researcher who was using AI to identify poses. (There are some poses that are common in porn, but uncommon in non-porn.) These poses became the attributes. (I never did hear whether his system worked.)

The problem with AI is that you don’t know what attributes it finds important. Back in college, some of my friends were trying to teach an AI system to identify male or female from face photos. The main thing it learned? Men have facial hair and women have long hair. It determined that a woman with a fuzzy lip must be “male” and a guy with long hair is female.

Apple says that their CSAM solution uses an AI perceptual hash called a NeuralHash. They include a technical paper and some technical reviews that claim that the software works as advertised. However, I have some serious concerns here:

The reviewers include cryptography experts (I have no concerns about the cryptography) and a little bit of image analysis. However, none of the reviewers have backgrounds in privacy. Also, although they made statements about the legality, they are not legal experts (and they missed some glaring legal issues; see my next section).
Apple’s technical whitepaper is overly technical — and yet doesn’t give enough information for someone to confirm the implementation. (I cover this type of paper in my blog entry, “Oh Baby, Talk Technical To Me” under “Over-Talk”.) In effect, it is a proof by cumbersome notation. This plays to a common fallacy: if it looks really technical, then it must be really good. Similarly, one of Apple’s reviewers wrote an entire paper full of mathematical symbols and complex variables. (But the paper looks impressive. Remember kids: a mathematical proof is not the same as a code review.)
Apple claims that there is a “one in one trillion chance per year of incorrectly flagging a given account”. I’m calling bullshit on this.

Facebook is one of the biggest social media services. Back in 2013, they were receiving 350 million pictures per day. However, Facebook hasn’t released any more recent numbers, so I can only try to estimate. In 2020, FotoForensics received 931,466 pictures and submitted 523 reports to NCMEC; that’s 0.056%. During the same year, Facebook submitted 20,307,216 reports to NCMEC. If we assume that Facebook is reporting at the same rate as me, then that means Facebook received about 36 billion pictures in 2020. At that rate, it would take them about 30 years to receive 1 trillion pictures.

According to all of the reports I’ve seen, Facebook has more accessible photos than Apple. Remember: Apple says that they do not have access to users’ photos on iCloud, so I do not believe that they have access to 1 trillion pictures for testing. So where else could they get 1 trillion pictures?

Randomly generated: Testing against randomly generated pictures is not realistic compared to photos by people.
Videos: Testing against frames from videos means lots of bias from visual similarity.
Web crawling: Scraping the web would work, but my web logs rarely show Apple’s bots doing scrapes. If they are doing this, then they are not harvesting at a fast enough rate to account for a trillion pictures.
Partnership: They could have some kind of partnership that provides the pictures. However, I haven’t seen any such announcements. And the cost for such a large license would probably show up in their annual shareholder’s report. (But I haven’t seen any disclosure like this.)
NCMEC: In NCMEC’s 2020 summary report, they state that they received 65.4 million files in 2020. NCMEC was founded in 1984. If we assume that they received the same number of files every year (a gross over-estimate), then that means they have around 2.5 billion files. I do not think that NCMEC has 1 trillion examples to share with Apple.

Perhaps Apple is basing their “1 in 1 trillion” estimate on the number of bits in their hash?

With cryptographic hashes (MD5, SHA1, etc.), we can use the number of bits to identify the likelihood of a collision. If the odds are “1 in 1 trillion”, then it means the algorithm has about 40 bits for the hash. However, counting the bit size for a hash does not work with perceptual hashes.
With perceptual hashes, the real question is how often do those specific attributes appear in a photo. This isn’t the same as looking at the number of bits in the hash. (Two different pictures of cars will have different perceptual hashes. Two different pictures of similar dogs taken at similar angles will have similar hashes. And two different pictures of white walls will be almost identical.)
With AI-driven perceptual hashes, including algorithms like Apple’s NeuralHash, you don’t even know the attributes so you cannot directly test the likelihood. The only real solution is to test by passing through a large number of visually different images. But as I mentioned, I don’t think Apple has access to 1 trillion pictures.

What is the real error rate? We don’t know. Apple doesn’t seem to know. And since they don’t know, they appear to have just thrown out a really big number. As far as I can tell, Apple’s claim of “1 in 1 trillion” is a baseless estimate. In this regard, Apple has provided misleading support for their algorithm and misleading accuracy rates.

The AI interpretation solution

An AI-driven interpretation solution tries to use AI to learn contextual elements. Person, dog, adult, child, clothing, etc. While AI systems have come a long way with identification, the technology is nowhere near good enough to identify pictures of CSAM. There are also the extreme resource requirements. If a contextual interpretative CSAM scanner ran on your iPhone, then the battery life would dramatically drop. I suspect that a charged battery would only last a few hours.

Luckily, Apple isn’t doing this type of solution. Apple is focusing on the AI-driven perceptual hash solution.

Problem #2: Legal

Since Apple’s initial CSAM announcement, I’ve seen lots of articles that focus on Apple scanning your files or accessing content on your encrypted device. Personally, this doesn’t bother me. You have anti-virus (AV) tools that scan your device when your drive is unlocked, and you have file index systems that inventory all of your content. When you search for a file on your device, it accesses the pre-computed file index. (See Apple’s Spotlight and Microsoft’s Cortana.)

You could argue that you, as the user, have a choice about which AV to use, while Apple isn’t giving you a choice. However, Microsoft ships with Defender. (Good luck trying to disable it; it turns on after each update.) Similarly, my Android ships with McAfee. (I can’t figure out how to turn it off!)

The thing that I find bothersome about Apple’s solution is what they do after they find suspicious content. With indexing services, the index stays on the device. With AV systems, potential malware is isolated — but stays on the device. But with CSAM? Apple says:

Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images. Apple then manually reviews each report to confirm there is a match, disables the user’s account, and sends a report to NCMEC.

In order to manually review the match, they must have access to the content. This means that the content must be transferred to Apple. Moreover, as one of Apple’s tech reviewers wrote, “Users get no direct feedback from the system and therefore cannot directly learn if any of their photos match the CSAM database.” This leads to two big problems: illegal searches and illegal collection of child exploitation material.

Illegal Searches

As noted, Apple says that they will scan your Apple device for CSAM material. If they find something that they think matches, then they will send it to Apple. The problem is that you don’t know which pictures will be sent to Apple. You could have corporate confidential information and Apple may quietly take a copy of it. You could be working with the legal authority to investigate a child exploitation case, and Apple will quietly take a copy of the evidence.

To reiterate: scanning your device is not a privacy risk, but copying files from your device without any notice is definitely a privacy issue.

Think of it this way: Your landlord owns your property, but in the United States, he cannot enter any time he wants. In order to enter, the landlord must have permission, give prior notice, or have cause. Any other reason is trespassing. Moreover, if the landlord takes anything, then it’s theft. Apple’s license agreement says that they own the operating system, but that doesn’t give them permission to search whenever they want or to take content.

Illegal Data Collection

The laws related to CSAM are very explicit. 18 U.S. Code § 2252 states that knowingly transferring CSAM material is a felony. (The only exception, in 2258A, is when it is reported to NCMEC.) In this case, Apple has a very strong reason to believe they are transferring CSAM material, and they are sending it to Apple — not NCMEC.

It does not matter that Apple will then check it and forward it to NCMEC. 18 U.S.C. § 2258A is specific: the data can only be sent to NCMEC. (With 2258A, it is illegal for a service provider to turn over CP photos to the police or the FBI; you can only send it to NCMEC. Then NCMEC will contact the police or FBI.) What Apple has detailed is the intentional distribution (to Apple), collection (at Apple), and access (viewing at Apple) of material that they strongly have reason to believe is CSAM. As it was explained to me by my attorney, that is a felony.

At FotoForensics, we have a simple process:

People choose to upload pictures. We don’t harvest pictures from your device.
When my admins review the uploaded content, we do not expect to see CP or CSAM. We are not “knowingly” seeing it since it makes up less than 0.06% of the uploads. Moreover, our review catalogs lots of types of pictures for various research projects. CP is not one of the research projects. We do not intentionally look for CP.
When we see CP/CSAM, we immediately report it to NCMEC, and only to NCMEC.

We follow the law. What Apple is proposing does not follow the law.

The Backlash

In the hours and days since Apple made its announcement, there has been a lot of media coverage and feedback from the tech community — and much of it is negative. A few examples:

BBC: “Apple criticised for system that detects child abuse”
Ars Technica: “Apple explains how iPhones will scan photos for child-sexual-abuse images”
EFF: “Apple’s Plan to ‘Think Different’ About Encryption Opens a Backdoor to Your Private Life”
The Verge: “WhatsApp lead and other tech experts fire back at Apple’s Child Safety plan”

This was followed by a memo leak, allegedly from NCMEC to Apple:

I understand the problems related to CSAM, CP, and child exploitation. I’ve spoken at conferences on this topic. I am a mandatory reporter; I’ve submitted more reports to NCMEC than Apple, Digital Ocean, Ebay, Grindr, and the Internet Archive. (It isn’t that my service receives more of it; it’s that we’re more vigilant at detecting and reporting it.) I’m no fan of CP. While I would welcome a better solution, I believe that Apple’s solution is too invasive and violates both the letter and the intent of the law. If Apple and NCMEC view me as one of the “screeching voices of the minority”, then they are not listening.

Update 2021-08-09: In response to widespread criticism, Apple quickly released an FAQ. This FAQ contradicts their original announcement, contradicts itself, contains doublespeak, and omits important details. For example:

The FAQ says that they don’t access Messages, but also says that they filter Messages and blur images. (How can they know what to filter without accessing the content?)
The FAQ says that they won’t scan all photos for CSAM; only the photos for iCloud. However, Apple does not mention that the default configuration uses iCloud for all photo backups.
The FAQ say that there will be no falsely identified reports to NCMEC because Apple will have people conduct manual reviews. As if people never make mistakes.

This is far from the complete list of issues with their FAQ. It does not resolve any of the concerns raised in this blog entry.