Apple defends anti-child abuse imagery tech after claims of ‘hash collisions’

> The goal should be to create a system where you couldn’t do so even of you wanted to (or the state demanded it).

There’s this bizarre notion that using end-to-end encryption can absolve you of responsibility, that the authorities will have to accept an answer of “we literally can’t access it”.

That’s just not the case for centralised things: you’re deliberately facilitating some service, government will find you liable for some things in its operation, and if you don’t comply, they’ll fine or shut you down. E2EE doesn’t absolve you from law; law is all about saying you’re not allowed to do things that are physically possible.

(Decentralised things, now they can be banned but not truly stopped because there’s no central party to shut down.)

> The goal should be to create a system where you couldn’t do so even of you wanted to (or the state demanded it).

You can still do secure backups of your phone without using iCloud, but there isn’t a way for Apple to do end to end encryption of backups transparently like you can with real time communication. The only way end to end encryption of backups works is to require people keep a separate secure key(s) to avoid losing their data, which means a universal implementation has real direct risk for users.

As long as Apple has access to these files the FBI can legally require them to do these searches. From a pure PR perspective they should have communicated what was already going on before releasing this system because people assume something significant changed.

> defend their favorite trillion dollar corporations

> is a short slippery slope away

Obviously people who trust Apple aren't concerned about slippery slopes. What's the point of your post?

That is the problem. CSAM is just smoke and mirrors paired with appeal to emotions to win approval more easily. I don't want anyone, neither Apple, nor Microsoft, Google and others to sneak into my files. Did anyone realize that in the 21st century our cellphone is essentially our wallet?

Apple controls the hardware, software, and cloud service. It was always a pinky promise that they wouldn't look at your files. I don't know why we should doubt that pinky promise less today than we did a month ago.

The only thing Apple scans are files you upload to iCloud Photos.

If you turn off iCloud Photos, nothing is scanned.

Microsoft scans everything.

>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)

https://www.nbcnews.com/technolog/your-cloud-drive-really-pr...

That's the point: catching the bad guy foolish enough to keep known CSAM images on their phone, while not technically invading the privacy of any good guys. Anyway, if apple wanted to covertly invade their users' privacy, they'd have no technical problems to do so.

What it takes to accept is the "nothing to hide" mentality: your files are safe to scan (locally) because they can't be known CSAM files. You have to trust the scanner. You allow the scanner to touch your sensitive files because you're not the bad guy, and you want the bad guy be caught (or at least forced off the platform).

And this is, to my mind, the part Apple wasn't very successful at communicating. The whole thing should have started with an educational campaign well ahead of time. The privacy advantage should have been explained again and again: "unlike every other vendor, we won't siphon your files unecrypted for checking; we do everything locally and are unable to compromise your sensitive bits". Getting one's files scanned should have become a badge of honor among the users.

But, for some reason, they tried to do it in a low-key and somehow hasty manner.

Presumably the system is novel enough that a very similar proposal which lacked one important insight (using perceptual hashes to turn image similarity problems into set intersection problems) was accepted to USENIX this year: https://www.usenix.org/conference/usenixsecurity21/presentat...

With the new system, it still arrives on the server, and they can't even know about it until some threshold has been passed - and then the servers still have to scan it.

So it's not even accomplishing "keeping it off the servers."

That's what they're saying, the key difference is Apples happens on your device, not theirs.

> people incompetent enough to have years old (otherwise how would it end up at NCMEC?) CSAM on their iPhones

Who do you think has 30 or more CSAM images on their phone?

Only the safety tokens are generated on your device, the action triggering scan happens in the cloud (just like all the others) and then it goes to human review, so if it's a hash collision it'd be caught there when they review the images, and they can only review the images that matched CSAM.

I honestly can't find the uproar here. Google devices can face match photos offline... so they are applying a neural net (scanning) ON THE DEVICE! How is that not worse than what apple do?

ON YOUR DEVICE (where it's encrypted in a way that Apple can't read until the 30 image threshold is crossed) is more private than doing the same scan on server where a single false positive can be misused by anyone who can get a subpoena.

When the scanning gets moved from the cloud to being on device, Apple itself cannot see the results of the scan until the risk that the result is only a false positive is greatly reduced.

You would have to have 30 false positives before Apple can see anything, which is unlikely, but the next step is still a human review, since it's not impossible.

Conducting the scan on the user's device instead of on the companies server is more private.

Apple can't decrypt the results of the scan until the ~30 image threshold is crossed and a human review is triggered.

Given Google's reluctance to hire humans when a poorly performing algorithm is cheaper, are they turning over every single false positive without a human review?

>a man [was] arrested on child pornography charges, after Google tipped off authorities about illegal images found in the Houston suspect's Gmail account

https://techcrunch.com/2014/08/06/why-the-gmail-scan-that-le...

You don't consider the contents of your email account or the files you mirror to a cloud drive to be your own private data?

At least Google does.

https://protectingchildren.google/intl/en/

> CSAI Match is our proprietary technology, developed by the YouTube team, for combating child sexual abuse imagery (CSAI) in video content online. It was the first technology to use hash-matching to identify known violative videos and allows us to identify this type of violative content amid a high volume of non-violative video content. When a match of violative content is found, it is then flagged to partners to responsibly report in accordance to local laws and regulations. Through YouTube, we make CSAI Match available for free to NGOs and industry partners like Adobe, Reddit, and Tumblr, who use it to counter the spread of online child exploitation videos on their platforms as well.

> We devote significant resources—technology, people, and time—to detecting, deterring, removing, and reporting child sexual exploitation content and behavior. Since 2008, we’ve used “hashing” technology, which creates a unique digital ID for each known child sexual abuse image, to identify copies of images on our services that may exist elsewhere.

No misconstrual needed. This technology is genuinely bad. It scans images against an arbitrary government-owned black-box database. There’s no guarantee that it’s only CSAM.

It's important to keep nefarious stuff on the server side because eventually someone will reverse engineer what's on the client side.

Imagine if Apple had done this on the client side without telling anyone, and later it was discovered. I think things would be a whole worse for Apple in that case.

As has been repeated over and over, apple only scans photos that are part if icloud photos (ie, uploaded).

Don't want your photo's scanned, don't sync them to icloud. Seriously! Please include the actual system when discussing this system, not your bogeyman system.

"To help address this, new technology in iOS and iPadOS* will allow Apple to detect known CSAM images stored in iCloud Photos."

To increase privacy - they perform the scan on device prior to upload.

> B) They scan everything that exists on my device

No ... They scan everything that I have released to apple photos that exists on my device.

Same scan - different place.

“IMHO” means “in my humble opinion”

Also have to agree: I don't see how this could originate from engineers. Every engineer I've spoken with at the company I work for has been mortified by this insanity.

What abuses apply to CSAM scanning in this hybrid pipeline which don’t apply to iCloud Backup? If they scanned on the server, why couldn’t governments “slippery slope” abuse the system by requiring that all files on iOS end up in iCloud Backup, where they can be scanned by the system?

Since the announcement, I can think of a dozen ways Apple could be easily forced into scanning all the contents of your device by assembling features they’ve already shipped. Yet they haven’t. At some point, people need to produce evidence that Apple cannot hold the line they’ve said they will.

Privacy means “you pay for the device so we don’t sell your attention to advertisers.” That’s it. There’s no protection against state level actors (Pegasus, CSAM scanning, etc.).

If your threat model includes being the target of someone who will plant child pornography on your phone, you are already fucked. And no, Apple isn’t suddenly going to scan Chinese iPhones for Winnie the Pooh memes. They don’t have to. China already has the 50 cent party to do that for them, on WeChat.

Basically everything everyone seems to think is just around the corner has already been possible for years.

This is a great answer but that’s not actually the GP’s contention. Their argument is essentially “so what if there’s a collision, the human review will catch it”. And to that I’d say that the same is supposed to occur for the no-fly list and we all know how that works in practice.

The mere accusal itself of possessing CSAM can be life ruining if it gets to that stage. More importantly, a collision will effectively allow warrantless searches, at least of the collided images.

Those are two almost identical images based off each other.

I know that that "think of the children" is a meme, but I think this illustrates the point for apple. If you have a croped or modified image of CSA the system will identify it. As long as your image is different enough from CSA, you are safe.

The point here is that Apple is specifically looking for matches against known CSA material.

If someone can demonstrate that a legal NSFW image (eg. regular old-fashioned pornography), can be collided with a legal, completely 100% SFW image then I'll be concerned.

But until then, this looks like a reasonable and supportable way for finding CSAM in real time.

Doing so would create a positive that still doesn't pass Apple's human visual check against the (blurred) CSAM content associated with that checksum, and if it somehow did, it would still then also have to occur at qty.30 or more, and they'd have to pass a human visual check against the (unblurred) CSAM content by one of the agencies in possession of it. It's not possible to spoof that final test unless you possess real CSAM content, at which point you don't need to spoof that final test and you'll end up arrested for possession.

This. Also try contacting a lawyer who knows this area and asking to pay for a legal opinion brief so that you can post it online to be debated by legions of software developers.

Lawyers I know would politely decline that.

Right. That's why there's never been an amicus brief or a friends of the court type of document drafted and signed by lawyers.

Sure - but if you are going to write blog posts / articles - and all you can say is based on the lawyers I talked to apple is committing child porn felonies, that is just unacceptable.

At least HN should flag these and get these taken down. Over and over the legal analysis is either trash or it's clear the article author didn't understand something (so how can lawyer give good advice?).

These conversations become so uninteresting when people take these extreme type positions. Apple's brand is destroyed - apple is committing child porn felonies.

I would have rather just had a link to the apple technical paper and a discussion personally vs the over the top random article feed with all sorts of misunderstandings.

And in contract law there are LOTS of legal articles online - with folks name on them! They are useful! I read them and enjoy them. Can we ask for that here where it matters maybe more?

From what I understand it's even more cut and dry than that. If you submit a report to the NCMEC you are generally required to (securely) keep a copy of the image(s) being reported and any related info for a period of time. That's part of the rules. You are also only compelled to report things that you know are bad, so verification before reporting makes sense. The idea that they would be charged with a crime for doing what is essentially required by law is flat out wrong. Unless they are storing the material insecurely once confirmed bad or otherwise mishandling the process, they seem to be following the law as I understand it. IANAL but I have an account with the NCMEC for a service I run, so I have looked through their documentation and relevant laws to try to understand the requirements placed on me as a service provider.

The problem isn't the idea of CASM, it's that what happens when they start scanning for "misinformation", or whatever else they want to come up with at that point.

PhotoDNA and NeuralHash are not the same thing.

Except when you're not measuring the similarity you think you are, because NeuralNets are f&#@ing weird.

You know, let me put it this way. Yiu know that one really weird family member I'm pretty sure everyone either has or is?

Guess what? They're a neural net too.

This is what Apple is asking you to trust.

Take 2 CSAM pictures, known. Combine into one image.

Swing and a miss. Not in the CSAM dataset. Take two images. Encode alternating pixels. Decode to get the original image back. Convert to different encodings print to PDF or Postscript. Encode as base64 representations of the image file...

Who are we trying to fool here? This is kiddie stuff.

This is more about trying to implant scanning capabilities on client devices.

This is new for Apple's customers. And its new in that the device you bought and paid for is nosing through your files.

Apple is introducing a reverse 'Little Snitch' where instead of the app warning you what apps are doing on the network, the OS is scanning your photos. Introducing a 5th columnist into a device that you've bought and paid for is a huge philosophical jump from Apple's previous stances on privacy, where they'd gone as far as fighting the Feds about trying to break into terrorist's iPhones.

You’re not understanding. Dropbox already does scan your files and I’m comparing Apples actions explicitly to that fact. I’m pointing out that people don’t care about that because you’re literally, voluntarily, giving Dropbox your files. Here, Apple is controlling your phone to tattle on you at an OS level (yes, I know, Apple says you need to turn on iCloud Photos for this to run but the precedent is the problem).

Open Source fanatics are the worst. Even if all the software on your phone was Open Source, you wouldn’t have the mental bandwidth to ever verify you trust it. You’re just farming out your trust to a different set of people and hoping they’re more trustworthy than those with closed source. At best this MAY be reasonable because of incentive alignment but that’s super weak.

Not to mention the meaning of ownership is completely unrelated to ability to view the designs of a given object. I think I own the fan currently blowing air at me without ever having seen a schematic for its controls circuitry just fine and everyone for all of history has pretty much felt the same.

There are two different "features" being implemented - Messages scanning for minors on a family plan (which can be turned off) and iCloud Photo scanning (which can't be turned off, as far as I know). So no, you can't just turn it off.

No, probably wouldnt even get to the desk of the police. End of the road would be NCMEC, to whom apple would refer hash matching images that pass human verification that are close enough to porn.

If your 30 or so hash matching images matched their corresponding known CSAM then that goes on to the police and then they knock on your door.

It's clear that EARN IT could literally be revived any day if Apple didn't do something to say "we don't need it because we've already satisfied your requirements."

Alternatively, "Apple has shown that it's possible to do without undue hardship, so we should make everyone else do it too".