Hash collision in Apple NeuralHash model

Of course, this is assuming everything works as intended and they don't find anything else they can use to charge you with something as they search your home. If you smoke cannabis while being in the wrong state, you're now in several more kinds of trouble.

You could take actual CSAM, check if it matches the hashes and keep modifying the material until it doesn’t (adding borders, watermarking, changing dimensions etc.). Then just save it as usual without any risk.

And because of this they calibrated their communication completely wrong, focusing on the on device part as being more private. Using the same line of thinking they use for putting Siri on device.

And the follow up was an uncoordinated mess that didn't help either (as you rightly pointed out with Craig's interview). In the Neuenschwander interview [1], he stated this :

> The hash list is built into the operating system, we have one global operating system and don’t have the ability to target updates to individual users and so hash lists will be shared by all users when the system is enabled.

This still has me confused, here's my understanding so far (please feel free to correct me)

- Apple is shipping a neural network trained on the dataset that generates NeuralHashes

- Apple also ships (where ?) a "blinded" (by an eliptic curve algo) table lookup that match (all possible?!) NeuralHashes to a key

- This key is used to encrypt the NeuralHash and the derivative image (that would be used by the manual review) and this bundle is called the voucher

- A final check is done on server using the secret used to generate the elliptic curve to reverse the NeuralHash and check it server side against the known database

- If 30 or more are detected, decrypt all vouchers and send the derivative images to manual review.

I think I'm missing something regarding the blinded table as I don't see what it brings to the table in that scenario, apart from adding a complex key generation for the vouchers. If that table only contained the NeuralHashes of known CSAM images as keys, that would be as good as giving the list to people knowing the model is easily extracted. And if it's not a table lookup but just a cryptographic function, I don't see where the blinded table is coming from in Apple's documentation [2].

Assuming above assumptions are correct, I'm paradoxically feeling a tiny bit better about that system on a technical level (I still think doing anything client side is a very bad precedent), but what a mess did they put themselves into.

Had they done this purely server side (and to be frank there's not much difference, the significant part seems to be done server side) this would have been a complete non-event.

[1] : https://daringfireball.net/linked/2021/08/11/panzarino-neuen...

[2] This is my understanding based on the repository and what's written page 6-7 : https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

Just insert a known CSAM image on target's device. Done.

I presume this could be used against a rival political party to ruin their reputation - insert bunch of CSAM images on their devices. "Party X is revealed as an abuse ring". This goes oh-so-very-nicely with Qanon conspiracy theories which even don't require any evidence to propagate widely.

Wait for Apple to find the images. When police investigation is opened, make it very public. Start a social media campaign at the same time.

It's enough to fabricate evidence only for a while - the public perception of the individual or the group will be perpetually altered, even though it would surface later that the CSAM material was inserted by hostile third party.

You have to think about what nation state entities that are now clients of Pegasus and so on could do with this. Not how safe the individual component is.

1. Obtain known CSAM that is likely in the database and generate its NeuralHash.

2. Use an image-scaling attack [2] together with adversarial collisions to generate a perturbed image such that its NeuralHash is in the database and its image derivative looks like CSAM.

A difference compared to server-side CSAM detection could be that they verify the entire image, and not just the image derivative, before notifying the authorities.

[1] https://news.ycombinator.com/item?id=28218922

[2] https://bdtechtalks.com/2020/08/03/machine-learning-adversar...

This part IMO makes Apple itself the most likely "target", but for a different kind of attack.

Just wait until someone who wasn't supposed to, somewhere, somehow gets their hands on some of the actual hashes (IMO bound to happen eventually). Also remember that with Apple, we now have an oracle that can tell us. And with all the media attention around the issue, this might further incentivize people to try.

From that I can picture a chain of events something like this:

1. Somebody writes a script that generates pre-image collisions like in the post, but for actual hashes Apple uses.

2. The script ends up on the Internet. News reporting picks it up and it spreads around a little. This also means trolls get their hands on it.

3. Tons of colliding image are created by people all over the planet and sent around to even more people. Not for targeted attacks, but simply for the lulz.

4. Newer scripts show up eventually, e.g. for perturbing existing images or similar stunts. More news reporting follows, accelerating the effect and possibly also spreading perturbed images around themselves. Perturbed images (cat pictures, animated gifs, etc...) get uploaded to places like 9gag, reaching large audiences.

5. Repeat steps 1-4 until the Internet and the news grow bored with it.

During that entire process, potentially each of those images that ends up on an iDevice will have to be manually reviewed...

A better collision won't be a grey blob, it'll take some photoshopped and downscaled picture of a kid and massage the least significant bits until it is a collision.

https://openai.com/blog/adversarial-example-research/

Just because the PoC used a meaningless blob doesn't mean that collisions have to be those. Plenty of examples of adversarial attacks on image recognition perturb real images to get the network to misidentify them, but to a human eye the image is unchanged.

Given the scanning is client-side wouldn't the client need a list of those hashes to check against? If so it's just a matter of time before those are extracted and used in these attacks.

It's my understanding that many tech companies (Microsoft? Dropbox? Google? Apple? Other?) (and many people in those companies) have access to the CSAM database, which essentially makes it public.

Is the process actually documented anywhere? Afaik they are just saying that they are verifying a match. This could of course just be a person looking at the hash itself.

(Added later:) I should note that the DoS attack is only possible with the preimage attack and not the second preimage attack as the issue seemingly suggests, because you need the original CSAM to perform the second preimage attack. But given the second preimage attack is this easy, I don't have any hope for the preimage resistance anyway.

(Added much later:) And I realized that Apple did think of this possibility and only stores blinded hashes in the device, so the preimage attack doesn't really work as is. But it seems that the hash output is only 96 bits long according to the repository, so this attack might still be possible albeit with much higher computational cost.

[1] To be fair, I don't think that Apple's claim of 1/1,000,000,000,000 false positive rate refers to that of the algorithm. Apple probably tweaked the threshold for manual checking to match that target rate, knowing NeuralHash's false positive rate under the normal circumstances. Of course we know that there is no such thing like the normal circumstances.

It is, actually. Remember that hashes are supposed to be many-bit digests of the original; it should take O(2^256) work to find a message with a chosen 256-bit hash and O(2^128) work to find a "birthday attack" collision. Finding any collision at all with NeuralHash so soon after its release is very surprising, suggesting the algorithm is not very strong.

SHAttered is a big deal because it is a fully working attack model, but the writing was on the wall for SHA-1 after the collisions were found in reduced-round variations of the hash. Attacks against an algorithm only get better with time, never worse.

Moreover, the break of NeuralHash may be even stronger than the SHAttered attack. The latter modifies two documents to produce a collision, but the NeuralHash collision here may be a preimage attack. It's not clear if the attacker crafted both images to produce the collision or just the second one.

Never? You sure that one or more human operators will never make this mistake, dooming someone's life / causing them immense pain?

Strongly disagree. (1) The primary feature of any decent hash function is that this should not happen. (2) Any preimage attack opens the way for further manipulations like you describe.

A devastating scenario for such a system is if an attacker knows how to look at a hash and generate some image that matches the hash, allowing them to trigger false positives any time. That appears to be what we are witnessing.

On their cloud services (Apple, Google, MS, etc) they can only get to content that a user has implicitly agreed to share (whether or not they understand the ramifications is another conversation). On your device, there's no more separation.

Ignoring the whataboutism in your question, we know that privacy once lost is practically impossible to get back. Once the genie is out of the bottle and Apple is doing on device scanning, what's to stop 3 letter agencies and governments around the world to start demanding ever more access? Because that's exactly what's going to happen. "Oh it's not a big deal, we just need slightly more access than we have now."

10 years down the line, we'd have people's phones datamining every piece of info they have and silently reporting to an unknowable set of entities. All in the name of fighting crime.

There needs to be pushback on this crap. Every single one of these attempts must be met with absolute and unconditional refusal. Mere inaction means things will inevitably and invariably get worse over time.

This, combined with human error during the manual review process might result in someone getting reported. Seeing as twitter (and other social media sites) jump on the bandwagon whenever someone gets accused of being a pedophile, this might destroy someones life.

The entire story might seem a bit to far fetched, but based on past events, you never know how bad something 'simple' as a hash collision can be.

For one, you can't know if that's true as the image could have been manipulated to appear as such. For example you wouldn't know if a kind of steganography has been used to hide image in an image and that neuralhash picked on a hidden image.

> but I don't understand how they can get it on someone's phone

There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc...

Hence why this approach was stated as a privacy win by Apple. They catch the CSAM and they don't have to look at your photos stored online.

I’d trust the algorithm more than the human reviewing it , considering the algorithm already shows flaws and loopholes , the human part of it does not bring any confidence.

This isnt facebook , this is your private photos , it shouldnt have reached till this stage , in the first place

Build surveillance to catch those people and I'll listen.

Siri: "You seem to be having unpatriotic thoughts. We are dispatching someone to help you."

https://github.com/KhaosT/nhcalc

The database of CSAM hashes is blinded and no one has the hashes. Without the hashes, this attack is useless.

It's also mitigated by a LOT of checks and balances. First they have to know 30 hashes to target (they're secret). They have to get 30 colliding images on your phone. The images have to be unnoticed by you (why not just infiltrate CSAM, then?) or sufficiently compelling that you don't just delete them. Thirty images have to pass human review at Apple. At least one has to pass human review by law enforcement. Then, and only then, will you be arrested and face a threat.

Short version: If somebody wants to frame you for possessing CSAM, there are much easier ways. There is no new threat here. https://xkcd.com/538/

You can whatsapp someone an innocent image doctored to have a hash collision with known CSAM. If they have default settings it will be saved to their photo reel, scanned by iOS and the police will be called.

Until the arresting police officer explains to them they are being arrested on suspicion of being a paedophile, they won't even know this has happened.

Billionaires at (Apple) don't give a flying f*ck about users privacy. It is all vertical integration in the name of world domination. How removed from reality they are. This is week after Pegasus/NSO and there is no law who requires them to "scan" on device. So the logical explanation is "growth". Way to go Apple.

If they stand ground it makes the decision a lot easier for others. If they dont they will continue to use the boiling frog tactics.

After all they dont think they are wrong and they are still righteous. And It would be far more interesting to see how the world folds. The whole computing market, Google, Apple and Microsoft are currently operating with a moat that is literally impenetrable. This will be a small push towards a new world of possibilities.

Yes. I really hope they stand ground. I wish I won the lottery I could put a few million into the Social Media echo chamber to support this. Or you know, cough certain Apple competitor could do this.

Apple is content with putting in backdoors. Theres collusion behind the scenes here, they know it's bad, but hey, it's apple, it's ran by manipulators and liars and forms it's own cult.

Not to go off topic from your main point, but what did you move to?

This can’t be used for a targeted attack. It’s no different from the previous false posting making this claim.

This can attack that system itself, though, by overloading those humans with too much work looking at random noise, but that requires quite a large organised effort. It also requires getting a hold of actual blacklisted hashes, which I doubt anyone has, unless they have actual child pornography.

Introducing the people asking us to trust them with the responsibility of mass, automated crime accusations.

But this will be evident by reading and understanding the technical description.

I also think that those downvoting you might've applied the principle of charity and taken the best interpretation of what you've written or at least ask.

Even if they were, it wouldn't matter, because NeuralHash is non-cryptographic by design.

I worked with a guy who was convicted of this very crime. Do you know what the FBI installs on his computer and phone? This kind of monitoring.

I am not going to be treated like I am on parole.

Now they can’t even use their phone for storing photos.

The thing with "only when iCloud is enabled" is only for now. It’s trivial to make Scanning all photos default in a future version.