Two fathers from two US states recently went under police investigation over “sexual abuse.” The reason: Google Photos’ algorithm flagged photos of their children, closed their accounts, and automatically reported them to authorities. The thing is: none of the dads were involved in such despicable crimes. Both cases took place in the middle of COVID-19 lockdowns. The parents photographed their children in order to send the photos to the doctors, but Google couldn’t tell the context which caused them a lot of trouble.
Both cases happened in 2021 when the entire world was in lockdown. One dad from San Francisco and the other from Houston both tried to get diagnoses for their children from pediatricians. What got Google alarmed is that, in both cases, the photos were of the children’s genitals.
The San Francisco case
The San Francisco case
When a man named Mark noticed swelling on his toddler’s groin, his wife scheduled an emergency video consultation with a doctor for the following morning. Because, you know, the pandemic. According to The New York Times, the scheduling nurse told them to send photos of the swelling so that the doctor could see it before the call. Mark took photos with his phone, the wife texted them to herself, and then uploaded them to the medical provider’s message system. It all went well: the doctor determined what caused the swelling, prescribed antibiotics, and the kid got well. But shortly after that, Google notified Mark that his account had been disabled for “harmful content” that was in “severe violation of Google’s policies and might be illegal.” Naturally, he filed an appeal, but Google rejected it. In fact, they even reported the photos to the National Center for Missing and Exploited Children’s CyberTipline, who further reported Mark to the police. Ten months later, the San Francisco Police Department notified Mark that they’d investigated him based on the photos and Google’s report. They had a search warrant to access everything in Mark’s Google account: messages, photos, videos, internet searches, and location. Eventually, they concluded that Mark didn’t commit the crime he was accused of, and he used the police report to try and reactivate his account. However, Google denied his request once again.
The Houston case
The Houston case
In the case of the Houston dad, it got even more complicated. The course of the events was similar: the doctor asked the father to take photos of his son’s genitals so they could diagnose an infection. Since automatic backup was on, the images were automatically uploaded to Google Photos and sent from the father to his wife via Google messenger. Once again, Google flagged the photos, filed a report, and the couple was investigated by the police who concluded that no crime was commited. What makes things even worse is that this man didn’t only lose all his emails and a bunch of data on his Google accounts. He and his wife were reportedly in the middle of buying a new home. Since the photos had the man’s email address disabled, they faced additional complications. Speaking with Gizmodo via email, a Google spokesperson said the following: “Child sexual abuse material (CSAM) is abhorrent and we’re committed to preventing the spread of it on our platforms. We follow US law in defining what constitutes CSAM and use a combination of hash matching technology and artificial intelligence to identify it and remove it from our platforms. Additionally, our team of child safety experts reviews flagged content for accuracy and consults with pediatricians to help ensure we’re able to identify instances where users may be seeking medical advice. Users have the ability to appeal any decision, our team reviews each appeal and we will reinstate an account if an error has been made.” As Gizmodo reports, the police concluded in both cases that the parents had committed no crime. However, Google still stuck with its decision and permanently deactivated the men’s accounts across all its platforms despite the appeals. Frankly, you can’t blame AI for flagging the photos, as they for once actually contain what they were flagged for. In fact, in cases like this, even human screening can go wrong if the context is missing. Still, if the police concluded that both fathers are innocent. The men used the police report as a proof of innocence and appealed Google’s decision, so I don’t see why their accounts remain suspended. [via Gizmodo]
