Law enforcement agencies are now leveraging AI to create images of fictitious children, aiding in the capture of online child predators. This tactic was highlighted in a recent lawsuit filed by the state of New Mexico against Snapchat.

The lawsuit reveals that the New Mexico Department of Justice initiated an undercover operation to demonstrate that Snapchat is a major platform for sharing child sexual abuse material (CSAM) and for the sextortion of minors. The complaint alleges that Snapchat's algorithm exposes children to adult predators.

As part of the investigation, an officer created a decoy account for a 14-year-old girl named "Sexy14Heather." Despite the profile being set to private and not adding any followers, "Heather" was quickly recommended to dangerous accounts, including those with explicit names, according to a press release from the New Mexico DOJ.

After accepting a follow request from one account, the recommendations worsened, with Snapchat suggesting over 91 users, many of whom sought to exchange sexually explicit content. The complaint claims that Snapchat is a breeding ground for predators to collect explicit images of children and to groom and extort them.

The investigator, posing as "Sexy14Heather," engaged in conversations with adult users who sent inappropriate messages and explicit photos. In one instance, a user named "50+ SNGL DAD 4 YNGR" sent sexually suggestive messages after "Heather" mentioned her age and shared a photo. Other accounts attempted to coerce "Heather" into sharing explicit content.

The investigation also found that Snapchat's search tool recommended users associated with trading CSAM, even when no explicit language was used. This suggests that the algorithm identified the decoy as seeking CSAM.

This innovative use of AI follows the indictment of Alejandro Marquez, who was sentenced to 18 years for raping an 11-year-old girl he met through Snapchat's Quick Add feature. More recently, Jeremy Guthrie was arrested and sentenced for raping a 12-year-old girl he met on Snapchat.

Historically, police have used photos of younger-looking adults or younger images of officers to catch predators. AI-generated images are seen as a more ethical alternative, according to sex crimes lawyer Carrie Goldberg, as they avoid using real children's images. However, Goldberg warns that AI could complicate investigations and raise ethical concerns, especially if AI-generated CSAM is created.

Experts caution against training AI on datasets that combine real children's images with explicit content to prevent the creation of harmful AI-generated CSAM. The New Mexico complaint includes only one AI-generated image, leaving questions about the extent of AI use and the ethical considerations involved.

The New Mexico DOJ did not respond to requests for comment. Goldberg emphasizes the need for standards in law enforcement's use of AI to avoid manipulative practices that could lead to entrapment defenses.

The lawsuit also questions Snapchat's liability for connecting children with predators. New Mexico Attorney General Raúl Torrez stated that the lawsuit aims to protect children from sextortion and exploitation. The complaint alleges that Snapchat targets children, addicting them with features while exposing them to harmful content and predators.

The New Mexico DOJ's investigation found Snapchat to be a significant source of non-consensual photos and videos on dark web markets. The lawsuit claims that Snapchat's design features foster the sharing of illicit material and facilitate sextortion and trafficking.

New Mexico seeks an injunction to stop Snapchat from practices that harm children, including advertising the platform as "more private" or "less permanent." The complaint notes that the FBI has identified Snapchat as a preferred app for criminals due to its design features.