Deepfake Fraud and Synthetic Sexual Harm on the Rise: AI Incident Roundup

This AI Incident Roundup highlights the growing prevalence of deepfake-enabled fraud and synthetic sexual harm. The report covers incidents added to the AI Incident Database between November 2025 and January 2026, revealing a disturbing trend of impersonation-for-profit scams. These scams often involve public figures endorsing products or platforms, leading victims into money transfer funnels. The report also notes a rise in health-adjacent deception, such as deepfake "doctor" endorsements. Synthetic sexual harm is another significant concern, with incidents involving minors and the commercialization of harmful content on platforms. The report emphasizes that these images feed a downstream ecosystem of social humiliation and coercion. Furthermore, the report points out that institutions, such as government agencies, are inadvertently amplifying the harm chain. The increasing sophistication and accessibility of deepfake technology make it challenging to detect and prevent these incidents. The report underscores the need for proactive measures, including increased awareness, improved detection technologies, and collaboration between platforms, law enforcement, and researchers. The permanent nature of online distribution exacerbates the harm caused by synthetic content, highlighting the urgency of addressing these issues. The report serves as a stark reminder of the ethical and societal challenges posed by AI-generated content and the need for robust safeguards to protect individuals and institutions from harm. The incidents underscore the importance of media literacy and critical thinking skills in navigating the increasingly complex digital landscape.