Deleting isn't enough

Trust & Safety teams have a couple of levers, primarily focused around removing content and limiting (or deleting) accounts. A post that violates policy is taken down. An account that keeps violating gets restricted, suspended or banned. Almost everything else in the field, from classifiers to moderation queues to transparency reports, is built to pull those two levers faster and at greater scale. In the EU alone, online platforms filed more than 3.8 billion statements of reasons for moderation decisions in a single six-month window.

Those levers are necessary. But they act on the visible output of harm, not on the people and networks producing it. A groomer who loses an account still has the child's contact details on another app. A radicalized teenager whose post is removed is still in the group chat that radicalized him. A network that coerces children into self-harm loses a channel and opens a new one by evening.

The limits of the removal model

The most serious harms online are not single pieces of content. They are processes that unfold across time and across services.

A child is approached on a game or social app, moved to a private messenger, groomed, and then extorted with the images they were manipulated into sending. A young person drifts from edgy humor into a radicalization funnel that starts on mainstream platforms and ends in encrypted channels where attacks are glorified and planned. Members of networks like 764 befriend vulnerable teenagers and coerce them into producing sexual content, hurting themselves or hurting others. By December 2025 the FBI was running more than 350 investigations tied to 764 and related groups, with victims estimated in the thousands worldwide, some as young as nine. Organized cybercriminals sell malware, stolen data, weapons and drugs through channels and marketplaces that, according to Europol, keep re-emerging after takedowns.

At every stage, a removal-focused system sees a frame, not the movie. It takes down the explicit image, the extremist video, the listing, the threatening message. It rarely sees the recruiter, the victim, the payment route or the next platform. And for the person behind the account, a ban is cheap to get around: a new email address, a new phone number, a VPN.

Content is the evidence of a threat, not the threat itself. Treating it as the main target means we keep cleaning up the evidence while the threat moves on.

Removal treats the symptom. Disruption targets what an operation needs to keep running: its accounts, its infrastructure, its payment routes, its recruiters and, ultimately, its people.

In practice, disruption means mapping the network rather than the post: which accounts recruit, which ones coordinate, which ones receive payment, and which invite links, channels and domains connect them. It means taking linked accounts down together, at the same time, rather than one at a time as each crosses a policy line, so the network can't warn itself and migrate. It means adding friction to the pathways that feed harm, such as unsolicited contact between adults and minors, or the jump from a public community into a private group. And it means preserving evidence, because the end goal of disruption is that someone with the power to arrest can act on it.

The 764 investigations show what that looks like when it works. FBI data shows a roughly 490% rise in arrests of nihilistic violent extremists from 2024 to 2025, and an early 764 member was recently sentenced to 77 years in prison. Those outcomes depended on evidence that, at some point, lived on platforms. Every account that was simply deleted without being preserved or reported is a case that could not be built.

A ban without consequences is a speed bump, not a wall. Disruption is what turns it into a wall, and no single platform can do that alone.

Safeguarding victims, not just removing violations

Enforcement workflows are built around the violating content and account, not the victim. That matters most where the harm is ongoing. Financially motivated sextortion is the clearest example.

Between October 2021 and March 2023, the FBI and Homeland Security Investigations received more than 13,000 reports of financial sextortion of minors, and the FBI has linked the crime to at least 20 suicides. Banning the extorter's account does not tell a 15-year-old that the images won't be posted, that it isn't their fault, or where to get help. Deleting the conversation can even destroy the evidence the victim needs.

The same applies to children drawn into self-harm by coercive networks, to people being targeted with credible threats of violence, and to someone whose post suggests they are at risk of suicide. Meta said it took action on more than 12 million pieces of suicide and self-harm content on Facebook and Instagram in a single quarter of 2024. Each of those removals is also a moment where a person might have needed support.

Safeguarding should be a standard outcome of enforcement, not an optional extra. When a platform identifies a victim, it should notify them in a safe way, connect them to support services and crisis lines, help them prevent their images from spreading (for example, NCMEC's Take It Down is built for exactly that), preserve the evidence, and escalate to child protection or police where the risk calls for it. Success should be measured by whether the person is safer, not by whether the content is gone.

Expanding critical harms beyond immediate threats to life

Today's reporting duties are narrow. In the United States, providers must report apparent CSAM to the National Center for Missing & Exploited Children (NCMEC), and the REPORT Act of 2024 extended that duty to online enticement and child sex trafficking. Since April 2026, UK services under the Online Safety Act must report child sexual exploitation and abuse content to the National Crime Agency. In the EU, Article 18 of the Digital Services Act requires hosting providers to notify police when they suspect a criminal offence involving a threat to someone's life or safety. The DSA contains no list of qualifying offences, and in practice platforms tend to reduce it to one question: is someone in danger right now?

That question leaves out a large category of content that is clearly illegal but does not signal an emergency. Someone posting ISIS propaganda that glorifies an attack. Someone re-uploading the Christchurch mosque shooting video years after the event. A 764 member circulating gore and coercion material and recruiting new targets, before a specific victim can be identified. Someone encouraging a teenager to hurt themselves. A sextortion attempt that is caught before the victim pays or is harmed. Someone selling firearms or drugs through a private channel. In most jurisdictions all of these are crimes. On most platforms, they end the same way: the content is removed, the account may be banned, and the record is purged when the retention period runs out. The crime is not reported; it is simply removed from a platform.

These cases are not marginal. They are often the earliest visible point in the harm lifecycle, which is exactly where intervention is cheapest and most effective. Waiting for a threat to become imminent before involving law enforcement means waiting until the harm is nearly done.

Article 18 of the DSA is the natural vehicle for this in Europe. It already establishes the reporting duty; it needs a scope that goes beyond threats to life or safety, and a list that tells platforms where the line is.

Reporting more to law enforcement, in a way that can be used

The REPORT Act shows what happens when reporting is required. One year before it passed, platforms filed 8,480 CyberTipline reports related to child sex trafficking. In 2025, NCMEC received more than 113,000 trafficking reports in total, along with 1.4 million reports of online enticement and a sharp rise in reports of sadistic online exploitation. The principle I would argue for is simple: once a platform has concluded that something is illegal and falls within a critical harm category, the default should be a report and preserved evidence, not just a deletion.

But more reporting only helps if the receiving end can use it. Stanford researchers found that CyberTipline volume overwhelms law enforcement, that only about half of reports were actionable, and that linked reports were hard to see across different police case-management systems. Outside child safety, the picture is more fragmented still. EU notifications go to each member state separately, and industry has asked the Commission to clarify which authority should even receive them.

That argues for central intake points, such as Europol at EU level alongside national hubs like Germany's BKA notification portal, using a shared reporting schema. If every report carries the same structured fields (harm category, content hashes, account identifiers, timestamps, the platform's assessment of urgency), reports can be deduplicated, prioritized, and linked. Ten reports from ten platforms about the same person are not ten cases. They are one case with ten pieces of evidence. Investigators also need the funding to match, because a reporting duty without capacity on the receiving end just moves the backlog.

Signal sharing and collaboration across platforms

No platform sees the whole chain. The groomer, the extremist recruiter and the self-harm network all rely on that. Signal sharing is how platforms stop working blind.

The industry already knows how to do this in a few areas. GIFCT's hash-sharing database lets members match known terrorist content, including attacker-produced footage, using perceptual hashes without exchanging user data. The Tech Coalition's Lantern program shares account identifiers, URLs and behavioral indicators related to child exploitation. In 2025, Lantern signals contributed to action on 31,504 accounts, 27,646 pieces of content and 17,778 URLs, and in 2024 they helped flag 81 contact offences and 45 trafficking cases. In one case, signals Meta shared helped Block identify a network of Cash App accounts linked to suspected CSAM sales, and helped Microsoft act on related Xbox accounts. Thrive, built with the Mental Health Coalition by Meta, Snap and TikTok, lets companies share hashes of graphic suicide and self-harm content and of dangerous viral challenges.

These programs work because a signal from one platform becomes a lead for another. The receiving company still reviews the activity against its own policies and makes its own decision. Nobody is outsourcing enforcement; they are just no longer working blind.

Outside those few areas, there is very little. There is no shared program for nihilistic violent extremist networks that move between gaming, chat and social platforms, for people banned for grooming or threats who simply sign up next door, or for sellers of weapons and drugs who rotate across services. The reason is mostly legal. Under GDPR, every disclosure of an identifier needs a lawful basis, and "this account was banned on our platform" is personal data about someone who hasn't been convicted. Recital 56 of the DSA states that the regulation provides no legal basis for profiling users to identify crimes. So each company's legal team weighs the risk alone, and the cautious answer is usually no.

Even the best-established exception proved fragile. The EU's temporary derogation that let messaging services voluntarily detect child sexual abuse expired in April 2026 after the European Parliament rejected an extension, and a replacement was only approved in July 2026. Collaboration cannot be built on a legal footing that disappears every couple of years. What's needed is a permanent legal basis for sharing safety signals across the same list of critical harms, with conditions attached: share signals rather than content, let each platform make its own decision, set retention limits, audit independently, and give people a way to challenge a signal about them.

The objections are real

Privacy. Sharing identifiers of people who are suspected, not convicted, is a real intrusion, and a false positive on one platform can follow someone everywhere. Even GIFCT has been criticized for opaque decision-making about what goes into its database. The answer is not to avoid sharing but to constrain it in law rather than leave it to each company's discretion.

Capacity. Flooding police with reports they cannot process would make things worse. That is why central intake and shared schemas have to come before a broader reporting duty, not after.

Platforms are not courts. What is illegal varies by country, especially for speech. That is a reason to define critical harms around offences that are serious and clear-cut nearly everywhere, rather than to report nothing at all.

Government abuse. In countries where dissent or LGBTQ+ content is criminalized, a duty to report "illegal content" becomes a surveillance tool. Any framework needs a defined harm list, rule-of-law conditions on which authorities receive reports, and public transparency on referrals by country.

Measuring what matters

As I wrote in the trust & safety paradox, what gets reported is what gets funded. As long as transparency reports lead with removal counts, removal is what the industry will optimize for. Platforms should also report how many networks they disrupted, how many victims they identified and supported, how many cases they referred to law enforcement by harm category, and what happened to those referrals.

Trust & Safety has become very good at removing things. It is far less good at stopping the people behind them and protecting the people they target. That isn't a failure of the teams doing the work. It is the result of a system where each platform fights alone, where the most serious response available for most harms is a deleted account, and where the legal basis for doing more is missing or temporary. Until that changes, removal numbers will keep going up, and the same threats will keep coming back.