Online Harassment #

When Maryam moved from her small village in Afghanistan to Kabul for high school, all her friends were obsessed with something called “Facebook.”¹ She was eager to join the platform but also anxious that her parents wouldn’t approve, so she did what so many of her friends and cousins had done: she opened an account under a fake name. For her profile picture, she used a picture that she found online of a child holding a giant heart shaped sign against a red and white swirl background. At school she told her friends her account name. She soon started receiving friend requests from other accounts with unfamiliar names and pictures, but accompanied by messages like “Hi Maryam! This is Zeinab!”

Maryam used Facebook like any other adolescent kid. She shared stories, chatted with her friends, wrote on people’s walls, and commented on their posts. She had group chats with friends from school and acquaintances from her hometown. It was exciting to be able to chat with friends all the time and to meet new people through group messaging. But she remembers clearly the day that changed.

About a month after Maryam created her Facebook account, a boy she didn’t know direct-messaged her. He lived in a distant province but was a friend of one of her schoolmates and had been in group messages with her. The first few messages were tame enough - “hi, how are you? Do you live in Kunduz?” Maryam responded cordially. But his messages quickly changed. He started sending aggressive sexual messages to her, suggesting things he wanted to do to her and acts he wanted her to do to him. Terrified, she turned off her phone and buried it under her bed.

The messages were not only aggressive and unsolicited, but also confusing. Maryam came from a conservative religious community and was still too young to even know about sex. She did, however, know stories of what happened to girls who were caught speaking with boys or sharing photos online. If a boy left a message on a girl’s Facebook wall that said “it was good to see you” or “your picture looks nice,” many would assume that the girl had invited this flirtatious behavior. In her community family honor is held in the highest regard; petty rumors could ruin a girl’s reputation and slander her family. Honor killings are a daily reality. Even if she clearly did not reciprocate, this boy’s comments could bring shame on her and her family.

Maryam deleted the messages and blocked the boy. She prayed that no one had seen them. Maryam’s first encounter made her extremely wary of any engagement on Facebook. She stopped posting photos of herself. She refused friend requests from distant relatives and family members in her father’s village, who had found her profile through the “suggested friends” function. The mere fact that she had Facebook friends who were boys could be enough to start rumors.

A few weeks later, she had another terrible experience on the platform. Her friend’s memory card was stolen at a wedding. The thief used the photos, which showed her in a dress with bare arms and no hijab, to create fake accounts. These accounts quickly gained thousands of friends. Maryam and her friends gathered together to try to report the accounts, but their English was not strong enough to follow the reporting flow. What had once been an enjoyable way to keep up with her friends had turned into the source of frequent misery.

A Pervasive and Dangerous Threat #

Almost everyone who uses the internet will either experience or witness harassment, be it on Reddit forums, in YouTube comments, or through direct messaging.² The concept of harassment is quite broad, encompassing a variety of different behaviors. Harassment might involve teenagers mocking each other, or it may involve strategic efforts by white supremacists to leverage hateful attacks against journalists. It can be intensely personal or part of a coordinated effort to “pile on” a target. It can stay in the online world or cross into real world harm. Sometimes, harassment shows itself clearly, such as when girls receive unsolicited pornographic videos or “dick pics” on Messenger*.* However, as we saw with Maryam, harassment can often only be understood within specific cultural contexts. Many human content moderators would not perceive a seemingly innocuous comment or a photo of a woman without a hijab as harassment.

Harassment underlies and interacts with many other types of abuse that we discuss in this book. Harassment can include hate speech; harassment can be used as a tool of domestic violence; and harassment can involve spreading malicious disinformation about a person. Harassment is sometimes accompanied by technical attacks, and, at its most extreme, can involve incitement to violence.

Figure 1: Harassment is complicated and adjacent to many other forms of online abuse.

In this chapter, we will break down some of the actors, tactics, and motivations involved in online harassment, analyze a few of the most notable examples, and discuss different policies and responses that companies can deploy to respond to the problem of online harassment.

Trolling Tactics 101: A Taxonomy of Online Harassment #

We can divide the tactics of online harassment into three broad categories: public, communal, and private harassment. Public harassment can be done by a single individual, but it occurs in public. It might involve publicly releasing information about a victim, incessantly badgering someone, or even sending the police to a person’s house. Communal harassment, while also occurring in public, involves coordinated harassment activities. Private harassment, by contrast, pertains to the variety of types of harassment that are done on a one-to-one basis. In this section, we will discuss several prominent examples from each category.

#

Harassment Tactics: Public #

Doxxing #

Doxxing (dropping docs) is the intentional broadcast of private personal information, such as an individual’s home address and phone number, usually with malicious intent.³ Releasing a person’s public information online often marks the individual as a target for further harassment, and puts the victim at risk, since unhinged or vindictive people could use this information to hurt them⁴. Doxxing has been a common form of harassment for celebrities, journalists, prominent business people, abortion providers, and suspected criminals. While doxxing is sometimes the product of sophisticated hacks, in most cases it involves merely publicizing information that can be obtained through simple public or private searches for personal information.

In rare cases, governments have used doxxing to silence and endanger critics. Consider, for instance, Ruslan Leviev, a Russian activist and blogger who started an open source intelligence group (“Conflict Intelligence Team”) that charts Russian military maneuvers in Ukraine and Syria. In addition to receiving regular threats on what was then Twitter and Facebook, a “patriotic” website posted his home address, phone number, and passport number. Hackers also stole images from his account and portrayed him publicly as a rapist and pedophile.⁵

In recent years, the rise of two trends has politicized the definition of the term doxxing, which is often used inconsistently by the same observers depending on the political alignment of the person being revealed and the one doing the revealing. A key question is whether unveiling the identity of a human behind a pseudo-anonymous account is ethical, especially if that account is influential and involved in itself driving harassment. One key example of this controversy played out in the United States in 2022 when then-Washington Post reporter Taylor Lorenz revealed the identity of the woman behind the highly influential (then-Twitter) account LibsOfTikTok⁶. The original version of the story backed up its conclusion with a link to a work address linked to the woman, Chaya Raichik, a link the Washington Post removed after criticism. It’s worthy to note that Libs of TikTok itself was and still is known for the same kind of behavior, driving harassment to otherwise anonymous targets via dogpiling (covered below) which has led to numerous bomb and death threats to targeted individuals and organizations⁷.

In my opinion, it would be correct to call the Washington Post’s initial story a type of doxxing, although one justified by the reach and impact of the Libs of TikTok account. I do believe that any linking to contact or biographical information was not appropriate for a responsible media organization. As some have written, the term doxxing perhaps is no longer useful as anything other than an emotional label⁸, much like “fake news”.

Swatting #

One reason that being doxxed can be psychological distressing is that, in many cases, bad actors will use the information to commit further harm. One type of harm that can follow doxxing is swatting*,* the act of intentionally calling in a false report of a crime to prompt police to send a SWAT team to the victim’s address.

In a typical swatting incident, the harasser might call law enforcement in the middle of the night and, while pretending to be their victim, admit that they have just shot their wife and are about to shoot the kids. After receiving the call, the police will quickly make their way to the victim’s house, where the victim is fast asleep. Given the content of the call, the police arrive with every expectation that the suspect is armed and has that intention to cause harm. While swatting may seem like a harmless prank, these encounters are dangerous and occasionally deadly.

In 2017, for instance, a swatting incident led to the death of an unintended target.⁹ During an online beef related to the video game “Call of Duty,” Casey Viner threatened to swat Shane Gaskill. Unperturbed, Gaskill told Viner that he’d “be waiting” and provided him with an address at which he no longer resided. Viner then contacted Tyler Rai Barris, a serial swatter, to make the swatting call. Barris called the Wichita police department, claimed he lived at the address provided by Gaskill and that he had killed his father and was holding his family hostage at gunpoint. Police urgently responded and demanded that anyone at the address immediately come outside. A confused resident of the home, Andrew Finch, stepped outside to see what was causing all the commotion. As he raised his hands, an officer opened fire and killed him.¹⁰

Planting False Stories Online #

In 2021, the New York Times Technology reporter Kashmir Hill exposed a Canadian woman who sought revenge against people in her life by creating fake, defamatory stories about them on sketchy online forums meant for reporting complaints, scams, and frauds. As a result, when anyone searched for the victim on Google, the first results would show up as “pedophile,” “rapist,” or “cheat.”¹¹ This harassment campaign, which targeted dozens of people (including the article’s author once the perpetrator discovered that she was being written about), made it difficult for victims to land jobs and caused significant distress.

Impersonation #

Another form of public harassment involves impersonation–the act of creating an account with another person’s name or image. Often, impersonation attacks are done to hurt another person’s reputation. An impersonation account might, for instance, publicly post racist comments, or they might send provocative images or messages to people known to the victim.

Misidentification #

While it can be harmful for someone to claim they are you, it can likewise be harmful for others to claim you are someone you are not. Such accusations can cause onslaughts of harassment. In the wake of the January 6 attack on the Capitol, efforts were made by vigilante online groups to identify people in photos and videos. Such efforts, while sometimes effective, open up opportunities for error and abuse. Even if ultimately the person falsely accused is proven to be innocent, their public reputation will already be tarnished and they will often continue to be hounded by people who believe them to be guilty.

Public Non-Consensual Intimate Imagery #

We define the sharing of non-consensual intimate imagery (NCII) as the release of intimate or sexual images without the explicit consent of the subject. While sextortion and NCII are closely linked, we list NCII separately because it relates to the release of explicit imagery whereas sextortion relates to the threat of release. The NCII may have been initially obtained with consent of the subject, but the person did not consent to the public release of the images. These cases are sometimes referred to as “revenge porn,” but because pornography comes with an implication of agency, we prefer not to use that term. Furthermore, the word “revenge” insinuates that the victim may deserve the harm and harassment they receive.

Celebrities and public figures have increasingly been victims of NCII releases through hacks of personal data or attacks by former partners. In August 2014, a successful spearfishing attack granted access to hundreds of iCloud image accounts, many belonging to A-list celebrities. Nude images of celebrities obtained through the hack were released on 4chan and Reddit.

NCII attacks tend to adversely affect women more than men. For example, on November 3, 2019, Katie Hill, U.S. representative for California’s 25th district, resigned after her estranged husband leaked photos of her engaged in sexual acts with him and another female staffer. The photos were circulated on the conservative blog RedState and the British tabloid The Daily Mail. In her resignation statement, Hill said:

“This is what needs to happen so that the good people who supported me will no longer be subjected to the pain inflicted by my abusive husband and the brutality of hateful political operatives who seem to happily provide a platform to a monster who is driving a smear campaign built around cyber exploitation. Having private photos of personal moments weaponized against me has been an appalling invasion of my privacy. It’s also illegal.”¹²

Hill has since become an advocate for stronger laws protecting women from NCII.¹³ She points out the glaring double standard that leads women to be more likely to resign from powerful roles than men when caught engaging in inappropriate behavior.

Sealioning #

Sealioning involves the repeated and incessant questioning of a person until they ultimately stop engaging with the tormentor. This trolling tactic is the internet version of a child repeatedly asking their parents, “but why?” The abuser may appear to be engaging in dialogue, but intends to turn the table on their target and inundate them with questions. When the target withdraws, the speaker claims that they were the ones attempting to be open and civil. Sealioning is best demonstrated in the comic (below) by David Malki that gave sealioning its name.¹⁴ Another term for sealioning is “JAQ-ing off”–an abbreviated form of “just asking questions”.

Figure 2: 2014 comic strip by David Malki in his webcomic Wondermark. (Source: Wondermark¹⁵)

Harassment Tactics: Communal #

Communal trolling describes when an abuser causes harm by stirring up trouble in the community around someone.

Dogpiling #

The name dogpiling originated in the world of sports, where it refers to everyone piling on top of one player (as dogs occasionally do when playing with each other). Dogpiling in the digital world is similar: typically a target will be selected based on their personal characteristics (e.g. race, ethnicity, religion, gender, sexual orientation, sexual identity) or because they have expressed an unpopular opinion. The target will then be overwhelmed with negative content (such as comments) by an online mob. This is designed to make a target recant or withdraw from online spaces.

Dogpiling is sometimes so extreme that a person cannot participate in online forums, thus becoming a DoS attack.The ultimate victory for perpetrators is if the victim is forced to make their Twitter account private or delete their account. Dogpiling can be challenging to regulate because in most situations, each individual piece of content falls below the standard that would trigger company action. If one person tells you that your sweater is ugly, it would not be considered abuse from the platform’s perspective. However, when thousands of people pile on, it can become a much worse situation.

Brigading #

Brigading is similar to dogpiling in that it involves rallying a group of people to harass an individual. However, as the name suggests–in military terms, a brigade is a group of troops– brigading entails a more organized effort to rally a group of people into a single line of harassment. Brigrading is often the term used when this group manipulates a third-party source of value, opinion or truth, for example a search engine, review site or comment board.

Sometimes, advocates use brigading to troll public figures or as a mechanism to demonstrate widespread displeasure for their powerful actions. In 2003, in response to then-US senator Rick Santorum’s statements supporting anti-sodomy laws, the columnist and LGBT activist Dan Savage started an online campaign to create a neologisim defining the word “santorum” as a specific byproduct of male sex. He purchased several domains including “santorum.com” and, with the help of thousands of Savage’s supporters, made it so that the top search results on Google, Bing, and Yahoo displayed the new santorum definitions before the senator’s own campaign pages.

Small businesses are often the targets of brigading attacks and counter-attacks, which can bring thousands of people from around the world to their virtual doorsteps in an effort to destroy (or protect) their livelihoods when they get involved in a national controversy. For example, the classic American campaign tactic of a candidate visiting a restaurant to shake hands with “normal folks” now regularly leads to waves of negative¹⁶ reviews for that location, which might not have known about a visit until minutes before the politician walked through the front door.

Harassment Tactics: Private #

Private harassment pertains to the variety of types of harassment that are done on a one-to-one basis. Many types of private harassment come with the threat of making information obtained public if the victim does not comply with the harasser’s demands.

Individual Threats #

Using direct messages, emails, and text messages, people can send private messages that threaten, intimidate, or harass individual users. The private nature of these messages often means that the content is much more graphic, violent, or offensive. While many email services have robust spam filters, there is otherwise little filtering of abuse. People can send almost anything, including death threats. The same is true for Facebook messenger, SMS, iMessage, WhatsApp, Instagram DM, and other chat platforms. Given the low barrier of access on platforms like Facebook or Twitter, an abuser merely needs to know a victim’s screen name to begin engaging in abusive behavior.

Message Bombing #

Message bombing involves groups of people intentionally flooding an individual’s private text, email, or other messenger inbox with messages. If the group sends enough messages over a short period of time, it can overload the servers and victims can lose access to their accounts. For example, in 2017, Propublica’s email servers crashed after retaliatory actors signed their journalists up for thousands of online email subscriptions.¹⁷

Cyberstalking #

Cyberstalking is the prolonged use of internet tools to stalk and harass a person with the intent to intimidate, harm, or, in extreme cases, kill them. The internet provides a wealth of open source, paid, or hackable access to personal data and allows a stalker to follow an individual’s behavior online. There is also a vibrant market in off-the-shelf surveillance products that enable stalkers to pursue those goals.

A cyberstalker may use information gained from their surveillance to create fake accounts or personas in an effort to defame or slander their victim. In the United States, cyberstalking is criminalized in numerous states and jurisdictions. As an example, in 2018, 39-year-old Joel Kurzynski was charged with cyberstalking two individuals and sentenced to two years in prison. The Department of Justice press release described his actions as follows:

Beginning in March 2017, Kurzynski orchestrated numerous spam phone calls to Victim 1. The conduct soon escalated to fake dating profiles wherein Kurzynski portrayed Victim 1 as seeking sadomasochistic or underage relationships. These profiles contained photographs of Victim 1 and his contact information, resulting in solicitations and harassing messages directed toward Victim 1 from multiple strangers. Kurzynski then sent several anonymous death threats to Victim 1, including the threat, “faggot. Time to die.” At one point, Kurzynski impersonated a journalist and contacted Victim 1, claiming that an upcoming article would levy sexual misconduct allegations against Victim 1 related to Victim 1’s work with a non-profit youth organization.

Kurzynski also admitted that in November 2017, he began registering Victim 2 for numerous weight loss and suicide prevention programs, resulting in a wave of calls and emails from entities such as Overeaters Anonymous, Weight Watchers, Yellow Ribbon Suicide Prevention, and others. Within weeks, Kurzynski started sending anonymous death threats to Victim 2, many of which referenced Victim 2’s work address. One threat claimed that he was waiting for her in the lobby, and another that said, “Looking forward to seeing you today and how much you bleed. Don’t go to the bathroom alone.”¹⁸

Catfishing #

Catfishing involves the creation of fake online profiles to lure someone into an online relationship. In most cases, the perpetrator’s goal is to convince the victim to send gifts and money. However, in rare cases, the purpose of the catfish is to hide one’s identity while pursuing a romantic relationship with the target.

In perhaps the most famous instance of the practice, Notre Dame linebacker Manti Te’o was catfished by an acquaintance. In 2011, “Lennay Kekua” reached out to Te’o on (what was then called) Twitter. Through Facebook and Twitter, Te’o and Kekua started a romantic relationship, although the two never met face-to-face. Te’o was a stand-out his senior year at Notre Dame, and a favorite for the Heisman, putting him in the national spotlight. In September, reporters shared that Te’o’s girlfriend had died of Leukemia shortly before Notre Dame’s big game against Michigan State. Headlines at the time praised Te’o’s “strength” in being able to continue through the tragedy,¹⁹ and the media hungrily shared the touching personal interest story. An investigation by Deadspin, however, revealed that Kekua did not exist.²⁰ She was the creation of a male acquaintance of Te’o’s who had a romantic interest in the football star but knew that his feelings were not reciprocated.

Sextortion #

Sextortion is the extortion of an individual using sexually explicit material in exchange for either money or further explicit material. Extortionists may obtain explicit material through hacking, coercion, or deception (such as catfishing), and then threaten to reveal the material to others, such as the victim’s friends, family, or coworkers, if they do not comply. Because sextortion can cause embarrassment to the victims, victims frequently fail to report the harassment to authorities. In a rare moment of activism, a British man, Jon Pearn, went public about his sextortion experience in 2016.

Pearn, a 62-year-old living in Plymouth, England, accepted a friend request from a pretty woman on Facebook. She started messaging him and convinced him to open a Skype video chat with her. Once on Skype, she convinced him to show her his penis. After he did, the person demanded Pearn send her £500, threatening to share the picture with his family and friends on Facebook if he did not comply. A self-employed carpenter, Pearn felt he didn’t have anything to lose from the exposure and instead took his experience to the media.²¹ He was likely the victim of an organized crime ring that creates fake Facebook and Skype accounts to lure victims into voluntarily sharing explicit images, screen capturing them, and demanding payment to keep the images private.

Sextortion can sometimes happen without the extorter being in possession of explicit material. A collaborator on this book, for example, received a phishing email that read:

“i know [redacted] is one of your password. I’ve recorded your cam while you were watching porn on [redacted] sites, also I’ve installed a keylogger on ur pc & collected all your contacts on social networks, messenger & emails. If you want me to erase the recording, pay me 1128$ in bitcoin on bitcoin address [redacted]. If i don’t get the bitcoins, i will definitely send your video to all your contacts.”

The password listed was a real password no longer used by the recipient, but they had never visited the site listed. The phishing scam sought to extort the individual merely by convincing them that the scammer possessed explicit material related to the target.

More recently, there have been cases of individuals in Cote d’Ivoire and Nigeria sextorting children in Western countries for money.²² In rare cases, these individuals have been extradited to the U.S. for prosecution.²³

The deceptive relationship-building phase of sextortion can be time consuming. AI, however, is making this more efficient. Perpetrators can now send an initial message to a child with an AI-generated nude image.²⁴ The message might warn that while the child knows it’s fake, family and friends might think it’s real, and leverage this threat for extortion.

Sextortion also discussed as an abuse type in our chapter on Child Sexual Exploitation.

Notable Cases of Online Harassment #

With the above taxonomy in mind, we will now analyze three notable cases: the harassment of a young Canadian teenager, the harassment of female game designers (#Gamergate), and the targeting of journalists by rightwing trolls. One common thread in the three cases is that the harassment combines several different tactics outlined in the previous section. Understanding how these different tactics work together is crucial to disrupting and preventing harassment campaigns.

Amanda Todd #

On September 7, 2012, a 15-year old Canadian Teenager posted an approximately nine minute video on YouTube narrating her struggles with years of harassment, bullying, and sextortion at the hands of an anonymous online tormentor and her real life classmates. The video, “My story: Struggling, bullying, suicide, self-harm,” showed Todd holding a series of handwritten cards detailing her experience over the past two years, with the song “Hear You Me” by Jimmy Eat World serving as the soundtrack.

Figure 3: Screenshot from Amanda Todd’s YouTube video. (Source: YouTube via The Hacker News²⁵)*

When she was around 13, Todd was convinced, after months of grooming, to flash someone she had met in an online video chat. The person screen captured the explicit material, which he then used to harass her into providing more material. He also created a Facebook account as a new student at her school and, after friending many of her classmates, changed his profile picture to her topless picture. She moved schools, but the harassment followed. She was bullied and harassed at school, while the extorter continued to harass her online.

A month after posting the 9-minute video, Amanda committed suicide. Fuelled by the attention from the video, an investigation by the Royal Canadian Mounted Police, Facebook, Cybertip.ca, and ultimately the Dutch police tracked the perpetrator down in the Netherlands. When Aydin Cobin was arrested in the Dutch town of Oisterwijk, he possessed hard drives with pictures of thousands of potential victims. Cobin was tried and convicted in the Netherlands on cases relating to other victims then extradited to Canada to stand trial, where he was convicted of extortion, possession of child pornography, and criminal harassment against Todd.

Amanda Tood’s case is an example of how the harassment techniques described above can be used together to ruin a person’s life to the point where they think it is no longer worth living. This case included multi-platform cyberstalking, sextortion, and doxxing.

#Gamergate #

While online harassment has been a feature of the internet since its birth, many point to “Gamergate” as the moment where several of the contemporary aspects of online harassment gained widespread attention. Gamergate was an online harassment campaign that targeted female game designers who had come out against sexism and violence in gamer culture. In 2013, game designer Zoe Quinn released a well-received game called “Depression Quest,” developed to foster empathy for those experiencing depression. One of her ex-boyfriends accused her in a blog post of cheating on him with game reviewers in order to garner positive coverage for her game. The blog post became fuel for increasingly hateful discussions on reddit and 4chan, where Zoe’s personal information was published. Her tumblr, dropbox, and skype accounts were hacked. Intimate photos of her were leaked. Brigades of internet trolls attacked Quinn on social media, spreading the hashtag #gamergate and sending rape and death threats. Death threats and doxxing against other Quinn defenders led to one selling his company and leaving the game world, while others were forced to cancel speaking engagements. The movement had no clear leader but rather relied on brigading and dogpiling tactics.

The vitriol directed toward Quinn and her defenders was seemingly driven by a general dislike of Quinn’s game and what it represented. A movement of mostly young men who loved video games argued that video game culture had been taken over by “social justice warriors” and the games were too politically correct, in large part because of a handful of mostly female video game authors, engineers, producers, and journalists. Part of the goal of the harassment campaign was to tear down any media criticism: if anyone criticized a game from a cultural perspective, they would immediately become a target of the mob. Gamergate resulted in doxxing, swatting, and threats of violence: many targets quit their jobs, moved houses, and went to live with family and friends.

Multiple cultural commentators have observed that the pattern established by Gamergate has become extremely influential, especially among English-speaking right-wing communities. The same pattern of hyperbole, dogpiling, doxxing and harassment has been repeated against dozens of targets that have been seen as overly progressive in the subsequent years. It has also spawned counter-movements in the gaming industry, leading to regular controversies and review brigading of games seen as “woke” several times a year.

Journalists #

In the past several years, journalists have become an increasingly common target of online harassment campaigns. During the 2016 election, several Jewish journalists, especially journalists who wrote unflattering articles about Candidate Donald Trump, became the target of a mass harassment effort. Here is, for instance, how the conservative commentator Bethany Mandel described her treatment following a negative comment about president Trump:

“I was deluged by this troll army after the South Carolina primary, when I referenced the anti-Semitic following that Trump had online. Suddenly, I was getting 100 tweets a minute. I’d open my phone at my kids’ playgroup and see a picture of myself in a gas chamber. My biggest issue at the time was determining if [something] was actually a threat and, if so, what to do about it.”²⁶

Figure 4: Antisemitic and violent images of Bethany Mandel following her negative comments about President Trump in 2016. (Source: Anti-Defamation League²⁷)*

Bethany’s harassment makes clear the importance of context in harassment. Consider, for instance, the image below, in which a picture of the famous Auschwitz Concentration camp has been altered to say “Make America Great” instead of “Work will make you free.” In context, it is clear that this is basically saying “we’re going to kill you because you’re a Jew.” But without historical context, it would be nearly impossible to identify this as a threatening image, which means that machine learning systems would be unable to prevent the posting of such images at scale. Further complicating the matter: terrible images are critical for remembering the atrocities of the Holocaust, and museums and other non-profit organizations will often share them on Remembrance Day and other important anniversaries. Thus, companies cannot prevent the images from being used for online harassment campaigns merely by banning the images (as they can with, e.g., CSE).

Figure 5: Antisemitic content harassing New York Time reporter Jonathan Weisman. (Source: Anti-Defamation League²⁸)*

Responses #

To prevent online abuse, companies need to be proactive: they need to stay ahead of bad actors by setting clear policies and enforcing them before a harassment campaign has time to gain momentum.

Policy Responses #

Set clear policies for awful but lawful content #

One of the most important preventative actions a company can take is to adopt policies for “awful but lawful” content. Many instances of harassment qualify as free-speech and thus are not illegal to post. If companies want to regulate this behavior, they need to set internal policies that prohibit it. Even in cases where there are civil and criminal statutes related to bullying and harassment, the legal definitions of the behavior are often much more permissive than what most platforms should allow.

Make solid partnerships #

In order to identify and anticipate online harassment campaigns, companies need to form partnerships with groups that can keep them informed about trends in abusive behavior. This is particularly the case when companies operate in countries with which they have less cultural familiarity. For instance, in the Philippines, a journalist named Maria Ressa published critical, investigative pieces about Rodrigo Duterte, the lawfully elected, but autocratic ruler of the Philippines. In the course of her work, she was harassed by people who work for him directly, as well as by his supporters. Unfortunately, the American platforms did not take this harassment seriously until Maria herself was able to create a public movement in her defense.

Engage victim advocates #

One of the worst features of being the target of harassment is it can be isolating: you feel like nobody’s on your side. Because abuse occurs across different platforms, the victim is often the only person who knows the extent of the harassing emails, messages, posts, and phone calls that are being directed at them. As a result, targets often feel that the technical solutions preferred by platforms–many of which place the onus on the targets to report harassment–fail to grasp the scope of the harassment. Reporting can be overwhelming and feel like playing a game of whac-a-mole for victims, especially in cases of cross-platform harassment.

One way to help victims is to connect them with victim advocates. There are a range of organizations, some of which deal with specific forms of harassment (e.g., domestic violence, journalist safety, or child safety), while others deal with the effects of online harassment more generally. In certain cases, these organizations can provide targets with strategies to minimize the harassment they are experiencing. In other cases, they will simply talk to victims and help them deal with the emotional pain caused by the harassment. They can also act as important advocates for the victims in dealing with platforms.

Product and Technical Responses #

Detailed and Effective Reporting flows #

The most basic product response that every should have is a reporting flow. If people are unable to report when they are being harassed, there is no way for a company to know that a harassment problem exists; if the company doesn’t know the problem exists, there is no way to respond to it, let alone respond effectively. We will dive deeply into reporting flows here, but they are a critical part of nearly every abuse type we discuss in the book.

You can tell a lot about how deeply companies have thought about these issues by looking at their reporting flows. This is a diagram of the reporting flow for different kinds of abuse in Whatsapp, which is very, very thin. The number of options presented to the user is quite small, and while the user is presented with the choice to block a contact or report them, reporting does not allow the user sufficient choices to generate data around the report that would help the company act aggressively in responding to the abuse.

Figure 6: WhatsApp reporting flow.

Compare the reporting flow on Facebook (below), which is much more sophisticated and provides the company with much more detailed information. Despite the fact that both platforms are part of Meta, the two approaches are wildly different. Whereas the whatsapp trust and safety team is essentially telling users, “report something we’ll get to it, maybe,” the Facebook trust and safety is much more aggressive, signaling to users that they take reports of abuse seriously. Beyond merely making the users feel like the report matters, the more aggressive approach gives the company a huge amount of metadata that allows them to take the report, and perhaps even deal with it automatically.

Often, when something is reported on certain kinds of flows, that will trigger automatic machine learning that, while perhaps not accurate enough to be run on every single piece of content, is accurate enough in a context where someone identifies a piece of content as, for example, an instance of harassment using a racial epithet. Deep reporting flows that provide context thus enable automation. They also allow trust and safety teams to prioritize different kinds of abuse and direct the reports to different groups, such as specialists in those areas and languages.

Figure 7: Facebook Messenger reporting flow.

In general, companies that have dealt with significant abuse issues tend to have well-developed reporting flows for harassment, while companies that have not tend to have poorly thought out or underdeveloped flows. Here, for example, is the reporting flow for Reddit, which has significant experience with bad behavior.

Figure 8: Reddit reporting flow.

In addition to the amount of information it captures, reporting flows also vary in terms of their ease of navigation. Difficult to follow reporting flows essentially tell the user that the company has no concern for online harassment. Consider the reporting flow for Telegram: while it is technically possible to block people on the service, the reality is that most people will not be able to locate this capability, and thus will be unable to use it.

Figure 9: Telegram reporting flow.

Without claiming to provide a comprehensive blueprint – every reporting flow should be tailored to the specific needs of the company and user base – here are some general characteristics of a good reporting flow.

1. Terms of service (TOS) are clear and consistently applied - The terms of service should provide clear definitions of the relevant terms that comprise the reporting flow. The median reporter will likely have never used the reporting flow before, so the terms like those defined above and elsewhere in this book (e.g., NCII, CSE, incitement) may not be obvious.

2. Encourage users to take action - The reporting flow should encourage users to take action themselves, such as blocking or muting the person they are reporting. This can minimize the harassment while it takes hours, days, or weeks for a company to respond to the report and take action.

3. Make it easy to complete - The report should present the user with clear choices or categories. For example, if someone is being bullied, it may be more helpful to present them with a reporting option for “bully / harassment,” than merely harassment. In order to develop user-friendly categories, companies need to invest in UX design and research. For the same reason, it is also important to tailor reporting flows to different cultures and communities. What is simple and obvious to tech companies’ college-educated employees may not be obvious to, for example, a high schooler, a non-native English speaker, or users from different countries.

4. Gives adequate data for queuing and ranking - The reporting flow should be sufficiently fine-grained that it provides adequate data on the back end, so that trust and safety teams are able to organize and prioritize the reports.

5. Reporting flows should differ for public and private spaces - Reporting flows for public and private spaces should differ, in part, because the appropriate action taken by the company should differ. In public spaces, companies will need to decide what threshold qualifies for removal, and how to handle cases that fall just shy of that threshold. For instance, if content is reported on Twitter, but it hasn’t yet been reviewed by content moderators or Twitter’s ML is unclear whether the content is abusive or merely a joke, the content may receive the following label:

Figure 10: Label on X. (Source: x.com)

In private spaces, companies need to provide uses with a different set of capabilities. For instance, a number of companies have implemented the capability to block people in a way that is not obvious to the person being blocked. These capabilities allow the users to disconnect themselves from the person without starting a fight or giving the harasser the satisfaction that they were able to get under the user’s skin.

Figure 11: Instagram’s “Restrict” feature. (Source: Instagram)

Instagram, for example, offers users the ability to “restrict” an account. A restricted user can still send messages, but these are filtered into a message requests folder. They can also comment on posts, but their comments are only visible to them and the account owner.

Proactive monitoring #

A key and necessary task for stopping targeted harassment is proactively monitoring for the use of language that might indicate individualized attacks. Most platforms use clusters of words such as racial, ethnic, gender and religious slurs as well as other types of profanity, to help guide their classifiers, when looking proactively for harassment and hate speech.

Using such lists can be extremely challenging in practice.

There are, however, certain signals that the platform could take into account when monitoring for harassment, such as the fact that the two users have followed one another for several years and have exchanged thousands of messages without a single report. Given this context, the odds are extremely high that whatever they’re saying to one another that is flagged by the ML classifier is playful, rather than an actual instance of harassment. Slurs and profanity should always be considered alongside more rich signals and should never be used as the sole reason for taking action against an account, unless a platform has strict and transparent rules against such language (such as platforms meant for children, or professional platforms that blanket ban unprofessional language).

Detect and Stop Pile-Ons #

Some platforms have made progress in slowing down the impact of group pile-ons through thoughtful designs or interventions, such as rate-limiting alerts during high-volume negative events or providing pop-ups giving users more choices to filter their feed or notifications upon detecting a high number of negative mentions from strangers. While such interventions will likely not help reputationally from an organized attack or somebody organically becoming the “main character” of a platform for a day, the psychic damage of having thousands of strangers tell a user to kill themselves can hopefully be lessened by a platform hiding those messages behind an interstitial and not buzzing their phone with each individual message.

Build-in productive friction #

How can you build features into platforms to make it “more work” or “less fun” for harassers? One strategy is to create what is referred to as “productive friction.” Instagram, for instance, has experimented with the following “rethink” feature: if somebody posts content that is not against the rules, but does not seem friendly, it makes users wait through a timeout and read a message before they are able to post it. In essence, the pop up is meant to remind the user that they are posting this content in public and that it might be harmful to another person. While this kind of interruption will do little to stop users who create fake accounts with the express purpose of being abusive, it may be effective at preventing people who are merely caught up in the moment, or may be unthinkingly joining an abusive dog pile.

Design for 3rd party anti-harassment tools. #

We’re starting to see development of third party tools to deal with harassment. I am, for example, a user of (and, full disclosure, investor in) Block Party, which offers a suite of anti-harassment tools for platforms including Facebook, X, and Linkedin. The app uses filters that help mute the people most likely to bother you with unwanted or harassing content. It also allows you to mass block users; on X, for example, you simply identify a particular post, and it will automatically block all accounts or users that like or share the post. Unfortunately, I have firsthand experience with the need for these tools. After appearing on CNN to discuss the rightwing media ecosystem, I became the target of several prominent rightwing media figures, including Jack Posobiec, Kayleigh McEnany, and Tucker Carlson. As a result, I attracted a fair amount of online harassment; over the next two days, Block Party blocked over 2000 people for me on Twitter.

Figure 12: Posts about Block Party and screenshot from Block Party. (Source: x.com and Block Party)

While X has closed many of the APIs that made external blocking tools possible, and many other commercial platforms have never offered such APIs, the move towards decentralized protocols and platforms, such as Mastodon (built on ActivityPub) and BlueSky (built on the AT Protocol) have created a great deal of discussion around this area. Our colleagues Frank Fukuyama²⁹ and Renee DiResta³⁰ have both written about the possibilities of middleware systems that put more choices in the hands of users, both of what they see algorithmically and what they block.