Content Moderation

Content moderation is the private governance of speech by digital intermediaries such as social media platforms, web forums, and hosting providers. It is a superficially simple concept: platform owners choose what to allow on their websites. A gardening forum may exclude speech unrelated to gardening. However, it’s more complicated than it might appear at first glance; content moderation implicates property rights, speech, and free association. It captures an array of practices, from user bans to automatic obscenity screening. Moderators may govern well or poorly and face different challenges depending on the purpose and size of their venue. Intermediaries may moderate for commercial, moral, or legal reasons. Differences aside, these efforts are all attempts to provide governance by setting rules for users.

The history of content moderation is inextricable from the history of the internet. Moderators’ priorities are driven by both the internet’s users and popular attitudes towards the internet’s role in society. Over the last few decades, both have changed dramatically. Content moderation is also bound by technology – who moderates, and how, is often determined by the structure of digital networks as much as by values or law.

There are two main justifications for content moderation. First, platform owners want to provide a good user experience, the details of which vary wildly across platforms. Second, they want to prevent off-​platform harms, usually criminal activities. While content moderation first emerged to address on-​platform conflict, it quickly assumed external concerns as well.

As a matter of libertarian theory, content moderation is treated as a form of private governance. Platform owners are not the government, but the decisions they make about their properties is still governance. Like the owners of theaters, coffee shops, and museums, platform owners set and enforce rules concerning the expressive use of their properties. That the property is accessed via the internet and the governed activity is communicative makes no difference in this analysis. Libertarians expect that moderation will be guided by platform owners’ desire to maximize the value of their properties, leading them to adopt whatever rules most satisfy their users.

As long as governments don’t impose barriers on the creation of alternative websites, users excluded from one platform by onerous moderation will be a prize to be claimed by more lenient websites. This again mirrors our expectations of physical venues. You cannot long heckle at the opera before being escorted out by an usher, but plenty of comedy clubs are happy to cater to your desire for interaction. And it is far easier to set up a competing website than it is to construct an alternative lecture hall or to launch a new satellite television network.

Nevertheless, content moderation reveals tensions between the rights of private property and freedom of speech. The internet offers an easy and instant means of communication. Despite websites’ posted rules, users have come to expect untrammeled expressive freedom online. Yet when organizing their private forums, moderators necessarily exclude some speech. All else being equal, this leads to less speech than we might have if moderators’ hands were tied. However, such restraint would make platforms ungovernable and likely make them less hospitable to most users.

In practice, moderation contends far more often with spam and everyday invective than it does heated political discourse, even if the debate over moderation tends to fixate on the latter. Speech is also filtered for simply being off-​topic. While this may seem harsh, it is necessary for productive discussion of any subject. Moderation of spam, harassment, and off-​topic speech is inward looking. It is intended to improve the user’s experience on the platform, making it a place they’re more likely to spend time. Like rowdy bar patrons, some users are excluded in order to improve others’ enjoyment of the space.

Moderators exercising their right to exclude does not violate the rights of excluded users. There is a difference between being suppressed and being directed to a more appropriate venue. Limiting redirection would limit the right to create a dedicated forum. Property can only be dedicated to particular purposes by prohibiting other uses. For example, a classic car discussion forum cannot be maintained without the ability to exclude posts about contemporary vehicles.

Increasingly, moderation also attempts to limit the off-​platform effects of platform misuse. As internet platforms’ importance to commerce, domestic politics, and international relations has grown, so has the salience of external justifications for content moderation. While some immediate external harms, such as imminent violence or financial fraud, may warrant platform action, libertarians are skeptical of platforms making legal judgements. Unlike liberal legal systems, content moderation lacks a robust adversarial adjudication process or right to appeal. When governments can foist determinations of legality on platforms, citizens lose valuable due process rights to platforms’ informal, private decision-​making.

Content moderation becomes a threat to free speech when it is totalizing, either as a result of a single firm’s dominance, through cross platform compacts, or when it is explicitly or implicitly directed by government. When all platforms follow, or are compelled by government to adopt, the same moderation rules, speakers will be unable to address willing audiences. So long as there are reasonably accessible alternative platforms—or so long as such tools can be built without interference—even disfavored speakers may still find a place to speak.

In America, the First Amendment prohibits the state from censoring speech. Nevertheless, politicians are able to engage in censorship by directing platforms’ moderation priorities with threats of regulation. In hearings on Russian misinformation, Sen. Dianne Feinstein (D-CA) urged Facebook, Twitter, and Google to “do something about it — or we will.” Platforms have largely complied. Shortly after Sen. Bob Mendendez (D-NJ) demanded the removal of a gun printing activist, Twitter suspended the activist’s account. Twelve state Attorneys General asked Facebook and Twitter to remove several specific anti-​vaccine accounts. This government bullying, known as “jawboning,” risks becoming a routine avenue by which speech is policed on private platforms, in contravention of the First Amendment.

Even if the content in question might have violated a broadly worded platform rule, because many violations of platform rules go unenforced government agenda-​setting can be dangerous. If government is able to direct limited moderation resources towards content that is different from that which a platform might have removed if left alone, government can effectively police content through platform moderators, directing their energies toward political opponents or disfavored speakers.

Foreign governments have become increasingly active and effective in demanding content takedowns. As the people of illiberal states such as Turkey and Thailand have come online, their governments often demand specific concessions in exchange for market access. Even democratic governments increasingly demand global takedowns of locally illegal speech.

Libertarians are hostile to this ongoing state territorialization of the internet. While not principally opposed to a fragmented or splintered internet, to see it divvied up between governments is anathema to libertarian hopes that the internet can provide a way to escape illiberal regimes. While libertarians often see moderation as unduly deferential to the state—for example, government and politicians’ accounts are often exempted from platform rules—they are skeptical of moderators’ removal of speech by popular politicians. These concerns are usually rooted in the interests of citizen listeners rather than government speakers.

Furthermore, while platform owners have every right to misgovern their platforms, the failure to offer anything resembling due process, particularly where automated moderation is concerned, offends a liberal sense of dignity. Even when platform rules are reasonably drafted, if applied seemingly at random or without regard for context moderation will be untrustworthy and incapable of guiding user behavior. When platforms fail to adequately explain their decisions—an almost impossible task at scale—reasonable decisions may appear capricious. Even as the largest platforms continue to grow, the private messaging, nonpublic groups, and ephemeral publishing services contained within them offer an escape from overbearing moderation. WhatsApp and Twitter Spaces exist within large platforms within extensive rules but cannot be moderated as easily as public text feeds.

Content moderation is as old as the internet. There was no halcyon period before content moderation. On the web’s predecessor, the government ARPANET, mailing lists were moderated by managers, though they mostly dealt with technical, rather than social, problems. In 1984, USENET, informally controlled by a ‘cabal’ of system administrators who maintained its backbone infrastructure, introduced moderated newsgroups in 1984. In 1987, the administrators reorganized the newsgroup naming structure, banishing some controversial topics to the newly created alt.* section. This change was intended to solve technical problems and “reduce the so-​called signal to noise ratio,” which had been exacerbated by vague or overlapping group names. However, in addressing these user experience issues, the new network structure made it easier to limit off-​platform harms. Many internet service providers blocked access to alt.* newsgroups that allowed filesharing to avoid liability for copyright infringement.

These early USENET decisions were undoubtably content moderation, however they were approached as solutions to technical standards problems rather than to social conflicts or political controversies. While many of their deliberations were carried out in public, the distance between the backbone cabal and everyday USENET users was in some ways more similar to contemporary moderation at scale than the local moderation that governed most forums of the 1990s and 2000s. This later period of moderation was much more explicitly intended to solve social, rather than technical, problems.

A 1993 Village Voice article, “A Rape in Cyberspace,” documents an account ban in LambdaMOO, a roleplaying chatroom. This early account of content moderation describes a personal, communal process, not unlike an ancient Icelandic moot. A player named Mr. Bungle used a voodoo doll “item” to control the actions of other players, putting their characters in unwanted sexual situations. This was very upsetting for other users who had emotionally invested in their characters and didn’t realize they could be victimized in such a fashion.

Users organized a meeting to discuss the problem. Many wanted Mr. Bungle gone, but none had the power to expel him. After two hours of discussion, a LambdaMOO programmer used authorities usually reserved for website maintenance to delete Mr. Bungle’s account. In the aftermath of the unilateral (but widely supported) expulsion, LambdaMOO’s creator implemented a voting system to govern future use of such powers. Crucially, the experience transformed LambdaMOO into a political community by forcing it to make a collective decision. Prior to the Mr. Bungle’s transgression, LamdaMoo had no laws or means of enforcing them. While still without formal laws, LamdaMOO developed a democratic procedure for adjudicating social conflicts.

Democratic deliberation is difficult, tiresome work and, in the early internet, prone to hijacking. As the internet grew, individual interactive spaces proliferated. Many early forums were bundled with internet service provided and administered by AOL, Compuserve, or Prodigy. Some were independent websites. Rather than belonging to, and being governed by, broader networks, they were standalone operations. Forums began to publish explicit rules outlining prohibited behavior. Instead of being governed by ad-​hoc democratic moots or a faceless “backbone cabal,” they were governed by local superusers or forum operators. This is when “mods” came into vogue as internet slang for moderators, referring to the group collectively responsible for decisions on a forum.

As content moderation became more routine, certain stable aspects of the practice emerged. Human content moderation is retroactive—the volume of internet speech is such that it cannot not easily be reviewed before being publicly posted. While the forums of the 90s and aughts were smaller than many modern platforms, they were also governed by smaller and less professional moderation staffs. It was common for users to get away with violating the rules during the hours when the mods were asleep.

Retroactive human review is often coupled with automated, immediate moderation by machine. In the 90s, this technology was limited. Forums used word filters to prevent the use of common obscenities, prohibited the registration of accounts with email addresses from certain domains, or filtered spam based on the frequency or origin of messages. These systems might have been completely automated, or have a human in the loop, reviewing potentially violative content flagged by a machine. While contemporary algorithmic moderation may draw upon a wider set of more nuanced cues, it still struggles to appreciate even basic conversational context.

Early misapplication of these tools and human review illustrates that content moderation has always been an imperfect process. Moderators subjectively interpret rules, or act on the basis of personal biases. In smaller forums, the line between application of the rules and moderator preference is often blurry. Early commercial word filters produced errors that wouldn’t seem out of place on Facebook today – banning the use of words that appeared in users’ names, or preventing dog breeders from using “bitch.” Then, as now, content moderation often amounted to a decision about the acceptable ratio between false positives (removals of innocent content) and false negatives (failures to remove rule-​violating content).

Moderators began to develop rules particular to the structure of their forums in response to novel misuses. In early live AOL chatrooms, users shared a common chat scroll. Users who spoke too frequently or spammed short nonsense messages could scroll the chat for everyone, precluding real conversation. In response, a 1998 AOL Terms of Service update prohibited using an AOL account to

This has since become the standard process for generating new content rules. While some misuses can be anticipated ahead of time, users will always find unintended applications – such as using a shared Google doc as a chatroom - that prompt new concerns and rules. Once the basic purpose and rules of a platform or forum are sketched out, most rulemaking occurs in reaction to new, unanticipated, misuses.

Moderation, and new platform rules, also come in response to legal or legislative developments. Two American civil suits in the early 1990s, Cubby v. CompuServe and Stratton Oakmont v. Prodigy Online, disincentivized any moderation at all. In Cubby v. CompuServe, the court shielded CompuServe from liability for users’ speech because it did not offer content moderation, while in Stratton Oakmont v Prodigy Online, Prodigy’s provision of moderation rendered it liable for its users’ speech, similar to a traditional publisher. This created an unsustainable situation; offering any moderation rendered intermediaries liable for anything they failed to catch. This “moderator’s dilemma” was rectified with the passage of Section 230 as part of the 1996 Telecommunications Act. Section 230 precludes lawsuits that treat web intermediaries as the publishers of users’ speech, while shielding platforms from litigation of their content moderation decision. While Section 230 recognizes platforms’ broad rights to set rules for their services, it also ensures that platforms are not required to moderate lawful speech, preventing litigation from putting a price premium on liberal platform governance. This arrangement has persisted with little modification even as the scale of content moderation has exploded over the past twenty-​five years.

Nevertheless, law still sets the limits of platform policy. In 2000, a French NGO sued Yahoo for allowing the sale of Nazi memorabilia on its auction sites in violation of French law. A French court ordered Yahoo to prevent users who were accessing the site from France from viewing these auctions. Yahoo complained that it could not effectively screen out French users, and it eventually prohibited the sale of Nazi memorabilia for all users. Platforms want to serve users from multiple countries, and it can be difficult to effectively exclude users from a particular country from viewing particular content. Platforms often face an unpleasant choice between fragmenting their network and applying the most restrictive local rules sitewide. LICRA v. Yahoo! dashed utopian hopes of a single global internet, which John Perry Barlow described in his 1994 Declaration of Independence of Cyberspace as “a new home of Mind” over which governments would have no sovereignty. Declarations aside, servers and users reside in particular jurisdictions, placing content moderation in the shadow of the state.

In the early 2000s, what was once an internet of hobbyists, largely drawn from particular western subcultures sharing certain informal norms, became an internet of everyone, regardless of class, color, or creed. In 2000, AOL had 23 million users. By 2008, both Facebook and Myspace had more than a hundred million. As more people began to use the internet, the scale of some general-​purpose social media platforms ballooned. These services leveraged network effects to supplement users’ personal social networks, allowing speech to “go viral” as it coursed between personal networks. Instead of pseudonymously visiting a forum, online socialization was increasingly linked to users’ activity in the real world. By 2006, model Tila Tequila had accumulated 1.5 million followers on MySpace, becoming the first of what we would now call “influencers” and inaugurating a new era of social media. Time magazine technology writer Lev Grossman presciently observed, “Pre-​Tila, your MySpace friends were mostly people you actually knew. Post-​Tila, the biggest game on the site became Who Has the Most Friends, period, whoever they might be.” She joined the platform after being repeatedly kicked off then competitor Friendster.

Even as the dynamics of platform scale and use shifted, the basic principles of content moderation remained the same. It was conducted imperfectly, post-​hoc, largely by human reviewers whose efforts were supplemented with content identification software. In practice, most early massive platforms were liberally governed. MySpace and early iterations of contemporary giants Twitter and Facebook just didn’t have the resources to moderate extensively. were liberally governed. These platforms’ initial rules were quite limited, and they had few staff to address complaints or review user speech. While the rules gave platforms license to take action, many were rarely enforced. Testifying in a 2008 lawsuit over an impersonating profile, a MySpace vice president acknowledged that despite a rule against fictional accounts, “MySpace has roughly 100 million unique members and at least 400 million profiles.”

At this scale, moderation became much less personal—unless you had a particularly large following—and moderators had little time or incentive to address individual appeals. In the early platform internet, conflicts between superusers and their legions of followers drove the creation of rules around harassment and mass interactions.

New platforms grew, some offering more specialized approaches to content presentation and moderation, establishing mass networks for photosharing, videosharing, livestreaming, fanfiction, fundraising, podcasting, knitting, dating, and many other purposes. The unique structures of these new offerings created new moderation demands. For instance, because live video is more difficult to moderate than static video, Twitch is forced to rely on account governance while YouTube can remove discreet videos at greater leisure.

More important than resource limitations, platforms faced less pressure to moderate during this era. The internet wasn’t as politically salient, and free communication was usually taken to be an unalloyed good by American elites. The Arab Spring was seen as evidence of an almost natural relationship between internet freedom and international liberalism or the achievement of American foreign policy goals. Before Edward Snowden revealed the American government’s surveillance of global web traffic, there were fewer concerns about internet surveillance and data gathering. In 2012, when the Obama campaign mined Facebook “friends of friends” data to inform advertising and turnout efforts, it was celebrated as canny digital campaigning. Four years later, Cambridge Analytica offered ad targeting tools based on “friends of friends” data to the Trump campaign, and it became a major scandal.

This was not simply a result of anti-​Trump attitudes, though the shock of his victory certainly set the stage for conspiratorial explanations. Rather, the Islamic States’s networked jihad, populist electoral victories around the globe, and the success of the Brexit campaign in the United Kingdom had all worked to sour elite attitudes towards a liberally governed internet. Livestreamed shootings, scams, data leaks, revenge porn, and cyberstalking illustrated how platforms could be misused. Indeed, because formerly offline misbehavior could be more easily tracked and recoded online, the internet was often blamed for causing problems that it merely revealed. As the rest of the world came online, platforms increasingly had to accommodate greater differences in values between users.

Advertisers responded to these misuses as well. Modern platforms are, for the most part, funded by advertising. Advertisers bid to show ads to users likely to buy their products in real time auctions and ad placement is handled by software. Major corporate advertisers balked at having their products shown alongside violent, hateful, or dangerous content, incentivizing platforms to ban or demonetize (refrain from running ads alongside) content that might offend advertisers. YouTube faced particular pressure to clean up its site, spurring successive waves of demonetization.

After 2016, platforms were increasingly expected to control the real-​world effects of misuse. While content moderation began as a communal or local process, intended to provide a positive user experience, by 2017 platform executives were routinely castigated in Congress for failing to prevent the Russian government from serving advertisements to Americans. For the most part, major platforms have acquiesced to these demands, hiring tens of thousands of moderators, expanding the use of algorithmic moderation, and creating new rules and procedures to address hate speech, false speech, election interference, and dangerous organizations. With billions, rather than millions, of users, the largest platforms have taken on broad civic responsibilities. They have formed cross-​platform compacts to improve their governance, ease moderation burdens, and forestall criticism and government regulation. While these compacts were initially limited to illegal pornography and terrorist content, they have expanded to cover violent content and domestic extremism. Major platform moderators increasingly prepare for controversial events, designating high risk locations and temporarily increasing the stringency of their algorithms.

These changes have provoked political criticism, particularly from the right, which understands the increasing stringency of private content moderation to be biased or directed against them. Several platforms, such as Gab and Parler, were created as ostensibly unbiased alternatives. While these platforms offer legitimate alternatives to more mainstream offerings, their content moderation policies reflect the values and preferences of the conservative subcultures that inhabit them. Courting pro-​Trump politicians and inundated by trolls, Parler prohibited parody accounts impersonating public officials. Fears about the misuse of alternative platforms and their liberal content policies prompted moderation deeper in the “stack” of applications that make up the internet. After the January 6^th riot at the Capitol, Amazon Web Services ceased hosting Parler, briefly knocking the site offline.

Emerging decentralized social media protocols have been proposed as a solution to the limits of centralized moderation. Private, encrypted messaging and group chat apps offer another exit from the increasingly moderated social media commons. Both approaches eschew scale in the pursuit of better, or more user-​centric, governance, taking content moderation back to its roots. However, these approaches have faced criticism from those who expect intermediaries to moderate content everywhere in the public interest.

Over the past 40 years, content moderation has matured and expanded. What was once an informal, local, and communal enterprise has been formalized and professionalized. Moderator-​operators have been replaced by salaried workers, bureaucracies, and advanced (although imperfect) algorithms. While moderation once concerned only the political community of a particular forum’s users, it is now regularly implicated in real world politics. It is a novel form of private governance, limited to speech, but coming to encompass more as we move greater amounts of our social and economic activity onto the internet. At its best, it can offer many competing, overlapping mechanisms for dispute resolution and filtering, each optimized for a different purpose. At its worst, it can allow governments to escape constitutional limits on speech regulation or import foreign censorship. If content moderation is to continue to develop in a liberal direction, it must happen across many platforms independently, without expectations of comprehensive reach or perfection.