What The Leaked Facebook Rulebook Tells Us About Why It Isn’t Com batting Online Violence

Yesterday The Guardian published a series of articles as part of its “Facebook Files” series documenting in unprecedented detail how Facebook polices content on its global platform based on leaked copies of more than 100 “internal training manuals, spreadsheets and flowcharts” from the company. What can we learn about Facebook’s evolving role as our global communications moderator?

Immediately clear from the excerpts published by The Guardian is that despite an ever-growing series of controversies regarding how it decides what content to allow and what to banish from its servers, Facebook has yet to mature as a company and bring onboard professional expertise that can help it better understand and recognize the impossible mission it has set out for its human moderators. According to the Guardian, the company has just 4,500 reviewers on staff (though it is hiring 3,000 more) who have as little as 10 seconds to make a decision on a given piece of content. Set this incredible time pressure against the backdrop of impossibly vague and complex guidelines and it is no wonder the company has struggled so much with how to cope with illegal and harmful content published to its site.

As someone who has developed the software for and helped lead very similar large human teams reviewing vast amounts of content and making snap decisions placing that content into large complex categorical hierarchies, I can personally attest to the absolute criticality of ensuring that reviewer guidelines be as concrete as possible, leaving the absolute minimum to human interpretation.

In contrast, as one reads through the Facebook materials, one finds the same kinds of problematic guidelines we faced when dealing even with what one might imagine are relatively simplistic questions. Something as simple as categorizing a news article as describing human rights abuses can lead to sharply divergent views, even among a panel of subject matter experts, as to what specifically counts as such an abuse (is screaming at a person abuse? is sensory deprivation abuse?) and whether the actions of state versus non-state-actors constitute abuse or legitimate self-defense and so on. In short, even what appears on the surface to be a simple categorical decision can in fact be immensely complex, even when working with experts in the field.

In Facebook’s case, the mystery of why the company has done such a poor job of removing violent content has been largely answered by The Guardian’s scoop. In short – it is because the company applies the distinction of “credible threat” and divides targets into categories such as “vulnerable groups” while bringing local complications into the interpretation of what constitutes membership in each of these categories. For example, a post in the Philippines that threatens drug dealers who prey upon society’s most vulnerable and who may themselves murder others in the conduct of their business, are considered a protected class and any threats against them will be removed from the platform. Threats against “Zionists” are prohibited, but no mention is made of followers of Islam who may be threatened.

Providing advice on the best way to murder a woman is permitted, as are calls to brutally attack young children or those of certain complexions or disabilities. In fact, the Facebook guidelines offer multiple examples of threats of extreme violence against women reaching to the level of murder that are entirely permitted by the company under its official guidelines. This is especially concerning given the level of violence against women in some parts of the world and goes a long way towards explaining why Facebook and other platforms have refused to take greater action against the use of their systems to glorify and promote violence and abuse against women, including only recently taking concrete action against “revenge porn.”

Facebook’s use of a “credible violence” threshold is immensely problematic from the standpoint of curtailing the platform’s use to promote, encourage and glorify violence. Looking at just the examples of wholly permitted violent speech in Facebook’s official guidelines, it is astounding just how much of a toxic and threatening environment Facebook’s guidelines permit, all while the company publicly condemns online violence.

It is particularly striking that the same company whose leadership has focused on the empowerment of women explicitly permits the glorification of violence against women on its platform, encoding in official policy that statements like “little girl needs to keep to herself before daddy breaks her face” and “unless you stop [complaining] I’ll have to cut your tongue out” and “to snap a [woman’s] neck, make sure to apply all your pressure to the middle of her throat” along with generic abuse like “I hope someone kills you” are all completely fine and acceptable to post.

The impact of such toxic and harassing statements, even if they don’t rise to the level of an actionable credible immediate threat to life or well-being is impossible to overemphasize.

By officially codifying that such statements are entirely permissible Facebook’s guidelines defacto sanction its users to freely threaten, verbally attack and harass others, as long as the attack does not veer into the realm of what a human moderator, reading the post for as little as 10 seconds, deems is a credible attack or determines that the victim is a member of a protected class. Yet, even if not a credible threat of immediate harm, being subjected to such verbal attacks is incredibly demoralizing and toxic to one’s well-being and it is interesting that Facebook has taken the effort to explicitly sanction such behavior in its guidelines.

The intricate detail of the guidelines also appears to increasingly undercut Facebook’s arguments that it is merely a neutral content publishing platform that plays no active role in moderation beyond removing illegal or disallowed content. Facebook’s manuals hew far more closely to the kinds of standards and content guidelines that govern traditional publishers and at the very least illustrate the level of investment and control that Facebook is wielding over its platform, making incredibly nuanced and fine-grained decisions over what is permitted or not permitted.

One could argue that Facebook’s problems are merely indicative of the web at large, but whereas the web is governed by a myriad national and local governments and jurisdictions who each determine what is permitted or disallowed in their area, Facebook centralizes all of these decisions in one place and thus far has taken the approach of applying a single universal set of rules to the entire planet, instead of extending the web’s history of local enforcement. Yet, more importantly, the web’s decentralized model meant that for the last two decades one could largely steer clear of abusive content by carefully curating the sites one visited and abusive individuals tended to have relatively narrow confines through which to air their voices. Facebook, through its single centralized global publishing environment, has created a parallel web in which someone advocating for violence against women suddenly has the ability to reach out to nearly two billion people and have their messages spread virally across the planet.

The leak of Facebook’s moderation guidelines to The Guardian also raises the question of why the company has fought so hard to keep them a secret all of this time. Despite a constant chorus of voices demanding that the company be more transparent about its moderation rules and release at least basic summaries for public discussion, the company has thus far refused all requests to share such information with its user community or independent experts. While publicly referring to its platform as a global “community” that permits open conversation, Facebook has never permitted that community to see what it is allowed or not allowed to talk about, never permitted its community to have a voice in setting those rules and has steadfastly refused to offer even the smallest amount of insight into where those boundaries lie. One could argue that providing such detail offers bad actors greater insight into just how closely they can hew to the edge while remaining in the envelope of acceptable conduct, but that happens anyway as users learn what gets removed and what they can get away with.

The leak of more than 100 documents to The Guardian also raises the question of why companies like Facebook even attempt to keep such highly contentious documents secret in a world in which not even US Government spy agencies like the CIA and NSA can maintain control of their most sensitive and secretive materials. In short, in today’s world companies must increasingly consider it inevitable that such materials will leak and focus instead on transparency and openness.

In an email the company declined to comment on the materials other than to issue a generic statement that it works hard to keep its users “safe” and that it receives millions of abuse complaints every week. The company specifically declined to comment on why it has steadfastly refused to release these materials on its own or whether it would be pursuing legal action against the source(s) of the leak.

Putting this all together, Facebook appears to remain mired in its failed mindset that technology solves all problems and that it does not need to engage with the broader community of expertise around issues like moderation and human coding workflows or to engage in an open, transparent and frank dialog with its global community, hiding instead behind secret manuals, opaque decision making processes and faith in technology over all else. The company’s emphasis on vague concepts like “credible threats” and “vulnerable groups” makes it clear at long last why it has struggled so much to date with the issue of online violence and abuse against women and others on its platform.

Looking to the future, only after companies like Facebook and Twitter take the issue of online violence seriously and commit to an honest dialog and real investment will they truly be able to turn the tide on our ever-toxifying online world, but from what we’ve seen thus far, that won’t happen until the global community speaks up and finally holds them accountable.

Courtesy : forbes.com