Abusive Behaviors in Software Platforms – Trust & Safety Perspective

Guideline: This article will share some observations I have made about abusive users and contributors in the use of social media sites and crowdsourcing products, and the challenges and opportunities in uncovering these users. I hope that it would shed some lights on data analysts/scientists dealing with the trust & safety side of crowdsourcing platforms, but I think it would also be an interesting read for data analysts who want to work in fraud detection.

Ever since becoming a data analytics professional in 2015, I’ve been lucky to have learned some interesting insights and trends about abusive behaviors by users and contributors in software products. Just as users and contributors are very different participants(i.e., in demographics, age and/or behaviors) of today’s software applications, the software and products these participants engage in are also highly distinct. While I have more hands-on experience regarding abusive contributors in crowdsourcing industry than abusive users in social media platforms, I had read some researches on malicious users in social media applications. As such, I am excited to share some of my observations in both users and contributors, as well as some of the challenges (and possible solutions) in uncovering these group of people.

Before we delve deep into the topic, it’s important to highlight the difference between users and contributors in general, and the software platforms they engage in. When I mentioned the term ‘users’, I was referring to users of social media platforms (think Facebook, Snapchat, Instagram etc.). There are multiple ways users could abuse their platform; some are easier to catch than others to. I will go over some of the possible abusive behaviors after I’ve discussed the concept of contributors in this article.

The ‘Contributor’ is more complicated than users and requires the understanding of crowdsourcing first. Crowdsourcing is a business model where software companies use contributions from Internet users, usually paid, to solve organizational challenges or to obtain ideas for their data science clients. For more information, visit https://dailycrowdsource.com/training/crowdsourcing/what-is-crowdsourcing. The most common crowdsourcing practices involve training of machine learning models or autonomous algorithms. For instance, the crowdsourcing company I currently work at specializes in large-scale training data enrichment for data science clients through the help of online contributor workforce as well as application of Microsoft Azure machine learning. As the Trust & Safety Data Analyst in the company, my job is to utilize various anomaly detection techniques to uncover malicious contributors who attempt to scam our client-assigned data tasks by taking advantage of loopholes in the design of these tasks.

Now that we have a good understanding of users and contributors and the platforms they are active in, let’s talk about behaviors of abusive users on social media platforms! While some of the more well-known instances of social media abuses in recent time had centered around terrorist propaganda (e.g. FBI cracking down on Facebook users who disseminated ISIS ideology or posted links to obtain weapons in support of terrorists), most social media abuses actually involve ‘trolls’, users who post comments that are sexist, racist, anti-social etc. For those of you who follow entertainment news, you might be aware that in 2016 Taylor Swift had became a victim of bullying comments on Instagram after her verbal attack on Kanye West’s controversial song stirred the fans of the popular rapper. These type of abusive users attack the social media apps primarily to express their anger. Another type of abusive users exhibits completely different behaviors by engaging in fraudulent activities. Specifically, these cyber crime users would share links or images containing attractive commercial deals that seem legitimate but would require people who click into the link or image to submit some confidential personal information in order to get the award and/or deals. In reality, these links are used to either obtain financial information or inject virus into the victim’s computer.

So how does social media sites catch these abusive users in a time-efficient manner? The answer lies within machine learning algorithm. First, the data analysts in these social media companies would compile months (if not years) worth of data containing photos, comments and/or links from both healthy and abusive users. Depending on the size of their media platform, this data set could range from millions to a billion or more records! To ensure accuracy, the decision to label a post as healthy vs. unhealthy usually involves a little bit of human reviews since computer can be wrong sometimes. The analysts would then feed these training data into some classification and/or sentiment analysis (if unclear what sentiment analysis is, refer to this blog I wrote earlier for a refresh) machine learning algorithms to spot the most offensive content. If the algorithm thinks a content is abusive or fraudulent, it would either remove it automatically or hide it from other users. There are small nuances among social media sites in the way they handle deleted or hidden posts. Take Instagram as an example. The censored posts will still be visible to the person who wrote them, but will be hidden to other users. Whereas on Facebook, the user will usually know that their posts were deleted due to violation of terms and services; depending on severity of the incidence, some of these users’ accounts will get banned as well. Regardless of the nuances, these algorithms are designed to stop abuse and/or fraud before anybody sees it, instead of waiting for good users to retroactively report them to moderators. The sites will continually monitor and train the algorithms, make adjustments to their parameters as necessary, so that they will improve over time and reduce the false positive rates to a minimum. Instagram is especially successful in doing this. According to Michelle Napchan, Instagram’s head of public policy for EMEA, the false positive rates of abusive comment classification was less than 1%, very impressive considering the amount of data they are dealing with!

The processes to discover malicious contributors on crowdsourcing products are arguably more complicated than the ways to uncover abusive users in social media sites. To understand how crowdsourcing tasks works, it helps to understand their fundamental designs. Most of the data tasks posted by data science customers onto crowdsourcing sites involve a set of ‘gold questions’, another term for test questions prepared by the customers which the customers already have answers for. Contributors are not told which assigned questions are test questions. These test questions are used to: 1. Assess contributors’ accuracy so that contributors who fall below a predetermined accuracy threshold would get rejected by the task. 2. Ensure that contributors keep doing good work in each question since they don’t know which of the questions they are being graded on. While the first point easily fulfills its purpose, the second point might not always work the way it should depending on how the test questions are designed. My experience as a Trust & Safety Data Analyst has shown that quite a few of customers have designed tasks in a way that they open some loopholes for malicious contributors to exploit. Sometimes this is due to customers not paying enough attention to the recommended task design setting, other times they are not creating enough test questions which means sophisticated contributors with multiple accounts can get in the task and identify all test questions. Another interesting phenomenon I begin to see is that some malicious contributor in higher levels (high-level contributors in crowdsourcing platforms have access to more jobs and these jobs are usually higher-paid than jobs open to lower level contributors) had actually opened up market places on forums to sell their accounts to low-level contributors who want to level up but do not want to go through the hassle of completing the qualification tasks required to level up. Even more dramatic, one of the most notorious contributors we had caught was involved with setting up a fake marriage arrangement website and is currently being investigated by the cyber crime bureau in his home country. Long story short, abusive contributors who scam crowdsourcing tasks are often associated with other kinds of wrongdoings, most likely as a way to maximize earnings in an illegal way.

There are many possible ways to identify malicious contributors in crowdsourcing. There is not a shortage of articles on the Web about malicious contributors. For example, a paper written by University of Sheffield students (linkbreaks down the different types of scammers by behavior and discusses detection strategies such as tipping point (the moment a contributor turned from good to bad), and task completion time. But many of these paper are either too lenient or too strict in their scammer classification. The paper aforementioned is on the lenient side because the tipping point itself is hard to define, much less to identify it. My approach to uncovering malicious contributors is a little different; most of the techniques I use look at question submission behaviors on the job-level and I have found task submission time to be a poor metric (I’ve seen legitimate contributors working incredibly fast and scammers working slower than expected) at least for the tasks I have analyzed. On the other hand, contributors with high test question accuracy and low answer agreement against the most common answers often turn out to be scammers. The reason is that scammer had identified all test questions and therefore can answer all test questions correctly to stay in the job, while picking random answers for non-test questions. Another avenue I look at is shared browser fingerprint among contributors. While not always true, malicious contributors usually have shared browser fingerprints with quite a few other accounts as these accounts are needed to harvest test questions and they all belong to either the same contributor or different contributors who are colluding to game the system. These metrics have proven to be highly reliable upon manual audits. Once a profile of good and bad contributors has been gathered, you could then feed these data into a machine learning model (e.g. classification, regression etc.) and have the model assign a likelihood score to each contributor.

Now you should have a good understanding of some of the techniques data analysts/scientists used to detect abusive users and malicious contributors in social media sites and crowdsourcing platforms. Keep in mind though, the metrics I found reliable for uncovering malicious contributors worked well in my company’s crowdsourcing platform, but may not work as well for all platforms. This is why human review is necessary in the early stage to ensure that the predictions from your assumptions are accurate or at least promising. After all, you can’t afford to have low-quality training data as that would ruin the machine learning algorithms no matter how advanced the algorithms are. So this is it for analyzing abusive behaviors in various software platforms. Hope you find this article helpful and interesting. As always, feel free to leave your comment should you have any feedback or thoughts about this article.  


One thought on “Abusive Behaviors in Software Platforms – Trust & Safety Perspective

  1. A very interesting article. As the issue of scammer is becoming prominent nowadays, I believe that your trust & safety data analyst will develop more effective methods to handle the problem.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s