The Rice Cooker 電飯煲 The JMSC's Computational Journalism Blog

It’s the holy grail of any media researcher working on China: how do you quantify contents removal from social media services such as Sina Weibo? Here at JMSC, we’ve been developing tools to scour and assess social media of all sorts for the purpose of researching online media in Hong Kong and mainland China. (This is the same project that generated WeiboScope, for those tuned in.)

With the extensive archive of weibos we’ve accumulated so far and mechanisms underlying its retrieval, we were able to develop a routine that finds and marks deleted posts (method explained here).

The result is an archive of deleted posts (and the CMP’s Anti-Social List). Not only is it possible to find a large number (which is not exhaustive, we admit) of such posts within the day of their doom time, but also be able to presume with clear-cut evidence whether these deleted posts were removed by the user itself or simply deleted by system managers (we first noticed the difference in August 2011…).

As described in a post last week, the idea behind this archive is simple and straightforward to implement, once you’ve got the infrastructure.

A previous copy of the user timeline, containing all posts

A current copy of the user timeline, with a missing post

Both copies of a user timeline (post IDs extracted from the full JSON response) are obtained during two consecutive API calls, which may span a few minutes or several hours. The smaller this interval between pollings, the more precise would the routine be in finding the exact removal time (and the chances of missing something, smaller too).

A post is found to be deleted when you could see it in the previous version, but not the current one. Since we keep a copy of every post we see, we simply mark this post, and can then view them all in a custom webpage. Easy enough, right?

The previous two images show you exactly what, from a programmer’s point of view, Sina Weibo returns us for two different kinds of deleted posts. The former, with an API response “weibo does not exist”, identifies a post that was presumably deleted by the user. The latter, which returns “permission denied”, is presumably a post deleted by the system.

We don’t know the intention between both types of messages, but we can guess based on what their contents are generally. The first are generally made of spam-like posts that would be deleted on any online social network in the world. The second seem to provide more legitimate contents, including some made by so-called VIP users verified by Sina (we only check a 2500-odd sample of users, so can’t really infer on their representation).

This feature is indeed powerful, because it finally puts a number on post removal on Sina Weibo. (We computer science majors strongly dislike conclusions not based on numbers and data.)

It is however currently impossible to tell with certainty what gets deleted (versus what’s not), since our user sample is strongly biased towards public commentators, and perhaps because the number of posts found is still extremely small.

What it does give is an understanding of how post removal works, how much time it usually takes for something to be removed, and whether reposts of posts or just reposts but not original posts, get deleted (in fact, it happens). It’s a privileged peek, indeed, at what’s going on on the Chinese Internets, right here, right now.

Until we accumulate a substantial archive to do anything useful, our colleagues at China Media Project have started compiling and explaining deleted posts on their Anti-Social List.

Since Sina Weibo has a pretty good API, and since we do download lots of data every day, it just makes good sense to keep an archive of deleted posts.

The strategy is very straightforward and only incurs a negligible extra number of hits against the API:
- Take the statuses/user_timeline function for each user in your list (we have 2,500 in a sub-list).
- Extract the IDs of all 200 posts in the response and save as a text file, one ID per line. They are already ordered chronologically.
- You should have a previous list of IDs. Use diff to compare both files.
- Loop through the output of the diff. Mark all the IDs that appear in the previous version, but not the new one.
- Those IDs are the deleted posts.
- Mark them, and send your alert, etc. (We also hit the API again on statuses/show to double-check if the post was really deleted.)
- Overwrite the old ID list with the new one.
- Repeat whenever you can fetch a new version of the timeline (you might be rate-limited by Sina if you do it too often).

In case people noticed, the quality of our WeiboScope declined quite a bit towards the end of last week. It was just caused because of the passage to the first ISO week of 2012 (which started on Monday). Consequently, only most popular posts made in 2011 counted. We didn’t lose anything, and things are back in track.

Featured today on WeiboScope:
- The rumoured coup in North Korea
- 36 years since the death of Zhou Enlai
- Some corrupt officials at the D & G in Hong Kong?

At some point in our research project, it was a good idea to take all the users with more than a certain arbitrary large number of followers (say, 1000) and download their posts and analyze them. This doesn’t always seem to be the case anymore. Results are variable depending on the days.

We are set to release WeiboSphere, but will wait a little before pushing it. Right now, we’re taking every user with 1000 or more followers and get all their recent posts from the API. We aggregate and produce an unfiltered (at least not with a human filter) classification of the most popular posts by 24 hours, 48 hours, week, two weeks and one month.

Alas, in the last two days, all we’re seeing are female body parts, shoes and celebrities who returned to an incredibly thin size after a pregnancy.

The hope for now, until we improve the filters, is that we can see posts such as this one on an abducted girl in Guangzhou, posted yesterday morning.

One of the biggest challenges in the project has been I/O. Throughout the networks that we check, we deal with large amounts of data that we need to write and read at every moment.

Lucene is a quick way to search through text, including that in Chinese language. We used to rely on the database to do this, but it turned out quickly to be terribly inefficient. To do a search, you had to visit every row (within the parameters given) and search for whether a term appeared.

We asked our HKU colleagues in the computer science department for help, namely Reza Sherkat, a former IBM employee, now a post-doctoral fellow with Nikos Mamoulis. He had previously given us advice on inverted indexes, which in a nutshell uses tokens of text (from the weibos, say) as keys in a gigantic array. The values in each element are what were the indexes in the table or type of objects that we are indexing (for weibos, it would be the weibo ID).

So, when you search a word, you effectively only go through a list of unique words/tokens, which returns a bunch of weibo IDs.

The second trick Reza told us about was the use of programs running in the background, or what are commonly called “daemons”. Like daemons, they are always there, waiting for a program to call it. A use we could make (or should make) would for instance be to keep a list of user IDs in memory. If you want to know whether a weibo was made by a user, no need to go to the database to check. You can do all of that in memory.

There are probably some more clever uses, such as for counting or going through large numbers of items.

It is known for instance that for Google and Facebook to achieve their levels of efficiency that all the data that passes through in fact just stays in memory. And the problem with memory is that it requires an electric current to stay alive. A power outage (which we think should never happen) and the data dies.

Operating in memory (in RAM, that is) is much much faster than having to fetch from a disk. It should make a difference, and we shall try it on our 48Gb of RAM.