The following is a map of completed buildings in Hong Kong from 2005 to 2011, according to data from Buildings Department as processed by us (errors may occur): http://opengov.jmsc.hku.hk/datamap/#completed It is still being worked on as we speak, so bugs might also be found.

Using a text processing tool on Linux, we extracted the text of every section 5.6 of Buildings Department’s PDF monthly digests (here’s one of the 80-something published between 2005 and 2011).

The data was cleaned with Google Refine to the best of our capacities, and mapped with Google Fusion Tables according to the town planning units. We couldn’t do with addresses were often messy, and those in the New Territories often referred to their lot number only.

There are still lots of semi-open data (because not originally in a raw text format) provided by the Hong Kong government that could use a bit of repackaging job. We’ll get back to you soon on this.

Despite the PDF documents being “protected” (preventing copying when opening with Acrobat), you can use a common utility for Linux called lesspipe, a pre-processor for less, that can process many file types into readable text.

Readable does not necessarily mean structured. By no means, the lesspipe output is usable as it (it looks like this after separating the sections and aggregating across different PDF files). With the fantastic Google Refine tool, you can however try your best to parse the data, clean the different fields manually and then even perform geocoding inside the tool (with “Add column by fetching URLs”).

After the cleaning was done (it took a few hours last Thursday, and a few more hours today), I did an export in TSV, and sent it to Google Fusion Tables. I customized the map visualisation with the “month” field, and here is the result:

2005-2011 data for “Table 5.2 Buildings for which building authority has issued demolition consent” from Hong Kong Buildings Department’s monthly digests (alpha)

This is not even close to our final product yet, because the Google Maps JavaScript API V3 now lets you add layers from Fusion Tables data. Effectively, it means that you can build Web applications with different kinds of filters (in pull-down menus, etc.) that dynamically change how the data is displayed. The example here above only shows the single view specified inside Fusion Tables by the owner of the table (me). You could take possibly use the ID of the table (3546150) and make your own visualisation.

For now, the data hasn’t been vetted after refining (maybe the govt will provide us with raw data?), so I would recommend using with high caution as to the validity of the data. It should be largely correct, but some data points may not have been geocoded properly, if at all. For this particular data, corrigendums to Buildings Department monthly digests are not yet taken into account.

Here is another Google Refine + Google Fusion Tables trick on Hong Kong government data:

Map for data from “Short Term Tenancy (STT) Tender Forecast” from Hong Kong Lands Department (alpha)

These are the Short Term Tenancy (STT) Tender Forecast from Lands Department. They are the sites for sale on short term tenancy, for a few years, for uses such as car parks. The color code on this custom map is based on the square meters area of each site for sale (from purple 0-1000 sqm to red for 5000+).

A year and half ago, I did a project called Twitterball for one of my classes at PolyU on information architecture. It played on the idea of visualising the frequencies of tweets in time.

I quickly remashed Twitterball into Weiboball, using data from our archive and the search engine built with WeiboScope Search. The result is Weiboball and Weibobubble.

[Edit (12:30PM): There was no shutdown of trending topics (thanks Charlie of Chinageeks), but I noticed that their weekly trending topics never seem (from visual inspection) to include posts made in the 3 past days. I was also confused by the daily topics, because I wrote the entry before 9AM and only saw the topics of 2 days ago as "yesterday's" trending topics, skipping one entire day. It's possible that the trending topics are only released once or a handful of times per day.]

The image embedded here below is a screenshot of the most commented on weibos of the last week on Sina. They are based on the number of comments made on the given post that week, but are all on rather innocuous topics this time.

The following is a screenshot of the page for the most reposted weibos during the last week (original page). Sina counts them based on a week time from today, and a calendar lets you navigate through the archive.

It’s not always inoffensive stuff, as sometimes the posts would touch on social injustice and events of political importance, like here. But don’t look for the Bo Xilai and Wang Lijun stuff, because you won’t find any of this. That said, not everything has to be political to be important, and celebrities posts often occupy the microblogosphere of the majority.

So it is a good thing that we are keeping our own trending topics.

We have been making our own index of trending topics for a long while already (more than a year now) and while it chiefly depends on our capacity to collect posts, it has always given us a good indication of what’s *really* trending, among people of slightly greater influence (we have a list of 270,000 people now, with more than 1,000 followers).

We solely look at popular posts based on the number of reposts (among a sample), because it’s not practical to do trending topics based on comments, when you don’t have the capacity to discover popular posts that way.

Our trending topics come in handy when the comments are completely gone, as it was the case this week on both Sina and QQ weibo, and reportedly most of the other microblogging platforms. WeiboScope is a visual representation of those trending posts, according to us. For instance, there is a strong representation of pictures of buddhist monks, either of what seems to be one of their leaders, and another of what looks like two monks deviating from their monastic life (see archived screenshot).

Also a bit of Yao Ming, cats growing old together and some ridiculous freak incident where a lady fell in a hole in the pavement where hot water pipes had ruptured.

The image is only indicative, as I didn’t check the actual sampling. But because of its consistency over time (in terms of matching Sina’s own trending topics or what I end up seeing in the news), I can believe that this would be what is interesting among a certain group of slightly more influential people. That’s what the chatter’s on right now on Sina Weibo.

新浪微博公告

各位微博用户：
　　最近，微博客评论跟帖中出现较多谣言等违法有害信息。为进行集中清理，从3月31日上午8时至4月3日上午8时，暂停微博客评论功能。清理后，我们将再开放评论功能。进行必要的信息清理，是为了有利于为大家提供更好的交流环境，希望广大用户理解和谅解。感谢大家的支持。
新浪微博

2012年3月31日

Weibos are a bit more “Twitter-esque” now, since comments are a unique feature of Chinese microblogs over their Western counterparts. It’s especially a critical feature, since reposts are often made through a comment first (you may then decide to repost something back to your readers) and allow to aggregate text of the same topic within a single stream.

Tencent had a similar message for their weibo, also one of the most popular ones in the nation:

腾讯微博客公告
關閉

用户朋友：
　　近期，通过微博客传播的谣言等违法有害信息造成了不良社会影响，评论跟帖中的有害信息相对较多，需要作集中清理。为此，本网站决定，自3月31日上午8时至4月3日上午8时，暂时停止微博客评论功能。由此给您带来的不便，敬请谅解。

腾讯网

2012年3月31日

The networks do not point fingers at a specific target of this “cleanup” in their message to users, but many understand that it is to rid their weibo systems of the chatter on a supposed “coup” in Beijing.

As illustrated in The Economist this week, but understood by any Chinese speaker, 140 Chinese characters are worth a lot more semantic value than in alphabet-based languages.

We’ll try to to monitor the blackout and keep you posted on it.

It was made as a repost made on March 2nd by David Bandurski of CMP, and was a repost of a Hu Shuli post commenting on the Wang Lijun incident on the same day. The post was archived, but can no longer be found on David’s timeline. Ms. Hu’s post was still alive on Weibo when I checked (and we have it archived, in case).

The strange thing was that this post was not marked as “permission denied” and was fully available when queried with the “statuses/show” function on the Weibo API (link — requires login). On the other hand, the user_timeline function, which lists the latest 200 posts made by a given user, reports that the post no longer existed.

Even more bizarre, still, when you searched on Weibo.com for David’s posts between the date range containing March 2nd, you’d see that the search page claims of 5 results, whereas only 4 are actually returned! This is shown in this following image:

For our monitoring tool, specifically the user_timeline function that casts the net for deleted posts, we have been using the first version (V1) of the API (because of rate limiting issues). V1 seems to me like the abandoned entrance for a data store that is more and more complex in its layers of visibility (and invisibility). These layers don’t seem supported with 100% fidelity on V1, and will return copies of user timeline that sometimes contained the “ghostly” post, and sometimes just won’t (probably depending on which physical server you end up accessing).

Not only there are “permission denied” or “weibo does not exist” posts, as described in our previous post about the methods behind the tool, but there are now posts wavering between a state of publication or non-publication (at least from the weibo.com website’s point of view, they were dead).

Anecdotally, I noticed that “permission denied” posts usually meant that their dependent posts (by people who repost) would also be marked as deleted with “permission denied” as well. In this case of the Hu Shuli post, the original post was alive and well, while the repost was deleted.

Ever looked for an automated archive of Chinese news websites? For months, we’ve been collecting screenshots and HTML snapshots of up to 20 websites based in China or covering China. We now have a webpage for it.

http://research.jmsc.hku.hk/social/chinanews/index.py/listScreenshots/

The screenshots are classified by news source and with a minimalistic (if not just minimal) interface, organised by day and regrouped by month. For instance, you could go to the QQ News archive, an archive for February 2011, and a particular link to today.

There’s also a version for accessing navigeable HTML pages when available.

With the extensive archive of weibos we’ve accumulated so far and mechanisms underlying its retrieval, we were able to develop a routine that finds and marks deleted posts (method explained here).

The result is an archive of deleted posts (and the CMP’s Anti-Social List). Not only is it possible to find a large number (which is not exhaustive, we admit) of such posts within the day of their doom time, but also be able to presume with clear-cut evidence whether these deleted posts were removed by the user itself or simply deleted by system managers (we first noticed the difference in August 2011…).

As described in a post last week, the idea behind this archive is simple and straightforward to implement, once you’ve got the infrastructure.

A previous copy of the user timeline, containing all posts

A current copy of the user timeline, with a missing post

Both copies of a user timeline (post IDs extracted from the full JSON response) are obtained during two consecutive API calls, which may span a few minutes or several hours. The smaller this interval between pollings, the more precise would the routine be in finding the exact removal time (and the chances of missing something, smaller too).

A post is found to be deleted when you could see it in the previous version, but not the current one. Since we keep a copy of every post we see, we simply mark this post, and can then view them all in a custom webpage. Easy enough, right?

The previous two images show you exactly what, from a programmer’s point of view, Sina Weibo returns us for two different kinds of deleted posts. The former, with an API response “weibo does not exist”, identifies a post that was presumably deleted by the user. The latter, which returns “permission denied”, is presumably a post deleted by the system.

We don’t know the intention between both types of messages, but we can guess based on what their contents are generally. The first are generally made of spam-like posts that would be deleted on any online social network in the world. The second seem to provide more legitimate contents, including some made by so-called VIP users verified by Sina (we only check a 2500-odd sample of users, so can’t really infer on their representation).

This feature is indeed powerful, because it finally puts a number on post removal on Sina Weibo. (We computer science majors strongly dislike conclusions not based on numbers and data.)

It is however currently impossible to tell with certainty what gets deleted (versus what’s not), since our user sample is strongly biased towards public commentators, and perhaps because the number of posts found is still extremely small.

What it does give is an understanding of how post removal works, how much time it usually takes for something to be removed, and whether reposts of posts or just reposts but not original posts, get deleted (in fact, it happens). It’s a privileged peek, indeed, at what’s going on on the Chinese Internets, right here, right now.

Until we accumulate a substantial archive to do anything useful, our colleagues at China Media Project have started compiling and explaining deleted posts on their Anti-Social List.