The Rice Cooker 電飯煲

About the Rice Cooker | Journalism and Media Studies Centre | JMSC GitHub

WeiboScope: image search by keyword

2011-12-25 23:47 | Cedric Sam

You might have heard of WeiboScope for its display of most important images by a sample of users we selected. “So what?”, some users have asked. WeiboScope is a suite of visualisation tools for an archive of Sina Weibo posts that we collect and store on a local database, which may currently range in the 2-3 million per week.

But the power of WeiboScope is not this particular visualisation (because there are many of them), but rather the data underneath that sustains it. Rather than let Sina Weibo dictate the way the data produced by users should be displayed, we borrow a bit from the open data movement and repackage posts in ways that may be a bit more useful to users. This is how a WeiboScope search by image came to be.

Consider these current use case scenarios:

1- A non-Chinese reader would like to know what the Chinese Weibosphere is now thinking about the death of Kim Jong-il. They can decide to type the Chinese name of Kim Jong-un in the search bar on Weibo.com and find a list of about 25 weibos. But because they are unable to read, they rely only on images. They feel lost, and give up on Weibo (for the day).

2- A person who has a native level of Chinese is doing research on suicide. Some cases are reported to be made viral on the Internet, sometimes because of the fake attention-seeking nature of them, or sometimes because their causes provoke deep societal debates. The researcher searches on the search bar on Weibo.com, finding sometimes irony, and some irrelevant news. It is hard for him to assess the importance of such case with respect to others within a certain period of time.

Now, consider that we had a sample of all Weibos ever produced and that our search engine is neutral as to what gets shown and what does not.

Scenario 1: Using the image search on WeiboScope, you can now find that one of the most popular images used in posts was this one. But then, by visual elimination, you may also notice some more odd pictures such as this one speculating on the younger Kim’s Christmas activities.

Scenario 2: Using the image search on WeiboScope, the researcher searches the word “suicide”. In March 2011, we tried this with an early version of this tool. Just by curiousity, we heard of this schoolchildren suicide case in Fujian through the popular image aggregation. At this point, we only saw one post that made it to viral level. We were curious of the impact of this case on the Chinese Internet, so we searched the characters for “suicide” on the search engine. The result? About 80% of the recent posts with the characters for “suicide” were related to the Fujian case.

The WeiboScope image search demonstrates that when you are allowed to mash and mix, and remix data, it may lead to some discoveries and realizations that may not have been made possible otherwise.

http://research.jmsc.hku.hk/social/obs.py/sinaweibo/#search

(For non-Chinese writers, the engine supports some automated Google Translate translation! For people searching in Chinese characters, please use quotes around your characters.)

Posted in Data, Data Visualisation, Online Social Networks | No Comments »

Android Ice Cream Sandwich on a Pandaboard

2011-11-28 08:11 | Cedric Sam

I got myself a Pandaboard a few weeks ago. It’s a community-supported fan-less board sponsored by Texas Instruments. It sports their OMAP 4430 chip, which is the same CPU as found in the Kindle Fire, Motorola Droid RAZR and a bunch of other smartphones (the Galaxy Nexus has its successor, the 4460).

So, when I first got it, I installed Ubuntu Linux (Linaro) on it. Used it very little, and sort of figured out the basics, such as installing an Apache webserver, but finally noticing that it lacked support for things I wanted to try out, such as Google Video Chat (which is not yet available for an ARM architecture, the one commonly found in most smartphones today).

So, I instead followed instructions on a YouTube video from the Pandaboard website that said you could install Android 4.0. And turns out you could, by following the instructions (you can find clearer instructions on the Web). So, now I have Android 4.0 Ice Cream Sandwich on the board… Next step is to figure out how to get (or just wait for) the Google Apps (Gmail, Gtalk, etc.), and support for basic hardware such as video and audio capture.

In terms of media and journalism, there is perhaps some potential to create new ways to interact with information, by plugging a projector and some sensors to detect human input. In computing power, the Pandaboard is probably as powerful as a top of the line smartphone, yet at a much lower price of US$178 (but then, so flashy touchscreen). The form factor is interesting for embedded systems, which is something I only discovered this year.

Posted in Hardware, Open-Source | No Comments »

Spawn more overlords?

2011-11-16 17:18 | Cedric Sam

One of the biggest challenges in the project has been I/O. Throughout the networks that we check, we deal with large amounts of data that we need to write and read at every moment.

Lucene is a quick way to search through text, including that in Chinese language. We used to rely on the database to do this, but it turned out quickly to be terribly inefficient. To do a search, you had to visit every row (within the parameters given) and search for whether a term appeared.

We asked our HKU colleagues in the computer science department for help, namely Reza Sherkat, a former IBM employee, now a post-doctoral fellow with Nikos Mamoulis. He had previously given us advice on inverted indexes, which in a nutshell uses tokens of text (from the weibos, say) as keys in a gigantic array. The values in each element are what were the indexes in the table or type of objects that we are indexing (for weibos, it would be the weibo ID).

So, when you search a word, you effectively only go through a list of unique words/tokens, which returns a bunch of weibo IDs.

The second trick Reza told us about was the use of programs running in the background, or what are commonly called “daemons”. Like daemons, they are always there, waiting for a program to call it. A use we could make (or should make) would for instance be to keep a list of user IDs in memory. If you want to know whether a weibo was made by a user, no need to go to the database to check. You can do all of that in memory.

There are probably some more clever uses, such as for counting or going through large numbers of items.

It is known for instance that for Google and Facebook to achieve their levels of efficiency that all the data that passes through in fact just stays in memory. And the problem with memory is that it requires an electric current to stay alive. A power outage (which we think should never happen) and the data dies.

Operating in memory (in RAM, that is) is much much faster than having to fetch from a disk. It should make a difference, and we shall try it on our 48Gb of RAM.

Posted in Data, Online Social Networks | No Comments »

New attributes in Sina Weibo API’s V2

2011-11-12 18:52 | Cedric Sam

Sina released a second version of their API about a month ago. It’s good to mention that we have moved some of our crawling scripts along to V2.

Of our interest, the status entity now had the following new attributes that can be used:

reposts_count
comments_count
mlevel

The first two are self-explanatory. The third probably means “maturity level”. We’re happy to get the first two and think it was about time that Sina start giving us exact numbers. To their defense, repost (and comment) numbers on Sina Weibo are much much higher than on Twitter, because status entities are much better preserved on Sina (on Twitter, those attention hoggers just keep re-writing posts to include their names).

allow_all_comment
avatar_large
verified_reason
bi_followers_count
verified_type
lang

It’s to be noted that the last two, verified_type and lang, were not documented yet, and I saw them just this afternoon (and promptly made it be reflected on our scripts). They are self-explanatory. Unlike for Twitter, Sina verification can be of several levels. My Weibo account is “verified”, but just because I was verified to be a JMSC employee (not because I am famous, bah). There are corpo accounts that get a different kind of verification, and there are probably others that I’m not certain about (power users?). “lang” is very interesting. We have mobile clients in English; we have Web interfaces in Traditional (for Taiwan) and Simplified Chinese (for the mainland). So, is Sina really preparing international versions?

***

After several e-mails from the audience, we do acknowledge that we also faced some auth problems, but were also lucky to have started the project early, such we don’t face some of the other problems (such as the need to specify in the devapp whether we are foreigners). I don’t know if it has anything to do with the rate limits you end up getting.

We’ve also had a few problems with the friendship (friends / followers) functions in V1. Those are still there in V2. Namely, the site won’t work with just OAuth. You also had to have cookies (thus a Web browser accessing the API URL, while signed on your Sina account). If you see some inconsistencies, feel free to e-mail us.

We’re submitting our social media project final report this week. So, expect in the next weeks (not months, I hope) that we release the tools we developed in the wild pretty soon. Some of them are already up on our GitHub.

Posted in Online Social Networks, Open-Source, Scripts | No Comments »