{"id":87,"date":"2013-06-16T18:12:23","date_gmt":"2013-06-16T16:12:23","guid":{"rendered":"http:\/\/heavenstone.net\/jesse\/?p=87"},"modified":"2025-08-01T19:00:59","modified_gmt":"2025-08-01T17:00:59","slug":"how-many-articles-on-computer-science-can-there-possibly-be","status":"publish","type":"post","link":"https:\/\/heavenstone.net\/jesse\/how-many-articles-on-computer-science-can-there-possibly-be\/","title":{"rendered":"How many articles on computer science can there possibly be?"},"content":{"rendered":"

Meet the PAF Peacock<\/strong><\/p>\n

\"peacock\"<\/a><\/p>\n

I just got back from a 3 day trip at the beautiful and mysterious PAF<\/a>, in the countryside near Reims, about 2 hours east of Paris. The place is made for artists, dancers, and musicians to work, think, and play. And at only 16\u20ac a night, an incredible deal. An ancient monastery, it’s still filled with objects of previous grandeur like out-of-tune pianos and stringy tapestries, but who can say no to a kick-ass ping-pong table?<\/p>\n

But as their website says, it’s a place for “production” not for “vacation”. And that’s what the 9 of us were there to do. Import Wikipedia into KnowNodes, visualize it on a graph, and let users quickly find these articles when making connections.<\/p>\n

Know your Nodes<\/strong><\/p>\n

\"\"<\/p>\n

So what is this KnowNodes project<\/a>, you may be asking. Dor Garbash<\/a>‘s dreamchild, it is a connectivist orgasm, a sort of map of human thought and knowledge, but focused the connections between resources (scientific articles, blog posts, videos, etc.) rather than on those resources themselves. Students could use it to find new learning resources, researchers could use it to explore the boundaries of knowledge in their field, and the rest of us just might love jumping from one crazy idea to the next.<\/p>\n

Much like a new social network, one recurring problem with getting this kind of project of the ground is it needs good quality content to jumpstart it. And what could a be better source of quality information as the world’s largest crowdsourcing project ever, Wikipedia?<\/p>\n

Now down to the gory details. How big is Wikipedia, anyway? Well, according to the “Statistics” page<\/a>, the English-language site is at 30 million articles and growing. Wikipedia (and the Wikimedia platform behind it) is very open with their data. You can easily download nightly data dumps of their database in different formats<\/a>. But here’s the rub: the articles alone (not counting user pages, talk pages, attachments, previous versions, and who knows what else) still weighs in at 42 GB of XML. That was a bit too much for our poor little free-plan Heroku instance to handle.<\/p>\n

So, we came up with a better idea: why not just focus on a particular domain, such as computer science? That way we could demonstrate the value of the approach without overloading own tiny DB. Now, we realized that we couldn’t just start at the computer-science article and branch outwards, because with the 6-degrees nature of the world, we would soon end up importing the Kevin Bacon article. But Wikipedia has thoughtfully created a system of categories and sub-categories and sub-sub-categories, and anyway, how many articles under the Computer Science category<\/a>\u00a0category could there possibly be?<\/p>\n

1+6+2+5+9+17+3+34+…<\/strong><\/p>\n

Hmmm, let’s find out. We wrote a node.js script that uses the open Wikimedia API<\/a>. The only way to find all the articles in the Computer Science category hierarchy is to recursively ask the API for the categories within it, do the same with its children, and so on, until we reach the bottom.<\/p>\n

The nodemw<\/a><\/strong>\u00a0module came in really handy, as it wraps many of the most common API operations so you don’t have to make the HTTP requests yourself. It also queues all requests that you make, and only executes one at a time. That prevents Wikipedia from banning your IP (which is good) but also slows you way down (not so good).<\/p>\n

Enough talk, here’s what we came up with:<\/p>\n