Contextify

Summify: top news from your networks, in your inbox

Tue, 21 Dec 2010 07:29:32 -0500

Summify: top news from your networks, in your inbox

How Contextify works, part two

Tue, 14 Dec 2010 04:27:00 -0500

Once our robots have gone out and scoured the web for new stories, they hand the fruits of their labour over to our team of software agents.

These aren’t people, of course, I just like to anthropomorphise. The agents are, in fact, a series of procedures that can be performed on each article. They act in one of two ways: either helping to understand what the article is about or discern whether it’s any good.

First comes understanding. We’ll start by looking at the obvious things about an article: the headline, the text, the standfirst, the date. Then we’ll examine some of the information that’s hidden inside articles: what sort of article it is, the length, the metadata, what categories it’s filed under, any keywords attached directly to the piece by the publisher. These give us a few clues, but not all publishers will provide much material for us there. So then we’ll perform some deeper investigation on the text itself: semantic analysis (probably using something like Calais) and looking for implicit keywords — such as the words used to link to an article (which often provide a better distillation of the topic than the headline itself). By the time all these processes are complete, we should have a pretty good idea of what an article is about at quite a deep level.

At this point, the ranking agents take over. Each of them examines an article, looks for certain sorts of information about it and then passes on a set of “clue packets” to the next agent. The point of these agents is to work out how authoritative, interesting and valuable an article is. We already know what an article’s about, so we ask: how does this stack up against other similar articles? It’s a process can be boiled down to a handful of subjective questions such as: Is it good? Is it authentic? Does it provide great context? Computers aren’t great at subjectivity, so that’s where we work hard to make magic.

Among the things we’ll analyse are the authority of an article (how highly rated it is by valuable sources), link patterns around the article (is it heavily linked or totally unknown?) and how it performs on the open web compared to its peers. We’ll also examine the sort of traffic an article receives: any spikes in activity (lots of traffic doesn’t necessarily mean great context) and social network popularity (is it shared a lot?). In addition, there’s something my friend Rev Dan Catt refers to as zeitgeistyness — how does this perform compared to the norm for that site? After all, an article on Wikileaks from the BBC is always likely to be heavily trafficked compared to that of an individual blogger like Aaron Bady. Yet it’s Bady’s zunguzungu which has become well known for providing readers a really important understanding of the topic.

By doing many kinds of ranking and rating at the same time, we can try to determine which articles are the most relevant, the most important and the most contextually valuable for any given topic. That will help us provide results for The ranking agents are the most complex and ever-changing part of the system. But the job they do is vital. And once they’ve done that job, they can pass results on to our front end services.

Gerd Leonhard on how we’re shifting from content (objects)...

Tue, 14 Dec 2010 04:25:38 -0500

Gerd Leonhard on how we’re shifting from content (objects) to context (activity around objects)

How Contextify works, part one

Mon, 13 Dec 2010 04:19:00 -0500

Contextify comes in two crucial parts: the engine (which powers everything;) and the applications (which present results to users). We’re not building either of those parts yet, since our Knight News Challenge application is still being considered — but it helps to sketch out what’s going to happen. Over the course of this week, I’ll outline how the different parts will operate, to give a clearer idea of what we’re trying to do.

Let’s start with the engine — the machine that goes out, finds stories on the web and then analyses them to discover what they’re about, whether they’re good and how much context they can provide readers.

The first part is to get the raw data to begin with. This starts with a robot that scrapes the web to find articles we can analyze. In the first instance, this is going to involve identifying a wide range of sources: news outlets, magazines, leading blogs and so on. This will provide us with a stream of articles to examine… and although you may think that the numbers will be daunting, it isn’t actually as big as you think. A major publication such as the Guardian, for example, publishes only about 200 or so articles each day. Of course, we’ll be looking at hundreds (and later thousands) of sources — but the numbers should be manageable.

To outsiders, “scraping the web” often sounds horrific. What this really means is simply building up an index of web pages that our engine can then begin to work on. These pages can be on the free and open web or, if outlets allow us, behind paywalls or registration barriers. While we understand that many readers don’t want to pay for stories, Contextify is going to remain agnostic: after all, we’re here to provide you the best understanding of why the news is happening: if that’s behind a wall, then that’s where it is. We’ll always make sure there is a free option.

And this doesn’t mean we make an archive of every story we’re going to look at. We look at articles our scraper has found, examine them to find what we need and build a database of information about them — always linking back to the original story on the web. After all, our intention is to direct traffic to websites — not take it away.

If you want a comparison for our scraping robot, think about Google’s spiders, those little programs that go off and explore the web to find pages that it can index. We’ll do something like that, except in a more narrow and focused way. To begin with, our robot will have a discrete list of sources to look at, but over time we’ll allow it to follow links between stories and bring in new material — if lots of people start linking to the same economics blogger, for example, our robot might decide it’s worth taking a look at what they write and adding it to our articles. Once we’ve got a good idea of what we’re looking for, we can also start going back in time, too, looking to find context that might be from a year ago, a decade ago or even further in the past.

Anyway, once the scraperbot is up and running, collecting hundreds or thousands of articles each day, it then passes off the information on them to the next part of the Contextify team: our series of context agents. More on them tomorrow.

"Taste is context, and the context has changed"

Mon, 13 Dec 2010 02:27:00 -0500

“Taste is context, and the context has changed”

- Susan Sontag

Part of a talk given to ABC by Jay Rosen, a professor at NYU,...

Mon, 13 Dec 2010 01:24:00 -0500

Part of a talk given to ABC by Jay Rosen, a professor at NYU, about “the future of context”. Click through and you can see the whole session, which is about an hour and a half long.

Boilerpipe: algorithms to detect and remove the surplus "clutter" around the main textual content of a web page

Fri, 10 Dec 2010 14:45:20 -0500

Boilerpipe: algorithms to detect and remove the surplus "clutter" around the main textual content of a web page

"We take a user’s location as a piece of context for finding what they want without them..."

Thu, 09 Dec 2010 06:44:41 -0500

““We take a user’s location as a piece of context for finding what they want without them actually searching for anything.””

- Marissa Meyer of Google talking about ‘contextual discovery’

Knight News Challenge Timeline

Thu, 09 Dec 2010 06:43:09 -0500

Knight News Challenge Timeline

How Reddit's ranking algorithms work

Wed, 08 Dec 2010 11:40:44 -0500

One of the most fascinating social sites online is Reddit, launched by my friend Alexis Ohanian and his pal Steve Huffman back in 2005. It was an early entrant into the social news space and, to my mind, has been the most successful since Metafilter. The community there is quite incredible — the only sad thing is that Conde Nast Digital, which bought it several years ago, has failed to build anything substantial from it (underfunding it to the point where the team was forced to ask members to pay in order to keep up with growth).

Anyway, Reddit is exactly the sort of community we might mine for the social layer in Contextify: an active and committed group of people who often point to stories online.

It also helps that Reddit open sourced almost everything in 2008, so the code is available to the public for digging around in. In fact, somebody’s done just that: Amir Salihefendic has done some looking to find out how Reddit’s ranking algorithms work.

What did he find?

Submission time is a very important parameter, generally newer stories will rank higher than older

The first 10 upvotes count as high as the next 100. E.g. a story that has 10 upvotes and a story that has 50 upvotes will have a similar ranking

Controversial stories that get similar amounts of upvotes and downvotes will get a low ranking compared to stories that mainly get upvotes

This is really important, because it can help us understand the way attention is used on a site like Reddit — and therefore help us tune our own algorithms to take this stuff into account.

It’s not directly applicable to our project, perhaps, but...

Wed, 08 Dec 2010 08:46:00 -0500

It’s not directly applicable to our project, perhaps, but James Surowiecki’s talk about context — the way we perceive who might be an “ideal” sportsman, for example — is very useful.

Finding out what stories are about: Muddy Boots

Tue, 07 Dec 2010 06:23:58 -0500

Talking with Jeremy last night, he reminded me about this BBC prototype from a couple of years ago called “Muddy Boots”.

The idea was fairly straightforward: parse articles to help machines interpret what they are about. As they put it, it’s asking “can a computer reliably identify the people and organisations in news stories?”.

Whilst a human reader takes for granted their abilities to pick up journalists’ cues and understand context, a computer has to be programmed explicitly. It is difficult to design a system that can identify people from text and disambiguate them. It is even harder to build a system that meets editorial standards of accuracy. However, in theory, it should be possible.

This is, essentially, one of the first steps that will be required to create a proper contextual recommendation engine. Of course, it helps if outlets properly mark up their journalism in order to tell Contextify what it’s about, but in most cases we’ll have to go and try and work out what a given item is actually about.

Muddy Boots has now been decommissioned by the Beeb and is now in progress as a standalone project from Rattle called Muddy.it.

Contextual TV with Google and Twitter

Tue, 07 Dec 2010 06:15:00 -0500

Contextual TV with Google and Twitter:

Michael Sippey takes a look at Google TV’s Twitter interactions: “Google TV shouldn’t just be another Twitter client, it should be a contextual Twitter client. See that picture-in-picture frame right there? The device knows what we’re watching right now, and so why not make the default view a real time search for tweets related to the show we’re watching? (It would be our own personal #vma display.) And if there’s not enough context to display real-time context, then take advantage of our history with the device.”

Bitly News — a tracker for the most popular links on Bitly, done...

Fri, 03 Dec 2010 05:22:40 -0500

Bitly News — a tracker for the most popular links on Bitly, done in Hacker News style.

"Some applications were outlandish, outside the scope of the News Challenge, restatements of..."

Fri, 03 Dec 2010 05:21:38 -0500

“Some applications were outlandish, outside the scope of the News Challenge, restatements of already-funded grant winners or dull rehashes of ideas whose time had passed.”

- Chrys Wu, the Knight News Challenge from one judge’s perspective

Remember Google's 'Living Stories' experiment? Here's the code.

Thu, 02 Dec 2010 04:20:22 -0500

Remember Google's 'Living Stories' experiment? Here's the code.

"Serendipity is not, in fact, at all random. In reality, it’s quite scientific."

Thu, 02 Dec 2010 04:12:30 -0500

“Serendipity is not, in fact, at all random. In reality, it’s quite scientific.”

- Henry Nothhaft Jr, Techcrunch

Explaining

Thu, 02 Dec 2010 04:10:00 -0500

Here’s an interesting project launched yesterday:

Working closely with the editors at ProPublica and led by NYU professor Jay Rosen, Studio 20 will research best practices, produce model explainers based on ProPublica investigations and identify useful explanation tools and techniques, not only for the benefit of its partners, but for the journalism community at large.
“An explainer is a work of journalism, but it doesn’t provide the latest news or update you on a story,” Rosen says. “It addresses a gap in your understanding: the lack of essential background knowledge. We wanted to work with the journalists at ProPublica on this problem because they investigate complicated stories and teach what they’ve learned to other journalists. It seemed like a perfect match.”

Explainers will form an important part of what Contextify does. Eventually, we hope to offer up the best stories, the best explainers, links to relevant photos, videos and audio content as well as clickthroughs to the relevant pages on Wikipedia and elsewhere. Right now, Explainer.net sounds more like an academic exercise than something that will build explainers, but it’s a step forward.

Another concept sketch from our Powerpoint deck: this time for a...

Wed, 01 Dec 2010 11:27:27 -0500

Another concept sketch from our Powerpoint deck: this time for a pop-up in-browser tool, in this case giving context and ratings to a recent story about the tensions between North and South Korea. One friend suggested that it was like a “News HUD”.

"The problem, of course, is that while Twitter makes it nearly effortless for folks to publish their..."

Wed, 01 Dec 2010 11:25:05 -0500

““The problem, of course, is that while Twitter makes it nearly effortless for folks to publish their own thoughts, it has done far too little to help those same folks glean value from the thoughts of others.””

- John Battelle