Finding news on social media can make journalism better

How finding news stories on the web can be a panacea for an industry that’s suffering from stretched budgets and accusations of lowering standards.

Digital journalists use tools like Tweetdeck, Newswhip, and Google alerts to rapidly discover things that are happening in the real world. Then they follow up their initial discovery, gathering more information and exploring the story to rapidly report on breaking news. It’s a powerful combination that cuts down the time that needs to be spent hunting for a story, without jeopardising the quality of the resultant journalism.

The problem they encounter is that they have to trade off coverage for speed. The searches are simplistic, and where more complexity and intelligence is provided in their tools, it tends to eradicate a lot of the speed benefits that digital journalism provides.

Let’s take an example — let’s look for shootings in Brooklyn. If that’s one of the beats I cover, I set up a Twitter boolean search that looks something like ‘Brooklyn AND “shots fired” OR shooting’ and add it to Tweetdeck. Then I wait. Ten minutes later, somebody fires a gun (it is Brooklyn after all) and a guy down the road tweets “just heard a gunshot in bed-stuy”. Well, shit, I missed that; it’s ok, I’m not worried — I don’t even know it happened. Then somebody else tweets “damn, who got shot? #brooklynsoundtrack”, and a video gets posted to YouTube — still nothing on my Tweetdeck search.

_Suspects in Bed-Stuy Gas Station Robbery_monitor.krzana.com

Now maybe some people who are following the people who tweeted see the message and decide to retweet it, and suddenly one of my other tools (let’s say Dataminr, or Newswhip, or Banjo) picks it up and somebody in one of those organisations gets to decide whether it’s going to be interesting to me. Maybe they decide it is, then they send it to me, and now I’ve got a news story I can follow up on. Alternatively, perhaps nobody sees it, in that case I keep waiting. Eventually somebody tweets “@BklynEagle #shooting on Halsey street, Brooklyn” and my Tweetdeck starts to move.

_Bed-Stuy gas station owner exchanges gunfire with robbers, cops say_monitor.krzana.com

Or maybe they don’t, and the story just drifts away, and nobody ever knows it happened.

What can we do better? Well, we need smarter search. “Brooklyn” isn’t a keyword, it’s a borough. Boroughs have loads of other stuff in them, streets and shops and local hangouts, important local people (@BklynEagle for example) and culture. When I type “Brooklyn” into my search box, I want it to know that mentions of Bed-stuy are mentions of Brooklyn. Similarly “shots fired” is a concept — it causes noise, it scares people, maybe it even hurts or kills people. If I’m looking for shootings, I’m looking for all of these things.

Google knows this, to some extent — when I type Brooklyn into a Google search I get a beautiful little box with a map and a history of the borough (although “shooting” is still just a bunch of letters to the search behemoth). The trouble is, that kind of comprehension takes work, and it’s inherently unreliable (Brooklyn’s also a pretty popular girls’ name, and a type of lager) — so services that try to do deep comprehension tend to be slow. They wait for corroboration, wait until somebody major reports it and gives them more context, before they send me a notification. Assuming one of them got lucky and picked up the story, Dataminr or Tweetdeck will have alerted me to the story at least within a few hours of it breaking; Google alerts might get to me before I leave the office, if I’m lucky.

What we need are tools that do deep contextual comprehension of content in real time and give me the information instantly, even if they might be wrong occasionally. That way, when I type “shootings in Brooklyn”, I see everything relevant coming out of the neighbourhood (it might be a car backfiring, or it might be an armed robbery, I’ll be the judge of that) and I can make sure that the stories on my beat never fade into the aether without being picked up.

We build those tools. Stay tuned and I’ll talk about how we do it.