Real-Time Search: How to Capture Consciousness

One of the hottest areas of search right now is real time search, which attempts to find results based on what is happening right now. Twitter’s search engine is fast becoming one of the key ways to navigate the service and discover what people are thinking about any subject at any given moment. Facebook is testing out ways to let you search your personal stream. Google (GOOG) is waking up to the challenge as well (Larry Page is particularly concerned with keeping up).

Every week, it seems, a new startup launches tackling real time search from a different angle. (Collecta, One Riot, Scoopler, Topsy, Almost.at, Tweetmeme, CrowdEye, Omgili, to name a few). They are trying to apply real time search to all the different streams of information flowing over the Internet right now: Twitter, Facebook feeds, Digg submissions, blog comments, RSS feeds, Flickr photos, YouTube uploads, shared links on bit.ly and elsewhere. The list keeps getting longer every day.

There is something about human nature which makes us want to prioritize information by how recent it is, and that is the fundamental appeal of real time search. The difference between real time search and regular search didn’t really crystallize for me until I had a conversation with Edo Segal, who sold his real time search company Relegence to AOL a few years ago and holds three patents on the subject. “Real time taps into consciousness,” says Segal, “search taps into memory. That is why it is so potent. You experience the world in real time.”

This raises an interesting dilemma. If real time data streams are akin to the living consciousness of the Web, how do you search them? How do you search consciousness? It is not the same as searching memory, which is what Google does when it looks at its indexed archive of the Web and how those pieces of information build up authority over time. The real time search dilemma centers precisely around how to rank results, and how to resolve the tension between recency and relevancy.

The default, or at least the starting point, for most real time search engines is simply to put the most recent results up top and then keep pushing them down in a free-flowing river of information as new results which match the query come in. That is what Twitter search does, for instance. It is a chronological stream of the most recent Tweets containing a particular set of keywords. Real time search startup Collecta also takes this approach of simply presenting the stream as it comes in, and letting you filter by source. Ranking results any other way would automatically reorder them and automatically make them less real-time.

Yet not being able to filter that stream generates too much noise. Other approaches attempt to add in other factors. OneRiot, for instance, is developing what it calls PulseRank, which takes into account the freshness of the information, the link authority of the Webpage where it is coming from, the authority of the person who is sharing the link, and the velocity with which the information is being passed around the Web. This seems like a reasonable approach, but it may not catch something important as fast as simply watching the unadulterated stream.

There are other approaches as well. You can look at what people on the Web are actually doing in real time or look for variations in the stream of mentions for any given keyword to notice spikes of activity. When everyone is talking about Michael Jackson or Iran above and beyond the normal level of chatter for those topics, that is when you want to know that you need to pay attention. So maybe real time search is more like an alert system.

Can you search consciousness, or can you only watch it pass by? We’ll be debating this at one of the panels on real time search at our Real Time Stream CrunchUp in July. But it is clear that in order to make sense of the stream, it needs to be ranked by order of importance as well as by time.

(Photo credit: Flickr/Andrew Sea)

Original post

Comments

[PastTense:]

Personally, this doesn't appeal to me at all; the problem with the search results I get is that the quality is too low, that they are irrelevant to what I am looking for--NOT that they are not "real-time" enough.

Jun 26 10:36 PM

[utilitus:]

From en.wikipedia.org/wiki/...:
"Today's SDI ( selective dissemination of information) systems owe a great deal to a 1958 paper by Luhn, "A Business Intelligence System"], which described an "automatic method to provide current awareness services to scientists and engineers" who needed help to cope with the rapid post-war growth of scientific and technical literature. Luhn apparently coined the term business intelligence in that paper."

Jun 26 11:02 PM

[Clive Corcoran:]

What would be really useful would be a search engine which can rank on the basis of the quality of judgments reached about the endless flow of "information"

Jun 27 05:51 AM

[The Lost Ag...:]

Great focus, based on what I have read OneRiot seems to have the support of the social community and its PulseRank product has the ability to limit how spammers can influence and ride the breaking news stories.

Im not sure about Twitters new verified profiles and how they will have an affect on their influence and how they show up in real-time search.

Based on a the recent coverage around Michael Jackson death, the tweets from celebrities who knew him even such as everyone's favourite tweeter Ashton Kutcher would likely be a better source than your local tv station who is just re-hashing the news feeds off CNN or AP...

Jun 27 09:21 AM

[frosty:]

Everyone should read "Feed", by M. T. Anderson. It might be the best $7.99 you will ever spend. Written in 2004, just after the internet had gone through its first explosive growth phase, it's real time social network via a computerized, wireless link to a neural implant (the Feed) is only a few evolutionary steps away. Remarkably prescient and profound in a humorous and entertaining way.

OMG! Kindle is implantable! I gotta tweet Amazon right away! Wait a few months to buy the book, then buy the Kindle Feed and it can be downloaded directly into your mind!

Jun 27 11:03 AM

[Henry Buttal:]

Good article! I like the quote you included from Segal, as it implies an understanding of search as a tool of the psyche.

Though I find the implications disturbing. Multiple studies over the last few years have linked attention problems to early TV viewing. I find it problematic enought that we have 24 hours beta wave stimulus external to our skulls. Now you can get Twitter, realtime ranked search, and, according to many experts (and frosty) like Ray Kurzweill, sooner than later, effective man-machine interfaces.

My concern has always been are we transitioning from an age of engineering to an age of tool users. Now I know its worse; we are becoming a generation of Fool-users...

Jun 27 11:18 AM

[Mayer Amsch...:]

Guys, bitching about technology and the speed of information is pointless. Every new technology has dumb and useless crap that comes with the useful stuff. The idea here is that real time search will *enhance relevance*. It is the same way that Google became as popular as it is - there already existed search engines before Google came along, but they found a way to research which of the search results people were talking about the most (which ones were linked to the most) and would put those results first, making it more likely that you'd get what you want. This is simply a new way of doing essentially the same thing.

Jun 27 03:28 PM

[Sovestor:]

focus on now is in...focus on real-time updates is in....

Jun 27 08:24 PM

[cameroni:]

This is a very interesting area of study. Because the internet, computer and misc. storage devices of all kinds essentially act as an extension of one's own brain. The opportunity then exists to peek into peoples minds, even deeper into their hearts.

It is a form of technological voyeurism really, taking snapshots of the thinking of individuals and doing it in real time. There is also an element of privacy violation despite the fact that the information itself is all publicly available. I am curious about where this will all go.

There is a warning in the advent of such technology for all computer users too. If you want to keep your thoughts and your life private then keep it off the web. Because one day, those many thousands of bits of unrelated information about you that have been preserved in cyberspace may suddenly come back to haunt you as an accurate collected description of your real personality.

Jun 28 02:33 AM

[Dave Dorgan:]

Thomson Reuters has done this for commodities by quantifying news related events. The results can then be integrated into trading systems.


On Jun 27 05:51 AM Clive Corcoran wrote:

> What would be really useful would be a search engine which can rank
> on the basis of the quality of judgments reached about the endless
> flow of "information"

Jun 28 09:40 PM

[Dave Dorgan:]

Very well put Mr. Amsch. IBM's Deep Blue beat Garry Kasparov with bruit force processing power. The human brain filters out a great majority of the "noise" virtually automatically i.e all of the irrelevant moves. The combination of computing power and intelligence are the key.


On Jun 27 03:28 PM Mayer Amschel Rothschild wrote:

> Guys, bitching about technology and the speed of information is pointless.
> Every new technology has dumb and useless crap that comes with the
> useful stuff. The idea here is that real time search will *enhance
> relevance*. It is the same way that Google became as popular as it
> is - there already existed search engines before Google came along,
> but they found a way to research which of the search results people
> were talking about the most (which ones were linked to the most)
> and would put those results first, making it more likely that you'd
> get what you want. This is simply a new way of doing essentially
> the same thing.

Jun 28 09:47 PM