Blog Search May Suck, But What Do We Really Want?

By Rob Diana of Regular Geek (Twitter/FriendFeed)

In the past few weeks, the 'blog search engines are bad' complaint has resurfaced. This seems to reappear every few months, and for good reason. Blog search is not really changing. I complained a few months ago that blog search was broken and a month later reiterated my feelings when Technorati went down and nobody cared.

This time, Steve Rubel complains that blog search is in a pitiful state. His lament seems to be that the blog search engines are not returning real links to his blog posts. I could be wrong in my summary, but this is the real point of trackbacks, which most blog platforms still support, I thought. I am not saying that I disagree with Steve, I do believe that blog search is in terrible shape. Mark Evans questions whether blog search is just too hard a problem:

Maybe the blogosphere is simply too difficult to track given it changes so quickly and there’s so much to spider. Or maybe Google believes there are bigger opportunities elsewhere.

Mark even asked for a better blog search (and a few other things) from Google for Christmas. Part of the problem that people are mentioning is that Technorati and Google Blog Search are returning links from blog rolls and not just links from within posts. Google Blog Search is actually asking for help if you see this problem.

However, I think the conversation regarding "finding links" is missing something. What do we really want in blog searching? Are we only searching for links to our posts? Are we searching for blogs that are talking about a specific topic? Are we just searching for new blogs to read? Are we really just trying to find out where our blog ranks, like Technorati's authority?

Part of the problem is that we are focusing on one issue with blog search. Basic link searching will probably always suck because of the problem with spam blogs. They will add links to some sites and detract from others. Spam is just a hard problem to fight, so we probably will have to live with some of those problems. The other questions are much more interesting, but I am not sure that people really want those types of features. Do we want to use something like Technorati to find new blogs? Or are we now using FriendFeed and Google Reader shared items to find new things?

My thinking is that beyond Technorati Authority, people have left blog search behind. Because of this, blog search is not going to be seeing innovation by itself. If you look at Compete and compare traffic of Technorati, Google Blog Search, IceRocket and BlogPulse, you will see that the combined traffic has decreased over the past year. My estimates put the decline in traffic from Technorati and Google Blog Search at 30%. Obviously, there is little traffic or revenue to be had. As you can see, there is no real reason for anyone to try to compete with Technorati. It would be a competition for a dying space.

If you are trying to find who is talking about your post, blog links are only a small part anymore. Social media has changed where the conversation is occurring. The conversation is still happening on other blogs and in the blog comments, but there are other sites included as well. Social news sites like Digg, Mixx and Reddit have comments from various people. Other social media sites like Twitter and FriendFeed also promote comments on their sites. It is time to stop complaining about blog search. It is time to start looking at the bigger picture, find the conversations and join them.

Read more by Rob Diana at RegularGeek.com.

BlogPulse Offers Insight into Blog Trends, Conversations and Influence

While BlogPulse has been around since 2005, I have largely ignored it, relying on Technorati, Google Blog Search and my own internal metrics to gauge momentum, trends and how conversations get shaped. But in light of this weekend's discussion, I was drawn to the site, and found it offers the best, closest, picture to how the story developed, who linked to who, and how a story can gain influence.

You can even see which people, famous or otherwise, are getting cited most frequently, or are the most "bursty", showing they are climing the ranks. (Key People for April 13, 2008)

Part of Nielsen BuzzMetrics, BlogPulse highlights the top blog posts, news stories and videos on the Web each day, and offers the ability to search for trends, track conversations across multiple blogs and get profiles of a site. Interestingly, I was alerted by Technorati to the fact that Friday night's post was somehow labeled the second-highest "top blog post" by Nielsen, and Scoble's follow-on "Era of Blogger's Control Is Over" ranked fifth. This was tabulated by the blog posts gaining the most external links. You can see the top forty for today listed on their site, ranging from technology to politics. Unsurprisingly, the weekend discussion on Shyftr figures prominently, with Scoble and me being joined by Tony Hung.


What makes BlogPulse most interesting, at least to me, is the ability to break out conversations between blogs, like a family tree, seeing who linked to who, and how while I may have kicked off the discussion, its clear that Scoble and Hung have their own spheres of influence. Of course, as some reactions linked to all sites, it's not a perfect measure, but BlogPulse is the best I've seen here. (See: BlogPulse: Conversation Tracker)

But BlogPulse does more than just track the conversations. Like Technorati, BlogPulse can show charts, displaying if one topic or another is capturing the fancy of the blogosphere as a whole.

Here is the chart showing Shyftr's spike over the weekend:


The same chart for FriendFeed:


And for Twitter:


And if you're so inclined, you can even search for yourself, like I did.


Drilling down further, BlogPulse offers site profiles for the many blogs they index. The front page of the site claims nearly 78 million identified blogs, with more than 80 thousand net new in the last 24 hours, with almost 700,000 new posts indexed. Now that would make for a big fat, RSS to-do list, would it not?

Looking at my BlogPulse profile, common keywords in my recent posts include "TechMeme", "Blogosphere", "Subscriber", "Momentum", "Anticipated", "Linking", "Embedded", and "Screenshot", to name a few. BlogPulse also offers graphs showing the number of posts per day, and how often the site has been cited in the last month. The chart for my site is below:


Can BlogPulse replace Technorati, as many have expressed frustration with the one-time blog search king? Maybe not, but it certainly has a lot of very interesting elements that I like. While it's not new, I'm definitely going to be paying a lot more attention now to BlogPulse than I ever did before. After a crazy blog weekend, it's offered us the best picture of how it all unfolded.