The Five Stages of Filtering, Relevance and Curation

Tonight's news of Gmail taking on information overload directly, using a combination of intelligent algorithms and your own feedback to build in box personalization is yet another hallmark move to taking on the increasing deluge of content approaching us from all directions - be it our e-mail, static Web pages, audio and video, or the many different social streams which we have subscribed to. There is no question that content creation and sharing is exploding and people are completely incapable of giving every single message and item their full attention. And many smart folks are looking to bring solutions to find the best and ignore the rest.

As I see it, there are five major ways companies and individuals take on the topic of relevance.

1. Editorial Filtering

Loose definition: I am the smartest person. I know best for you, and I deserve to decide for you what is the most important.

Example: The New York Times, Wall Street Journal, CNN and most mainstream media outlets today, who for years have been trusted arbiters to find the most important news and bring it to us in the way that they decide.

New media examples: The Drudge Report, which has grown from one man's curation and sorting to a full team, and Techmeme, which was once almost completely algorithm driven, and now is staffed around the clock by savvy editors who pluck the best of the tech Web.

Of course, it is easy for an individual to be a curator. I share a lot of content, manually, through @lgstream on Twitter, as well as on Google Buzz, FriendFeed and Facebook.

2. Global Popularity Filtering

Loose definition: The will of the people can be trusted, and they will decide what is most important, thanks to the most votes.

Examples: American Idol, Digg and Reddit. He with the most votes wins and gains a coveted front page slot.

New media example: Twitter Trending Topics display the most frequent topics and hashtags, not necessarily the most important. Also, you can see tools like Tweetmeme and FavStar which watch for number-driven popularity online.

This would also have included RSS shared items counters of the past, such as RSSmeme and ReadBurner.

3. Social Filtering

Loose definition: What your friends like, you will like. If it's important to them, it's important to you.

New Media Examples: Facebook recommended friends and pages, which display how many friends like them, Google Social Search, which pulls results from your friends content, and FriendFeed Best of Day, which shows the items from your friends that gained the most activity over a time period.

4. Explicit Personalization

Loose definition: You told us what you like or don't like, and since you know yourself best, you know what's important.

New media examples: Netflix's star ratings and TiVo's Thumbs Up or Thumbs Down feedback mechanisms, as well as Kosmix's MeeHive project.

5. Implicit Personalization

Loose definition: Just be yourself. Read what you want, do what you want, and the system will learn from you, continuously updating.

Examples: Amazon.com, my6sense.

There is a time and place for practically all types of filtering. Mona Nomura of Pixel Bits today talked about the serendipity algorithm and what it means to marketers looking to leverage machine learning. With Gmail's announcement, ReadWriteWeb's recent coverage of TrapIt and ChatterApp in an article on consumer information overload, and the high visibility of Facebook's News Feed, against the recent feed, the challenge has grown to a level where you no longer have to convince people there is a problem in filtering, but instead, you need to make a concrete decision as to how to approach that problem.

I believe that there is a role for trusted curators of news, people who have unique access or unique insight, who can get to news more quickly than anybody else, or dive into it more deeply. I believe that social similarities are a good hint at an individual's interests, but they cannot replace your own preferences - which go beyond your ability to fill out a form and try to tell the truth on what it is that you really like. The best systems, as Gmail is trying to do (with some help from your own feedback on whether they are getting it right), happen naturally and transparently in the background.

It's natural I would think this given my work with my6sense, but I have long believed in there being a perfect place for humans to act as curators and guides, while there is another perfect place for machines to provide, to the best of their ability, resources to aid your discovery. So when you are challenged with a mountain of information coming at you from any angle, think of the best way to get it handled. Should you turn to an editor, to the will of the people, to your friends, or to code? The options are all there, and more tools are coming to help you attack the noise - because there's little chance it will fade away any time soon - and a very strong chance it could get much worse very quickly.

Disclosures: I am vice president of marketing at my6sense. ChatterApp and TrapIt are assumed competition. In addition, Kosmix.com is a client of Paladin Advisors Group, where I am a co-founder. I was also previously an advisor to ReadBurner, since closed.

Future US Shuts Down DailyRadar, Ballhype, 'Blips Sites

As Andy Beal from Marketing Pilgrim discovered earlier this week, a once-promising news and social networking family of sites, owned by FutureUS, including ones dedicated to sports, Hollywood, politics, and much more, was snuffed out with little warning. In its wake, sites I often visited in the last three years, including BallHype, MacBlips and many others, are gone, and aren't coming back. The move was said to be a "reluctant" one, derived from "continuing shifts in consumption and sharing patterns", which made the sites stodgy dinosaurs in a Twitter and Facebook dominated landscape.

Ballhype sparked the genesis of the network way back in April of 2007, gaining visibility not just from me, but from TechCrunch and MG Siegler, before most folks in the Valley had heard of either of us. The promise was to make a Digg-like site focused on specific topics, and tap into real communities. For a while, it worked. Six months later, the Ballhype team went for the glitz of Hollywood and launched Showhype.

The success of these sites, and promise for even more subject-centric vehicles, led to Future US picking them up in July of 2008 for a rumored $3 million, which brought even more sites, including Beltway Blips for politics, spawning an avalanche of "Blips" clones, like MacBlips and GadgetBlips. The sites themselves were very cool, pulling headlines from around the Web by topic and letting you vote them up or down.

But this alone seemingly didn't help the long term success of the network. The engagement once a hallmark of Ballhype didn't make it to all the other properties, and it was well known that Future US scaled back its plans in 2009. Now, it seems the two year experiment has been closed.

The husband and wife team behind Ballhype, Showhype and the rest, Jason and Erin Gurney, ended up being good friends with my own family, and even provided our twins with some of our most-enjoyed toys, after their kids had outgrown them. It seems the social Web outgrew the Daily Radar and Blips sites, and we'll be watching the pair to see what they come up with next.

Braden Adds to the Gray Family to Make a Full House

As of Tuesday night, after 11 pm, we are officially outnumbered. My wife and I now get the opportunity to contend with three kids, the oldest pair being just over two years old, and the newest, starting his second day as I write. The expansion of the family continues the rapid rate of change brought on with our moving, joining my6sense, and setting other dominoes in motion. One person, in response to the many updates, even included my switch from iPhone to Android as a big move. I guess for any geek this is the case, but the addition of a new member of the family is a huge deal, much more than any operating system, office suite, or handheld.

Like Matthew and Sarah before him, Braden Thomas Gray (his full name) joined us four weeks ahead of schedule, deciding a little over 35 weeks of baking was enough. And as Sarah did in 2008, when the twins arrived, he will be spending a little more special time in the hospital as he gets needed attention.

As many parents know, the act of bringing a child into the world isn't always as beautiful as they show it in the movies. It's not always a few pushes, a cry, a smiling doctor, and a quick cuddle with mom as time of birth, weight and height are announced. It certainly hasn't been this way with us - and last night's experience was far from perfect, even if Braden's smile and fantastic skin show different through the pictures and short video I managed to escape with today. I'll save you the details, but we have had to measure our excitement with concern over the last 24-plus hours, and will keep watching until we get an all clear to take our newest guy home.

I enjoy a lot of things. I think I have more fun than most people. I enjoy interacting with all of you here and in the various places we find each other. I think I listen to more fun music than most people, and I have to self-censor constantly just so I don't ramble on like a comedian who has had a few too many drinks. But nothing has been more fantastic than raising Matthew and Sarah and watching them develop, sometimes slowly, and other times, extremely fast. They are a joy. Braden is here now, and even though I have never held him, or even touched him, he's already part of our family, and has a place.

With his rapid arrival, we're not fully ready. We don't have a car ready for three car seats, one facing backward, and two facing forward (as dictated by law). We don't have a dresser in Braden's room, and haven't found more than two newborn baby boy outfits in our garage, even though we know they are out there from when we did this just over two years ago. But Braden says he's ready, and after a rough start Tuesday night and Wednesday morning, he was looking me in the eyes through the incubator's plastic today, as he took deliberate breaths and balled his fists.


Braden's first YouTube Video

Coming on the back of late Monday's news, you would think I wanted to get everything out of the way in one week, but it's not true. Parenting is not something you get to draw up and then execute against. It takes permanent flexibility, and I look forward to the chance to bring Braden home soon and see just how much flexibility it takes to tangle with a newborn and two two year olds.

Edge Theory: Talking my6sense Tech With Chris Saad

As word of my joining my6sense as vice president of marketing broke late last night, Chris Saad (@chrissaad) put on his journalism hat, and the two of us recorded a quick Edge Theory Conversation to explain the opportunity ahead, and what this means for all the other roles I have been carrying.

Chris' insight into the opportunity for my6sense's ability to prioritize and personalize content from all streams on the Web is especially important for me, given his background as a geek and his dedication to data portability, open standards, and his day job, working for Echo, where they aggregate reactions of all types. With Echo becoming the back-end conversation repository for major media sites, it's of course possible those discussion areas are ripe for personalization and prioritization...

This and all other EdgeTheory conversations can be found on the dedicated ET Conversations site.

Listen to the full recording below:

Why My6sense: Right Time, Right Place, Right Context

After a post talking about the "Me Me Me" portion of social media, it seems only right to follow up and talk about news with me. If you caught Marshall Kirkpatrick's story on ReadWriteWeb (See: My6Sense & The Geek Who Rode His Blog to the Edge of the World) or caught the CinchCast interview I did with Robert Scoble, you know that I have joined digital intuition pioneer my6sense as the company's vice president of Marketing, after working with the company for a year as an advisor as part of my work with Paladin Advisors Group. By joining my6sense, I am the company's first employee in the United States, as the bulk of the company is at their Israeli headquarters, and I get the opportunity, once again, to take a major participatory role in a tech startup which has the very real potential to change the way we consume information, of all kinds, in all places.

As you may know from my previous coverage of my6sense, the company has an iPhone application today that prioritizes content from your subscriptions and social streams, based on your own behavior, automatically. Even more importantly, the company has an "Attention API", which can plug into any content provider, stream client or social network, breaking us out of the box of editorially-divined content prioritization, or best guess social circle recommendations, or the most-common, simple chronological sorting.

Truth is, even for the most focused of us information consumers, we are missing interesting content. More people are sharing more information more easily in more places, and it is impossible to read every Tweet, every Facebook share, every RSS entry or every Google Buzz post. Our intuition tells us quickly what are the most important items, and we find ourselves skipping the rest, or getting annoyed at what we find less valuable, considered "noise". I fear it is only going to get worse.

Solving for this information overload dilemma has seen new companies debut interesting filters, or rely on popularity or community to push items that may interest you to the top, but this is not true human ranking function, or it expects you to explicitly list your interests by keyword or subject. From all the companies I've talked to, interviewed, tested, and written about, my6sense has the best underlying technology, which works implicitly, leveraging your own behavior and not only already works very well, giving me a great mobile feed reading experience, but also sets up the opportunity to make this the official prioritization and personalization engine for the Web.

Even if we wholly adopt the belief that there is value in real-time reactions and recency, there is much to be said for the right content to find its way to you at the right time, in the right context. Factors that make you decide whether to read one article or update over another can be dependent on many different things - the source, the author, the headline, the subject, or even the time of day or location, and my6sense's Phds are working on divining the best stuff just for you - not anybody else. And in the year-plus that I have used the iPhone app, I practically never use its alternate time-based view, as the content is unfiltered and usually off-topic.

What makes me excited enough about my6sense to put Paladin on the backburner, despite the firm's doing very well, and having just hired a new partner, is that this is the right time to start plugging in prioritization and personalization for the entire Web. If the entire story were "just" an iPhone application, it wouldn't be interesting. But consider the benefits of personally curated content sites from major media outlets, social networks and clients. Consider how valuable it would be for those sites if users found content more relevant and stayed just percentage points longer per site, read more stories, and clicked on more relevant ads. Consider the real value of not contributing to the world of information overload, but cutting through the noise like a warm knife through butter, find the best and ignoring the rest.

My6sense already has the best technology, from all I have seen and heard, to make this happen. Now comes the opportunity to help do my part, an active role, to transform the company from technology leader to market leader.

And if you are curious what this means for the future of louisgray.com, I will of course be changing my About page and will offer disclosure if there is ever a time that warrants it, just as I always have. I have tried to bring solid content and ideas and smart technology here since 2006, when I worked for a hardware company, and in the last year, as a new media Partner at Paladin. The perspective changes, but the goals remain the same. Please do continue to keep me honest, and always let me know if you think I have crossed the line.

If you haven't yet had a chance to catch the call I did with Robert Scoble from Monday night, it is embedded below.

Social Me Me Me Me Me Media

  

With more places to share content on more networks, each having their own diverse social graphs, connections and features, many of us are splintering our online identities, choosing one or a few communities over others, and optimizing our sharing and consumption to fit the model that best works for us - and that self-assessment should be constantly changing as our own preferences, and those sites' capabilities change. But what will never change is people's desire to feel important and interesting, whether they are creators, sharers, commenters or even lurkers. As I've seen time and time again, people often measure a site's "success" for them by how much attention they receive, and if this attention fades, so too does their opinion of that community.

Despite significant attempts at automation, people continue to be human, and thus, emotion is involved. With so much content being created and shared in more places, and the hours in the day not having increased for some time, the potential for data to be distributed and lie untouched is increasing, even as the total number of users of communities grows. Sharing is outpacing user growth in practically every case, so far as I can tell.

This means that many blog posts are getting no comments to fewer comments. Tweets are not getting replies. Shares on FriendFeed and Google Buzz that once got conversations in times of heightened awareness are lying fallow. Photos on SmugMug and Flickr are getting fewer views. This isn't true for everyone, but it happens, unless the person sharing is aggressive about engaging with that community, doesn't mind the community morphing, staying flexible as it does, or if the person is a brand worthy of following no matter what, even if strangers are forced to interact with their "real" friends less just to keep pace with the top dogs.

As someone who has worked hard in a few dedicated communities to engage with people, I have been lucky enough to forge real relationships with people I have never met, as we each share our lives and interests together, engaging where they intersect. But I have seen many people, friends real and imagined, get frustrated when they find their own actions are going unnoticed. After periods of decreased engagement and assumed invisibility, it's not unreasonable to think they get disillusioned, and the truth is, it's not a surprise if I didn't even notice. With so many streams flowing by and so many sources and people demanding my attention, even the strong soldiers that fatigue during battle are left behind in our own self-directed charge up the mountain.

The well-respected Leo Laporte had an epiphany of sorts this weekend, finding that a gap in his sharing to Google Buzz, one of my favorite communities, had gone unnoticed. This led him to a conclusion that "no one was even paying attention to it in the first place", and his years of effort had been in vain. This is the same conclusion often rapidly reached by much-less visible people, but carried more weight thanks to Leo's seeming pervasiveness, and his typical upbeat mood, so when Leo barks in frustration, you know something could really be wrong.

With the many different places demanding our time, I've long held to the belief that blogging is the foundation in a world of streams. If forced, I would keep the blog going and dump all my social media accounts, because the blog is a history and it is me, even if I write often about others. This is a searchable, indexable, discoverable entity with longform, permanent content, not an ephemeral short-time share that adds to the minute or the hour, but not much longer. I have embraced all the other networks, and will embrace many more, because they can be valuable tools, and that's where real communities are thriving, but the millions of shares pouring into the largest networks are usually air. Even as I've cut down the many people I follow on Twitter and other networks to a select group, if I try and read every one, I quickly fatigue from the inanity. It may not be all about "what you had for lunch", but it is often about where you've checked in on location based networks, discussions of preferred tools, brief updates of what people are thinking, or disjointed conversations between mutual followers. One of the hardest things to do for anyone is to find real value amidst the noise, and the massive volume means that people can get missed.

I follow Leo Laporte on Google Buzz. I also follow him on FriendFeed and Twitter. Did I notice that he had stopped posting to Buzz, due to a technical glitch? No. Not especially. Is it directly attributable to my being self-centered? I don't think so. Instead, it's due to the fact I follow many others as well. My Google Buzz and FriendFeed and Twitter and Facebook (and so on) are always full. There's always new content to get. And to be honest, even if I had noticed Leo had gone dark on one site, I likely would have attributed that to an unannounced August vacation, a change in how he prioritized networks, or a change in how he distributed content. I still follow him in other networks, and didn't pick up on the bug.

Adam Singer of the Future Buzz says it's "absurd" to give your social presence to the stream in entirety. I agree that for people who can provide longform content, this is absolutely true. There should be a place of permanence, a place for conversation, which you own outright, because no other platform is entirely safe. Companies go out of business, get sold and change their rules on data retention or discovery all the time.

I believe in authenticity. I believe that if you initiate a conversation in a far-flung place, be it a blog or a social network, you should be there to respond and engage. I believe Leo does this and I know I've seen him engage in my Buzz stream, so those saying he doesn't engage are wrong. Monitoring activity on your streams, no matter where they are, is a basic must-do for any social media participant. Respond to replies on Twitter. Follow up on conversations in FriendFeed, Google Buzz, Facebook or anywhere else they may be. It's common sense. But I also know not everybody spurs conversation, and a lack of activity on their items can be felt as a personal slight.

Meanwhile, the ease of counting reactions is contributing to the problem. If "top dogs" are seen to get dozens or hundreds of retweets, comments and shares, it's clear. And it's clear when you don't. If you do get a big hit that gets engagement, it's tempting to try and recreate the flurry of attention. Who doesn't like attention? But when the meter doesn't go past zero, or the numbers drop, there is a real feeling of lost self-worth.

Many people do post about themselves. It's what they do. The person who shouts the loudest can get a lot of attention over a short time, but shouting is hard to listen to for long periods, and it is hard to sustain. You cannot replace engagement, true conversation, a give and take of ideas, and an exchange with a real community. The answer is not to pick on a specific network or a group of people, or to abandon the entire practice of social media, but to adjust one's expectations accordingly.

Those who engage in these networks know who is all about themselves and who is all about growing the community and sharing ideas. You can't blame the tools, because at the end of the day, it's just code. This code can bring people together, or it can push them apart. And if the networks go away, we'll still be talking about ourselves, or engaging with friends. Offline or online, people want to feel desired and interesting and nobody likes being ignored.

iPad Facebook App Friendly Adds Support for Places

With Facebook's iPad app still missing in action, the Friendly Facebook browser has not only maintained a top position in Apple's iTunes store (currently ranking #4 for best-selling iPad apps and #8 for top grossing iPad apps), but it is keeping pace with continued updates from the world's largest social network. As has been much discussed throughout the tech Web, Facebook's launch of Places brought them squarely in the world of location based services (LBS), with functionality mirroring Foursquare. Through some fast coding, Friendly already has introduced support for Places in their app - letting you check in to Places, add venues and tag friends, just like the Web version, and Facebook's official apps.

The latest iteration of Friendly, upon login, features three main tabs, including the Live Feed, Events and Places. Places supplants the original third tab, which was Requests. (See: Friendly: The Best Facebook Experience for iPad)

The Newest Friendly Build Features Places

Adding a New Place via Friendly

As you would expect, the Places tab offers for you to "See where your friends are and share where you are." To check in to the place, just click the location pushpin, commonly associated with Google Maps, and select an existing location, or create a new one yourself.

Posting the Check-in to Facebook via Friendly

The Resulting Check-in and Tag on Facebook Places

Upon adding a place, you can add a description, and tag other Facebook friends who may be at the same location - just like the Web site.

The implementation of Places within Friendly is very straight forward, and looks like pulls data from the mobile-optimized version of Facebook, at touch.facebook.com, which no doubt speeded the coding. You can grab the Friendly app from iTunes for $4.99.

Social Aggregator Cliqset Undergoing Turkish Invasion

While the Silicon Valley seems to largely have turned a blind eye to any social network not named Facebook or Twitter, one country's citizens have flocked to alternative sites more than practically any other. Turkey, well-known for being the #1 country driving traffic to the Facebook property FriendFeed, appears to be having a similar impact to the well-designed, if not fully embraced by the masses, Cliqset. Partly due to the uncertainty around FriendFeed's domain, thanks to recent downtime, a year after the product was acquired, Turkish users of the network are seeking out alternatives, and have seemingly found Cliqset as the best alternative.

Cliqset, which supports integration with more than 80 networks, has closely integrated with other sites, including Twitter and Google Buzz, to highlight mentions and migrate comments between networks, is now seeing the results of this migration, be it temporary and permanent, as the site's public feed (highlighted as Community) sports a great deal of Turkish posts and updates from the FriendFeed refugees. The sheer volume from the new arrivals, either testing the site or setting up their profiles, drowns out all other languages, amusingly.

FriendFeed's Turkish Fans are Flowing to Cliqset

The New Voice of Cliqset - Via Turkey

The Turkish invasion aside, Cliqset has quietly developed a fantastic tool for discussions, letting you easily discover your friends from other networks, search across the site for topics and tags, join groups, and see direct messages from Twitter from their centralized interface. In fact, the ties between Cliqset and other networks are so strong that you can see items you have favorited in Twitter, and should you follow a person on Cliqset who has registered their Twitter account, you may end up following them on that network as well.

Cliqset Offers 80+ social integrations and a solid conversation platform

CEO Darren and I Trade Turkish Barbs

Earlier this month, TheNextWeb highlighted the tie-ins between Cliqset and Buzz, arguing the former was trying to save the latter, by offering an extremely clean interface to Google's network. And whether you agree with the author's premise, his comment that "Both Buzz and Cliqset have a similar struggle: strong technology that is not quite matched with equivalent user activity," rings true in light of the more-discussed and more utilized alternatives, especially Facebook and Twitter.

As somebody has always investigated edge case social networks, from FriendFeed to Strands, MergeLab, Socialmedian and others, the new generation, including Cliqset, Amplify and Google Buzz are innovators I am rooting for in the battle for attention and activity. While I don't speak a lick of Turkish outside of Google Translate, it's always good to see entire groups find community in a new place and have ties so strong that they can all migrate at once. As for FriendFeed, I am sure the site will do just fine if their Turkish fans are now congregating in multiple places.

You can find me on Cliqset at http://cliqset.com/louisgray. Your language choice is up to you.

Real-Time News Needs to Reward Authenticity, Curation

At the Search Engine Strategies San Francisco event Wednesday, Echo's Khris Loux sat down with me and representatives from UltraKnowledge, CBS Interactive (formerly CNET) and CNN.com to discuss the rise of citizen journalism and social media, intertwined with traditional journalism, and how mainstream media outlets can adapt their practices to include curation as part of their offerings. Our panel's determination could be summarized as saying the ability for the public to launch stories and extend storylines related to realtime has removed the ability for mainstream media to be gatekeepers, furthering the need to accurately determine authenticity and truth. And where mainstream media cannot have exceptional access, be it to geography, sources or timing, part of their new role is to discover and highlight high quality content, regardless of its source, in effect, adopting the role of curator.

Best exemplified by CNN.com's iReport, citizen journalists are being given a chance to report on equal footing with the mainstream media. While iReport gives these would-be reporters a place to piggyback on a major brand, many others have taken to their own blogs, social networks or Twitter to report live, and unlike the traditional approach where media authors would issue a story and call it a day, the life of a story now often starts when it hits the Web, spawning comments, shares and retweets. But as citizen journalists and bloggers are given the same capability to gain visibility, mainstream media's much-celebrated structure to put articles and stories through the vetting process sometimes falls astray in the race to get the story out quickly.

CNN's iReport Has "Vetted" about 5% of all Submissions Today

All time, about 7% of all Submissions are "Vetted".

A little over two years ago, I highlighted how quickly incorrect news can spread in the age of microblogging. (See: Smart People, Stupid Tweets. Fake News Spreads Fast on Twitter.) While many of us know not to believe everything on the Web, even if it is in Wikipedia, there remains a need to separate fact from fiction. With so many publications out there reporting on the same information at the same time, without having additional insight, the task of curating the very best of the analysis and repeaters can become a very serious task.

In January, I broached this issue, asking, "Can You Filter for Quality News Amidst Instant Analysis?" and earlier still, in December, highlighting how some news organizations are in fact incentivized for behavior which may lead away from taking the extra effort sometimes required to get the story right. (See: Growing Grumblings on Tech News Don't Address Incentives)

As excited and promotional as I am about the ability for anyone to create and share content, I am equally worried that incentives to promote the discovery of accuracy are not clearly defined, and it is often very difficult to find a real authority on a given topic in a world of instant analysis. When a story breaks in a real-time platform, like Twitter, the person who first reported the earthquake, or was present on the scene of an event, is only an expert for a short time, and when that topic has faded, new influencers for new stories are needed. This leads to a need for trusted sources to harness the truth from the originator of news, wherever it may be, mass media or citizen journalism, and rapidly verify accuracy.

But the sheer volume of feedback can overwhelm even the most well-intended. CNN.com reported one recent popular article achieved more than 30,000 comments - a practical impossibility to moderate by humans, leading to the desire to somehow separate the wheat from the chaff and surface the most accurate and supported insight. As one knows, trusting the community is possible, but usually results in an across the board maligning of those who share opposing views. See any political blog or sports blog for this particular variety of trench warfare. See any comment stream on Yahoo! News or on YouTube to see the quality of the masses at work.

The answer lies in finding a way to reward the truth-seekers in the real-time Web, and to reward media, be they an established brand or an emerging one, for discovering accurate insights, or simply harnessing them. The world of real-time storytelling is evolving before our eyes, and in some cases, its sheer ephemeral nature promises to introduce as many issues as it solves. We, as readers, creators and sharers, should be working diligently to keep our streams clean and provide value to all those who come in contact with our content.

For more thoughts, see Echo's essay: Real-time Storytelling

In Storage & Networking, Big Numbers In Dollars and Data

Today, at least for those of us who watch the enterprise space closely, the big news is that Dell Computer has offered to acquire Fremont-based 3Par for $1.15 billion, a premium of more than 80 percent over the company's stock price. In a world where much of the tech news is dominated by small companies taking money from angels, it's interesting to see the gulf between what it takes to grow a successful hardware company and the more ephemeral Web-based or application based companies that play significant roles on practically everyone's smartphone. And while I haven't talked about it too much on the blog, trying to keep a black and white separation between my day job for much of the last decade and my more hobby-oriented interests here, I've lived it, participating in one venture backed storage startup for more than 8 years, from 2001 to 2009, seeing companies raise, rise, fall and fail. In storage, the big winners, with few exceptions, can raise hundreds of millions of dollars before reaching break-even, and may be worth billions on the other side. Others may never find traction at all. 3Par, which took on tech titans like EMC and IBM, proved to have a winning formula.

There are three major truisms in technology. The first, and most well known, is that of Moore's Law, which while it has slowed in recent years, dictates that CPU processing speed increases at a regular clip while reducing in price. The second is that data storage capacities and densities are doubling at practically the same rate. Just look at the gigabytes or terabytes on your desktop or laptop hard drive and compare that with 5 or ten years ago. And the third is the speed of the network, both wired and wireless, increases - from the Kbps-rated modems of yesteryear to the fast-flowing networks of today, including 10 gigabit Ethernet on the client side and high speed Fibre Channel on the back end of many data centers.

These three advances mean simply this - more data can be created, shared, transmitted and stored more quickly than ever. Entire industries have been spawned around managing the data flow and storage, enabling branch office access to centralized data, deduplication and compression, load balancing and virtualizing the resulting complexity. If you watch consumer companies, such as Twitter, Facebook and Google, you probably see each of those companies creating new standards for global file systems and redundancy. You see them eschewing traditional storage companies and building their own devices in an effort to keep costs down as usage spirals upward. The trends are both amazing and incredible.

Back in January 2001, as Web 1.0 was crashing, I left a Web services company (eventually sold to Oracle) and joined a small company called Synaxia Networks, which later launched publicly to the world in March as BlueArc. At the time, comparable network attached storage devices from EMC and NetApp were capable of scaling to a then-massive 7 terabytes, and performance was not a metric either of them dominated. Our approach was simple - by converting aspects of the file system from software to hardware, we could dramatically accelerate storage. We scaled not to 7 terabytes, but to 225. We promised five nines (99.999% uptime) of reliability, and performance that was ten times the competition. And if we were less than that, everybody knows that two to five times the speed of the incumbent is still pretty darn good.

As we debuted, the press attention at the time was incredible - as our launch, backed by $30+ million in funding and our CEO being a former top guy at Compaq computer, gained massive attention. We had headlines in the Wall Street Journal and New York Times. George Gilder proclaimed that our product "imperiled" all software based storage devices, and after a successful debut at PC Forum, one reporter at TheStreet.com said it was like offering crack to CIOs. Pretty heady stuff, and not unlike other dramatic booms seen from companies that captured the tech press's attention, including the currently hot Twitter and Foursquare, to those less successful, like Handspring and Transmeta.

But building a storage company takes a lot of real money. BlueArc, which raised another $20 million just last month, has raised $200+ million over its lifespan. 3Par, purchased today by Dell, similarly raised $100 million in 2001 (as we were raising $72 million) and others raised similar amounts. Cereva Networks, whose assets were later purchased by EMC, had raised $137 million and laid off 140 employees back in 2002 after not getting off the ground. Zambeel closed in 2003, having raised $66 million, but selling only a single system. Panasas raised $25 million in 2008, one of multiple rounds for the firm. Maxiscale raised $12 million before coming out of stealth. Pillar Data, funded largely by Oracle's Larry Ellison, is expected to have raised between $300 and $400 million alone. So when I hear tech reporters hem and haw about Web startups raising $10 or $20 million, it doesn't make me blink, considering the world of big dollars I've operated in for a decade.

So why the big dollars? Why are venture capitalists so willing to put such big bets into spinning disk and faster networks? Because when things go well, the customer benefits are very real, and the returns could be even better. Customers will pay top dollar to reduce the amount of time it takes to build special effects or bring pharmaceuticals to market. Fast network storage devices are key in mapping out the earth's terrain from satellites, and combing its ocean floor for potential oil deposits. Fast network storage is being used to collect mountains of data by the government, to simulate nuclear weapons' testing, and build next generation vehicles. And those companies that won't compromise on the speed of execution will buy from new storage startups not named IBM, EMC and HP.

That's why Isilon, a competitor to BlueArc during my time there, is worth more than $1.1 billion today, even after its own public struggles. 3Par earned its way to the discussion and is now cresting above $1 billion. Ocarina Networks, a client of Paladin, was purchased by Dell last month, for an undisclosed sum. Ocarina's competitor, Data Domain, was caught in a bidding war between EMC and NetApp, eventually going to EMC for more than $2 billion last year - simply with the promise of reducing storage capacity!

Today, some of the biggest debates in the Silicon Valley are around angels versus venture capitalists, and whether a $500k round can tip you from one side to another. Some of the best known Web startups today are begging for a $25 million acquisition by Google, or so it seems. FriendFeed, one of the biggest acquisitions by Facebook, was rumored to be "only" $50 million. But on the other side of the datacenter, it is an entirely new ballgame, where hundreds of millions of dollars go in one side, and you could get billions out the other end, or you could get nothing. Companies like 3Par, BlueArc, Isilon, DataDirect Networks, Panasas and others have put pressure on EMC, NetApp and IBM to innovate, and expand their product portfolios. Companies like Data Domain, Ocarina Networks and Permabit are working to optimize storage throughout the datacenter. Emulex, Qlogic, Brocade and Cisco are working on faster networks, cards, adapters and protocols to make sure data can go between client and server and back again at rates previously impossible, and everybody is betting on standards they hope will put them in the best spot.

So congratulations to 3Par for their fantastic exit and sale to Dell. Congratulations to Isilon for fighting a tough battle and living the life of a public company, worth $1 billion and up. It's fun to see companies and people I once saw as competitors, partners and allies, who I rubbed shoulders with at trade shows, and with whom I traded taunts on Twitter, taking things to the next level. There is no doubt in my mind that others will be good stories, and some will go the other way with spectacular flameouts, equally incredible to watch, but for much different reasons. It's a very different ballgame over here.

Disclosures: As a former BlueArc employee and investor, I own private equity stock in the company. In addition, Emulex is a current client of Paladin Advisors Group. Prior to their sale to Dell, Ocarina Networks was also a client of Paladin Advisors Group. Maxiscale was also a Paladin Advisors Group client in 2010. At times, I may seek to do business with or engage with many companies in this list, or their competitors.

Redfin Highlights Schools Alongside Real Estate Search

Two months ago, I told you about how my wife and I used Redfin for practically the entire process of searching for and buying a home. If there were properties not listed on the site, we simply didn't see them, as we leveraged the company to track our favorites and discover new options. Even after finally picking one and moving in at the end of July, I haven't stopped window shopping - partly to ensure we made the right choice and to watch macro changes in the area - and I was pleased to see the service has added a new wrinkle to the site, displaying area schools with every single query - making area education options as important as any other facet when choosing a home.

Now, when you search for properties, not only due you see which properties are on the market, and which have upcoming open houses, but you also see school locations, as well as how they rank relative to their peers on a 1-10 scale, based on standardized test scores. Obviously, for a family with children, the higher an area school's ranking, the more appealing the property, and the lower the school's quality, the more likely it is that one's neighbors and neighborhood will have some issues.

Redfin Adds Schools to Real Estate Searches

The integration of education data in Redfin's real estate service is more than skin deep, as each school is now profiled on the site, with basic facts and even parent's reviews. According to Redfin's blog, the service will even soon let you search for homes within a specific district, a big deal for some families.

An Area High School's Profile On Redfin.com

In the areas of the country served by Redin, the product has eclipsed Zillow and others in many ways, and it looks like it could deliver the same level of detail for schools as it has for new properties. You can even start with a school's description and find homes for sale in the neighborhood, effectively turning the real estate search process on its head. If you're looking for a home and want to find data on schools in the area, Redfin looks to be leading the pack.

48ers.com Offers Realtime Social Search Across Networks

For the most part, realtime search engines, including OneRiot, Topsy and Twazzup, have focused on indexing the world of Twitter. That makes some sense, given the significant number of updates flowing through the microblogging network, but it covers only part of the worlds real-time sharing spectrum. A relatively new site, called 48ers, is looking to return results not just from Twitter, but also Facebook, Google Buzz, Digg and Delicious, and more interestingly, it lets you show results from each network on its own, should you choose. While Twitter may still win in terms of the total number of discovered results, they are intertwined with people who have chosen to post in other places.

Much like practically every search engine that has debuted after Google's spartan interface displayed the UI of the future, 48ers.com's home page is relatively light, showing only a search field, and in the age of Twitter, Trending topics. Click on a Trending Topic, such as "Hannah Montana Forever" and you see results that match the query.

A 48ers.com Search for VC Funds Shows Results from Buzz, Twitter

No big deal, right? True. But if you look at the top left of the search result page, you see you can now filter the search by any of five networks. Click on Facebook for the same query, and you will see shares on Facebook with the same match. Similarly, clicking on Google Buzz, Digg or Delicious will further segment the results by their source.

Searching 48ers for Oakland A's Updates from Buzz

Clicking on the results of any query takes you to the status update, if on Twitter or Facebook, the Digg submission, the Delicious bookmark, or the Google Buzz entry. (For example, this special Google Buzz entry on Hannah Montana)

Breaking News updates via 48ers from Facebook and Twitter

Like Twitter's vanilla search engine, 48ers offers useful searches, including "breaking news" and the more social ""totally recommend", which gets used surprisingly often.

Where 48ers comes up short is in continuing the realtime flow. Twitter's search engine, despite its failings, prompts to refresh as new entries are posted. FriendFeed's has them flow from the top. 48ers does not do that, so while the initial query may be made on realtime sources, the results themselves are static.

You can find 48ers on the Web at http://www.48ers.com or on Twitter at @48ersdotcom.

TweetDeck Looms to Battle Seesmic, Twitter on Android

    
In the two years since TweetDeck first launched on the desktop, it has set the standard for multi-functional multi-account Twitter-centric applications, bringing the concept of columns, grouping and integrated search to what had previously been a bland market of simple apps. With the product's success, we saw the metamorphosis of Seesmic from a video comments company to one that has gone head to head with TweetDeck on practically every screen, even as the two saw focused effort from Loren Brichter on Tweetie pass them by and become the official Twitter application from the microblogging mothership for both the Mac desktop and the iPhone.


Now, the two are competing for visibility on yet another screen - that of the fast-growing Android platform, and unlike previous battles, which usually saw TweetDeck innovating first, it was Seesmic who got there first - in a big way, going live on Android and BlackBerry back in November of 2009. Tomorrow, TweetDeck will officially launch the first iteration of its Android product on the Android Marketplace, and those who prefer it will once again have that choice.


   
The Accounts Pages for TweetDeck and Seesmic for Android

As Twitter quite visibly has filled holes in its lineup with the addition of Tweetie, both TweetDeck and Seesmic have positioned themselves as the more robust alternatives, providing support for not just Twitter streams, but also Facebook, Google Buzz, LinkedIn and Foursquare, depending on the platform. Seesmic's Android app, so far, in the absence of an offering from TweetDeck, has been my choice, thanks to its integrated multi-account and multi-service support - something Twitter's native app simply won't do.


   
TweetDeck Updates Put Multiple Sources In One Stream

Now, for Android, both applications will allow you to view streams from multiple sources, and update multiple accounts simultaneously. But where TweetDeck and Seesmic differ is that Seesmic's activity is very hierarchical. You view one of your accounts at a time, and go back to the top level to see other accounts, while TweetDeck is looking to "mix" the streams with updates from all different sources in a single column.


   
Updating from TweetDeck Android and Browsing a Profile

Updates display in TweetDeck in different colors depending on their source. Gray updates are from Twitter, Red are from Google Buzz, Blue are from Facebook, and Aqua updates are from Foursquare. Now, you can see all these in one place. Also true to TweetDeck's background is the option for you to swipe right or left and see more specific "columns", including a new one called "Me", which captures mentions for you across multiple networks, including Facebook comments and likes, as well as Buzz updates on your content.

A New "Me" Column in TweetDeck for Android

For somebody who needs that mixing of updates across networks on a single mobile screen, TweetDeck will be the best option. But the early version, which I have been using all day, definitely feels brand new, while Seesmic's offering is more stable and feels easier to use - no doubt the product of having been publicly available on Android for more than 8 months. TweetDeck's Android launch has been much-awaited, and its integration of a column for "Me" is interesting. The product also offers deeper integration with Foursquare, tapping into venue details for check-ins.

For TweetDeck addicts, the introduction does not come too soon, even if Seesmic, today, is a stronger offering. This should be a fun space to keep an eye on over the next several months.

Google Reader Shows If You Are a Skimmer or a Clicker

With most RSS feeds providing the entire contents of a blog post, for many people, the RSS reader is not just the place where news is discovered, but also where it is consumed. This means that if one's RSS subscribers triple, downstream traffic may not increase at a similar rate, as even if readers enjoy one's content, it is likely that many don't click through to the original story. On Tuesday, Google Reader introduced a metric that shows how often you are clicking on feeds within the service, and for somebody like me, who uses the service to read and share downstream, it's no surprise I only log a few clicks a day - with many of the sites I click to being different than the most active sources.

As a power consumer and human filter for what I portray as quality content, I recognize the numbers I am dealing with in Reader are large. In the last 30 days, I read more than 22,000 items from just over 850 disparate sources. Of those, Reader says I shared about 11 to 12 a day (down from my previous pace), but that I only clicked through 112 times in total, just under 4 clicks a day, or a click rate of just over one half of one percent. That means for every 200 stories that flow into my Reader, I click through once, and share three, picking the best 1 to 2 percent of the tech Web.

My Monthly Stats Show I Share More than I Click

Google Reader's latest addition to its array of statistics is found in the Trends section of the service, and given the array of options that you have to interact with the feeds, you can now see your 30 day trends for Reading, Starring, Sharing, E-mailing, Mobile Access, and finally, total clicks.

I Click On Gizmodo and Daring Fireball More Than Other Sites

The Most Frequently Updated Sites Don't Dominate My Clicks

So what drives a click, rather than reading within the site? For me, it can be as simple as a truncated feed that forces one to click and find the full story. In other instances, the article may be interesting enough to comment on immediately, or there may be rich media, such as embedded videos, that just don't work perfectly in Reader. When I go to Reader, my goal is to read all items in the service, and venturing out of the site to a specific story simply gets in the way, often. When I do open up a story, I do so in a new tab, so I can finish the task of getting my unread count to zero, and then marching left to right through the individual stories that have action.

Reader's data shows I am a skimmer. I have to be a skimmer who analyzes fast and acts fast to choose the next step for each new piece of content. Others may show much different percentages - and I bet even if I spend less time staring at a Twitter stream than I do on Reader, I do more clicking in Twitter URLs, given they don't ever include the full text. So I wouldn't chalk up my numbers as saying that RSS decreases engagement, but it does show it's likely a massive majority of readers are the strong silent type who read within Google Reader and don't go to the downstream blog.

Find your stats here: http://www.google.com/reader/view/#trends-page.
You can find my shares here: http://www.google.com/reader/shared/louisgray

Google Revamps and Simplifies Contacts In Gmail

Gmail's introduction in 2004 changed the way people interact with Web-based mail, moving us away from needing to carefully prune old messages to save space, and introducing a conversation-oriented interface, with new label-based hierarchies, away from the desktop metaphor of folders. But while Gmail's innovation has been clear, Google's approach to Contacts has lagged, making it the target of frustration from many users. Today, the company admits as much, saying it's the thing people "are least satisfied with" about Gmail. They hope a number of improvements, rolling out immediately, will help.

The goal of the many changes introduced to Google Contacts today is to make using Contacts as intuitive as Gmail, bringing keyboard shortcuts, automatic saving, and the ability to undo edits you may have made in error. This way, if you already are a Gmail power user, you will be a Contacts power user as well.

Sorting by Last Name in Contacts

In addition to the "Gmail-ization" of Contacts, the company added the option to sort by Last Name, the ability to add custom labels for phone numbers or other items, structured name fields, for titles and other details, and a more prominent notes field, so you can say just why somebody is in your address book in the first place. Also, with the upgrade, Contacts and Tasks also occupy a more prominent position in Gmail than before, sliding up the left sidebar to the top, just underneath the Gmail logo.

Editing A Contact in GMail

You Can Undo Changes In Contacts

The new Contacts view makes it easy to get all of a person's details from one place, and make edits. You can easily add job titles and family relationships, or "view recent conversations" to see the last time you traded e-mail or chats with the person. This, thanks to the tie in with Google Buzz, also surfaces Buzz posts that match the search criteria.

Merging Duplicates In Gmail

Also, for power users, you can now merge contacts from the "More Actions" menu, if you have made duplicates, or you can choose to "Find and merge duplicates", handy for the many of us who have made imports only to find friends' records multiplying. I myself merged 75 duplicate records, so I don't have names repeating ad nauseam when I go looking for numbers in my Android phone - closely tied to Contacts.

As somebody who was more closely tied to the Apple Mail and Address Book infrastructure, Google Contacts hadn't always taken a front seat in how I manage people and relationships. Moving to Android, adopting GMail more strongly, and seeing the rise of Buzz as a platform has made getting this right increasingly important. Today's moves will make it seem more natural and fluid - as Apple used to always get right, but for the Web, not the desktop.

WeeSpaces.com Makes Inane Foursquare Check-ins Cool

Little warms a geek's heart like new and innovative data visualizations. With our every activity being tracked and reported as data - including our comings and goings, our preferences and purchases, we too are becoming part of that data graph. While each individual check-in to your favorite location based service may not pique your interest, a visualization of all your activity over months, displayed in chronological order, can take you out of humdrum land and make you look like a person of action.

WeePlaces.com built and released a visualization of FourSquare activity this weekend, and it has already gotten quite a bit of use from those addicted to the new world of geolocation.


Now, all my silly check-ins at 7-Eleven, church, client offices and more, show me racing up and down the Peninsula, with the occasional jaunt into the East Bay (see the embedded video above). If you're a Foursquare user, and you want to relive your own life, start at http://weeplaces.com/foursquare and run your own visual.