Editor’s Note: Part 12 in an irregular series of stories from my many years in Silicon Valley. Part 11 talked about the time I got called into HR's office to meet with lawyers over industrial espionage. This time, a story involving gray hat search engine marketing in the early days of the Web.

DMOZ is now closed.

Believe it or not, before the world of automated spiders that crawled the entire Web and ranked the results for your searches, much of the way we found content on the Internet was thanks to manual updates from an invisible army of directory editors. Yahoo! defined the initial dot-com era, with its hierarchical oracle making or breaking traffic downstream, as sites were organized and shuffled into categories by unseen text tweakers, much like the editors of Wikipedia try and keep its tens of millions of article pages up to date, with a seemingly fluid mass of edits to keep the live encyclopedia current.

But Yahoo! wasn't the only Web directory. Rich Skrenta and others, also behind Web 2.0 efforts Topix and Blekko, introduced the Open Directory Project, referred to as DMOZ, for it was hosted on the directory subdomain of the Mozilla.org site, with an objective of harnessing contributions from around the world (like Wikipedia), to build a directory, similar to Yahoo!, that could plug into any site that wanted to host one. In a time when many sites were seeking Internet traffic and riches by acting as the front page for the Web, attaching the open directory project to your portal could give you an edge and not require you to bring on a ton of staff.

As with Yahoo!'s directory, a company's inclusion in the DMOZ directory could act as a binary gate as to whether or not potential users would find you. In 1999, working as a Web marketing manager for a Web startup that offered internet faxing and conference calls, I found myself irked to see that our services were not included in DMOZ. Making things worse, the categories I would expect to see us listed in seemed slapped together and without an official owner. Given my understanding of the space and knowing our many competitors, I registered an account and requested to moderate the relevant category.

The DMOZ Internet Fax Listings from 1999 (via Archive.org)

Not too long afterward, I was given the option to update the category, including those of our competitors. As it was nearly two decades ago, I honestly don't remember if I used my company email or a Yahoo! email or equivalent, but I didn't try and disguise where I worked. My application had gotten through.

FaxCube from the year 2000.

When I did log in, I found the content in a state of abandonment. There wasn't much you could do with a site's listing. Give it a title, a link, and a short description, literally about a dozen words. It was fairly impossible to differentiate services from one another, especially in a commoditized space where the core function was pretty straight forward. But I cleaned up the descriptions for all the entries, including our competitors, to accurately display their offerings. Some offered email to fax services, while others offered the reverse. Some offered broadcast faxing. Some required a proprietary fax viewer, and others were completely Web based. That kind of thing.

When content was edited in DMOZ, edits would later propagate downstream. Sure enough, my colleagues noticed a spike in Web traffic to our main sites, with referrers coming from all the places DMOZ was set up. For no cost, I had a clear impact in our customer acquisition funnel, and maintaining the DMOZ became part of my ongoing work.

Later, DMOZ added the option to highlight two entries in the group, which were solely up to the moderator. This, of course, gave me the option to expand from a gray area, to clearly going over the line into promotion. It went without saying I thought our service was the best, and highlighted it at the top. I also chose to highlight a partner site (the Netscape Fax Center) that was white labeling our service, essentially the 1 and 1A positions.

This got even better when we soon realized DMOZ was fueling search results for AOL. The more people who searched for Internet Faxing on AOL, the better we were to see results.

DMOZ listings for Internet Telephony in early 2000. (Via Archive.org)

In early 2000, we introduced a new Web based conference call and meeting product. Addicted to the free traffic from DMOZ, my team asked me to see if I could get our content similarly included on whatever the equivalent was for Internet conferencing. I poked around, and, again, applied to be an editor for the Internet Telephony page. It wasn't a perfect fit, but it was pretty good.

Again, I got approved, and again, I added our sites, and again, I promoted our main site and a partner site to be included as the pair at the top of results. Again, we started to get plenty of Web traffic from DMOZ and its downstream partners, accounting for more than double digit percentage of our traffic to both properties. But this time, it wouldn't last long. Maybe it was the clear marketing copy promoting both services, but one day, I logged in and saw our service demoted to the standard results, with another in its place. I reversed the change, and it wasn't too long until there was a debate upstream as to whether these listings were in good faith.

PhoneCube, in all its glory, from 2000.

Not too long afterward, my moderation rights had been removed from both sections. I'd essentially been fired from my side job of gray hat SEO, long before most of us knew what that meant. And yes, my engineering peers lamented the loss of traffic, as our more organic listings, on AOL search and elsewhere, didn't carry as much weight, once we became one of the fold.

Had I just stayed content to include our services in the listings, or even just to stay a moderator of the less exciting Internet faxing space, it's possible I could have been editing text descriptions for our sites and our competitors indefinitely. But pushing our own products and our partner sites turned out too good to be true. The "Wild West" Internet traffic rush didn't last forever, and, as it turns out, neither did our products. I was laid off after we couldn't get a funding round closed at the beginning of 2001, and a few months later, the company was parceled off to become part of Oracle (see 2006 post). Some of my colleagues still work there nearly two decades later. As for DMOZ, it too shut down a few years ago, a relic in the world of Google and Facebook.

Disclosures: This whole post is a disclosure! I worked at 3Cube during this time. I work at Google now, a perceived competitor to whatever Yahoo! and AOL call themselves now. And as I work on Google Analytics, this is not a post about SEO to all you SEO/SEM folks.

                              
Two of the most heavily anticipated and visible product debuts in the past month have been Blekko, the slashtag-happy curated search engine powered by humans and fancy algorithms, and RockMelt, the new social-centric Web browser that drags Facebook alongside throughout your page perusing experience. Both took years of development, sport dozens of employees and raised significant capital. Both are intriguing enough to gain curiosity from early adopter tire kickers. And both are raising eyebrows from skeptics who wonder why otherwise smart folks would charge headlong into what are clearly mature markets with many high profile competitors. For every ubergeek checking out the latest toy, there is somebody else wondering if the world needs "yet another" search engine or Web browser.

Short answer: The world always needs new innovation, and there are countless examples of companies entering into crowded markets that needed some shaking up, who found tremendous success - including in these two markets.

Corollary: To consider either Blekko or RockMelt a success does not require either of them to topple the dominant market leader. Blekko is not a Google search replacement in most cases, and Rockmelt won't necessarily have you uninstalling Firefox. If they can get 1%, 5% or 20% of the market, it is significant indeed. (See: Don Dodge from 2007: Why 1% of search market share is worth over $1 Billion)

Google Takes On Crowded Search Market, Firefox Takes On IE

Google's entry into the search market at a time when Alta Vista could be considered the quality leader, with additional players such as Excite.com, Infoseek, HotBot and yes, Yahoo!, may seem like ancient history now, but the company looked at an industry which needed disrupting. It obviously worked. Meanwhile, Firefox emerged, through Mozilla, out of the ashes of Netscape, burned by the overwhelming market leader, Internet Explorer. It's now safely assumed Firefox is the strong #2 to IE, with Google's own Chrome holding the #3 position and Apple's Safari at #4.

One could highlight the opportunity for reinvention of new markets for some time strictly by looking at Apple and Google alone, even though there are many more examples, from big companies and small alike who have looked big brand names in the face and pushed ahead anyway.

Google Reader Emerges to Take On Bloglines

If you read Chris Wetherell's fantastic two part series about the birth of Google Reader, and Mihai Parparita's companion piece about how he joined the team, you can see significant skepticism over whether the world needed yet another feed reader when NetNewsWire and Bloglines did just fine. Reader now is easily the #1 share holder and Bloglines just closed up shop.

Apple Introduces an MP3 Player and a Phone!

In October of 2001, Apple introduced a hard-disk based MP3 music player. The MP3 player, called iPod, launched with competing players like the Creative Nomad Jukebox and Rio's line of digital audio devices already in the space. That they launched an MP3 player seemed like madness. To others, their launch of the iPhone was similarly crazy, even after the iPod's runaway success. Resident curmudgeon John Dvorak warned in 2007 that Apple should pull the plug, adding "It's the loyalists who keep promoting this device as if it is going to be anything other than another phone in a crowded market."

The crowded market had plenty of room for Apple.

One doesn't have to look too far away from this space to find that Google's Android OS took on the successful iPhone, as well as Symbian, Blackberry, Windows Mobile and others, and found opportunity to grow significant market share. Facebook launched into a market occupied by Friendster and MySpace. Gmail was introduced years after Yahoo! Mail and Hotmail had amassed millions of users apiece.

Is it going to be a challenge for Blekko and RockMelt to separate themselves from aggressive competitors? Absolutely. But often, empty markets without participants are empty for a reason - and crowded markets are there because there is potential seen by multiple players. Blekko and RockMelt each now have the opportunity to compete on their merits, and attract their own target users. It's not necessarily a zero sum game, and both seem to competitive enough to bring value, with enough humility to not trip over their own hubris.