February 07, 2008

Limelight Networks Searchme Spider Picking Up Speed

For the better part of this year, I've seen some odd traffic to my site from Limelight Networks, a content delivery network similar to the more well-known Akamai. Multiple times a day, I would see a visit, with no originating page, drop in on the site, look at a page or two, and then, just as quickly, it would leave. Later, this pace quickened, to the point that if I saw a visitor came in with no referrer, I assumed it was llnw.net. But, in the last few days, the rate has dramatically accelerated, to the point where this spider from llnw.net is more than 10 percent of my total traffic, and it rates as the number one domain accessing my site, ahead of even Comcast.

So what the heck is it doing?

Well, it's not 100% clear. The assumed spider drops in and advertises itself as Linux UNIX, running Mozilla 1.8.1.11, and displaying a monitor with the odd square resolution of 1300 x 1300. It generally looks at one page, takes off, and then comes back in a few minutes to get another one. No real rhyme or reason, and it's just as happy to suck down old pages as new ones.

So, is it caching my blog so that customers of Limelight Networks can access the pages faster? Is it taking a graphical snapshot, in the same way that www.archive.org has done to show how Web sites looked over time? I'm not exactly sure.

One theory, voiced in the forums at Webmaster World, titled "Unusual Traffic from Limelight Networks", says the activity is from a robot called "Searchme", a LLNW client. Going to www.searchme.com shows this as a possibility. Searchme, Inc. says, " Searchme delivers more meaningful and targeted search results to its users," and that its "intuitive category suggest technology provides users with a dynamic and rewarding search experience by delivering relevant results that are tailored specifically to their unique areas of interest."

But... alas, no search engine and no demo, yet.


LLNW.net keeps hitting the site, every few minutes.

While I'm sure my site and others get hammered all day long from Google and Yahoo! spiders, they don't trigger my Web statistics software to think they're actual people, as LLNW is doing. It could be because their spider acts so human-like that the JavaScript code I use to track accesses is fooled. But regardless, the activity is picking up steam, and at this pace, I wouldn't be surprised to see LLNW take in one of every five visits here, even if they are "junk" visits.


LLNW.net is grabbing 13% of my traffic, beating out even Comcast!

The question is, if it is Searchme who is pushing this spider, and if they are indeed planning to reveal their work at some point, will the world have the need for yet another search engine? I have no idea. But if these oddities are any indicator, something's going on worth watching.

For those of you who have blogs and Web sites that track this detail of activity, are you also seeing the LLNW traffic, and has it increased over time? Also, has anybody ever seen Searchme in action who can let us know what they're doing?

4 comments:

  1. Following on... regarding Searchme.com, yes I realize they are behind WikiSeek, at http://www.wikiseek.com/. But I don't know that it's WikiSeek generating this oddness.

    ReplyDelete
  2. I think it is a site profiler, which is different than a spider. Profilers test for latency, MIME types, SEO, and other arcane stuff.

    ReplyDelete
  3. I keep getting it too. . .I thought who is this and what are you doing on my blog. Then I thought hey, maybe I am special.

    I still don't understand why on earth they would come to my blog though.

    LOL

    ReplyDelete
  4. Don't know if this might help, but I just installed "Flock" as a web browser since Netscape is now obsolete (by the way, Netscape users are getting a message to upgrade to Flock). While I was opening Flock for the first time and taking a look around I started getting constant connection alerts for llnw.net. In trying to do some research on who they were and why there were so many connections I came across your posting. Flock has feeds which might be the connections you are seeing on your server, and they might look like users because they are coming from client sessions. Just a guess. All I know is as a user I don't like the constant connections and bandwidth usage I'm experiencing! It also makes me very suspicious as to what it is doing!

    ReplyDelete