By popular demand, I've been asked by other FriendFeed users to highlight how I use the popular social lifestreaming site. So far the series has covered the "Hide" function, the bookmarklet, advanced search, how to integrate with Google Talk, how you can incorporate comments and determine an item's original source. Today, figuring out who you're engaging with on the service.

So far, unlike most social networks out there, FriendFeed doesn't utilize a person's profile as the central nervous system around which the service is built. There's no way to post my location, my age, my job title, my birthday, my religion, or marital status. For some, this is surely a blessing. For others, it seems like a feature gap. Filling that gap, until FriendFeed does it in the future, is a GreaseMonkey script developed by Hao Chen. But for non-Firefox users or those who don't play with scripts, you can use FriendFeed's stream to get clues.

FriendFeed's stream consists of a collection of updates from disparate services around the Web, from you and your friends. The nomenclature, as discussed in last week's tip, consists of service icons, an active verb, and the service itself.

Clicking on any of these icons reduces the total stream to show only updates from a specific service. For example, I can click the orange RSS icon and see only blog entries from my friends. I can click the Twitter logo and see only Tweets. But if, over time, I've started following a number of people who are friends online but not in real life, as is common, the very best way to find out who they are is by showing only updates from LinkedIn. If your contacts are using LinkedIn, and have registered it with FriendFeed, there will be at least one entry, and if they update, you'll see if they changed jobs or got promoted.


The URL to show all LinkedIn updates from your friends is:
http://friendfeed.com/?service=linkedin.

The URL to show all LinkedIn updates from people throughout FriendFeed is:
http://friendfeed.com/public?service=linkedin.
(Have at it, recruiters!)

From this, I found that Alan Steele, formerly of Mergelab, is now the VP of Engineering at Identity.net, Drew Olanoff of ReadBurner is now the Community Manager and Evangelist at MyStrands and Atul Arora is the Director of Product Management at Vimo, for starters.



Yes, it turns out that most of the geeks who participate online are also geeks in real life. That's not a surprise. But rather than just knowing somebody from their blog or their tweets, or even their photos from Flickr, with a single click, you can browse all the titles on LinkedIn and see what these people do in the real world. And just like with any other FriendFeed entry, you can like a LinkedIn entry or make a comment. It's a great way to congratulate somebody on the social ladder as they move up the corporate ladder.

As for more MySpace/Facebook looking profiles, check out Hao Chen's scripts or... just wait. I bet FriendFeed will solve this soon enough.

Data publishers and data consumers can be both friends and enemies, when it comes to the seemingly infinite demand for and growth of real-time data from Web services. Services like Digg, Flickr, Del.icio.us and Twitter are happy to see their user bases expand, and to see developer communities be built around their products. But with each new application hitting their API, and each new user, comes new demand that can put strain on their infrastructure, even if the outside application is just checking for updates which aren't there. Gnip is looking to act as a go-between for data publishers and data consumers, delivering updates from the services to the applications, and reducing the queries that can drag popular sites to a crawl.

Today, Gnip made headlines with an announcement that Twitter notifications would be sent to the service via XMPP, letting outside developers tap into Gnip instead of adding more strain to the embattled microblogging giant. And while this won't solve all of Twitter's issues, it does offer developers an alternative, taking some of the power out of Twitter's hands. The announcement did not contain any money changing hands, done quid pro quo.

After the morning's hubbub, I talked with Eric Marcoullier, CEO of Gnip, to better understand how adding Twitter to the team's growing array of partners would help users and developers, and whether this solved the growing concerns around Twitter's API limits that have seen application authors frustrated. And the answer so far, is that Gnip can solve some problems today, and is preparing to solve more issues soon. But it won't make Twitter's problems disappear.

"All these data protocols can be an exponentially scaling hassle. I like how people thought Gnip would single-handedly fix Twitter's problems, but that minimizes how big Twitter really is," Eric said. "Some developers don't just want the user stream, but the reply stream as well, and others want Track, which bangs against the Summize (now Twitter) API, to find if there is a new tweet that has a followed word. We might not ever solve that. It's a big scaling problem of reading the content, and it doesn't fix all of Twitter's problems."


Gnip's Data Flow Architecture
What Gnip is looking to do is help publishers looking to syndicate their data, and consumers, who are building businesses off user generated data, by simplifying the complex back-end work needed, and giving entrepreneurs more time to work on the front-end of their product, delivering tangible benefits.

"We're able to go to them and say, all of the effort you are doing to aggregate that data, stop now," Eric said. "Tell us what services you like, what protocol you like, and the data magically appears in real time."

While Twitter has been the most visible client so far, it's by no means the first for Gnip, which launched with two partners out of the gate, in Plaxo and MyBlogLog, where Eric was a co-founder. Since launch, Gnip has also penned partnerships with additional services, including Lijit and Iminta.


Gnip's Growing Partner Roster
Today, developers of applications are authoring products that query popular sites, like Twitter, Digg and Del.icio.us, and do so thousands of times a day, even if the overwhelming majority of the time, there are no updates. Where Gnip works well is for centralized services, like Plaxo, who can dramatically cut back on the amount of times they need to make requests. "It doesn't matter how many people are following an individual on Plaxo Pulse. They just have to ask once." Eric said.

But the way centralized services make queries to Twitter is different than the issues faced by the many apps that are struggling against the 100 API calls per hour per IP address limit discussed yesterday. For that, more work is needed.

"For the average user, 100 queries per hour is fine, as long as you're only querying the API when there is new data," Eric said. "But for, say Thwirl, where they each have their own user connections, they would have to query maybe 50 times, and that's half the load. We're looking for a simple way of creating anonymous buckets, so somebody like Twitter Karma can say 'we have 10,000 users with this collection', and we can centralize it. We're still a ways away from helping folks with distributed clients."

Gnip's initial efforts and partnerships have been completed on the first version of their product. In about a month (or two), the company expects to not just send service notifications to partners, but also, the full metadata, which will bring more rich information from its many supported services, including Del.icio.us and Disqus, Flickr and others. Maybe, at that point, you'll also see Twitter passing on reply streams as well, but that's not set in stone.

"Working with people like Twitter, we want to be sure we are serving their best interests and the developer community," Eric said. "It's a huge win for us."

Guest Post By Colin Walker (Twitter/FriendFeed)

On Wednesday, Matthew Ingram posted a reaction to Mike Arrington's interview with Twitter founder Evan Williams with regards to the way Twitter is just handing their data to potential competitors for nothing.

It has seemed obvious to me for a while that an ideal aspect of a business plan for a social networking service such as Twitter would be to charge partners for premium access to the API, but once you have started down the free path, is it possible, or wise, to backtrack and start charging?

At the inception of a service, the desire is to gain popularity as quickly as possible - the API allowed Twitter to gain this popularity - the openness fuelled a separate ecosystem and Twitter usage spread far beyond what the service would have been able to generate on its own. At this stage charging for access would have stifled the creativity surrounding Twitter and the vast majority of third party apps would never have existed.

The question becomes: who would want to use the data from the Twitter API so badly that they would be willing to pay for it?

Let's look at the four companies allowed access to the full XMPP data:
(prior to this morning's announcement with GNIP)

• Summize would never have existed in its current format and would be pointless with only limited access to the API
• Twittervision is merely a mashup and the developer would, no doubt, have never gone this route if there was no free access
• FriendFeed itself also has no obvious business plan so where would the incentive be to spend money it doesn't have?

Perhaps the only company with full access that may consider paying would be Zappos but only because it is a "real" (if online) business rather than a social media startup. As Evan alludes to in the interview it is the distinction between commercial and non-commercial use that is the potential driving factor here.

The web can only exist on the handouts from VCs for so long before someone, somewhere starts demanding a return on their investment. Where is this going to come from?

So, is it possible to start charging for access?

The news that Twitter have limited unauthenticated calls to the API as well as authenticated calls could well be the first step on the road to pay-per-play but who should or shouldn't be paying and can Twitter now justify a shut out of this nature against the revenue it could potentially receive?

How many third party tools will wither and die? How many developers will quit before their project ever sees the light of day?

When I wrote about Twitter shutting off the Replies tab in favour of keeping the API open I queried if this was the most sensible route - why restrict your own application to protect those of others? The argument was that Twitter receives ten times more traffic via the API than it does from the web site so, as Duncan Riley commented, why "would you willingly block half your user base?" isn't that what restricting API access is effectively doing?

This information came from an interview with Biz Stone in September last year in which he also explained:

the API becomes not only crucial for us on a creativity level and something that we can offer to the developers so that they can build their own applications and experiences, but it also becomes a way for us to grow and a way for us to potentially - depending on what business model we choose - do well there, business-wise (my emphasis)

The constant denial that Twitter knows where they are heading with their business model seems at odds with the constant messages we get when we read between the lines. Even back in September it was viewed that the API could be a money spinner. The moves to restrict API access, in my opinion, are reinforcing that message even if Twitter themselves argue that they don't know where they're going.

As Bob Dylan said: The times thay are a-changin'.

Twitter's struggles with handling high user load have been well documented. To help the embattled site stay up between Fail Whales, they've at times reduced features to just try and keep afloat, blocking the replies tab, reducing pagination, and blocking Instant Messaging, for example.

They've also visibly tweaked the rate for authenticated API hits, first down from 70 to 20 and back up to 100. But until recently, unauthenticated API requests were unlimited, which all changed Wednesday night around 5 p.m. Pacific Time, when Twitter ratcheted them down to the same 100 per hour per IP address, effectively crushing many external services that relied on Twitter for their data. And this was done without public mention on the Twitter blog.

As you can see in this Google Groups thread, Alex Payne of Twitter told developers last Thursday:

"In our continued effort to keep things fast and prevent abuse we're planning on introducing rate-limiting by IP for unauthenticated API requests. We'll allow 100 unauthenticated requests per IP per hour, just as we currently do with authenticated requests. Please let us know if you foresee any ghastly issues with this change."

And while that sounds all nice and dandy, there were a number of developers who did see "ghastly issues", ranging from Kee Hinckley, CEO of Somewhere, Inc., Richard Cunningham of FriendBinder, and Joel Strellner of Twitturly, each of whom runs a product that depends strongly on unauthenticated API requests from the microblogging service.

But it didn't stop Twitter from making the change, even as Jodee Rich of PeopleBrowsr wrote, "this will blow us out of the water."

Since throttling down the unauthenticated API requests, services that rely on this data are a mere shadow of their former selves.


Twitturly says on status page: "Due to recent changes in Twitter's API, Twitturly is slowly dying."

Similarly, Twist's Twitter charts only show a fraction of the data they did at the beginning of the week. (See: "morning", "starbucks" and "computer".)


Twist's Twitter Search for "Morning"

Twist's Twitter Search for "Lunch vs. Dinner"
In response to the changes, FriendBinder's Cunningham said he will now be changing all API requests to Twitter to be authenticated, to work around the problem. He wrote me, in an e-mail, "There are also some parts of the API we cannot even attempt now. User authenticated requests limits are shared with other sites that the user has used - so we might not be able to get any updates for them if some other site hit the limit for them."

You can see this yourself by trying sites that aim to help you bulk follow those Twitter users who are following you. If you use a site like Twitter Karma, and then try to use Less Friends, you will get a nasty note that says you have exceeded your 100 requests per hour limit. And if you're like me, having fallen way behind on following folks, you might be revisiting one of these sites religiously every hour for the next few days until you're caught up.


Less Friends and Twitter Karma Get a Mere Fraction of the Updates
Last month, Jesse Stay wrote that developers were bailing on Twitter, thanks to outages and broken APIs. With changes like this, despite improved uptime of late, the end to Twitter's problems is nowhere in sight, and services that hoped to tap into their API this way are sore out of luck.

Note: These restrictions do not impact the four golden partners of Twitter's XMPP feature, including Zappos, Summize, FriendFeed and Twittervision.