Disruptive Telephony

As I've written about previously (here and here), with all the excitement about Facebook's application platform there haven't been a whole lot of apps focusing on VoIP.  Back on August 15th, though, another VoIP/telephony app did emerge, although at the time I was too caught up with VoiceCon travel preparations to blog about it. 

The application is "Reb Me" (actual FB application here) from the folks over at RebTel.  Essentially it allows RebTel users to very quickly call each other at cheap rates.  (With RebTel, you get a local phone number to call international friends.  You therefore only pay the costs for your local call, and some small rate to RebTel (or free).) Given that I'm rather "late" to the game on this app, I'll point to other coverage for more details:

• RebTel blog:
• A developer's take on our new Facebook app
• Reb Me brings mobility to the party
• blog of some of the developers of the RebTel Facebook app
• Jeff Pulver: Rebtel: One Year Later. And now available on Facebook (Q&A with RebTel CEO Hjalmar Winbladh)
• Russell Shaw, ZDNet: Rebtel adds Facebook calling capability
• Facebook Observer: Facebook Gets Another VoIP Application

According to the Facebook stats (in the new way they are counting) there are 84 daily active users of the app, as of today.  The RebTel folks have also launched a Facebook group for RebMe users, although there doesn't seem to have been much activity in the group after its initial launch.

I'm also not entirely sure I'd call it a "VoIP application", because it's really all about making cheap calls from your mobile phones.  No doubt there is VoIP involved someone in the infrastructure, but when I think of a "VoIP app", I'm really thinking of something that does use VoIP protocols (like SIP)  and that involves IP endpoints in some fashion.  In any event, it's cool to see another telephony app inside of Facebook.  Kudos to the RebTel team for launching the app.

FYI, over on his blog, Pat Phelan put together a list of Facebook voice apps.  Nice to see... and I was not aware of the GrandCentral app (although, since you have to copy/paste the web call button embed code from your Grand Central page, the "app" really doesn't seem to be anything more than a box on your profile that can contain HTML code... and a way for the app developers to put some Google Ads in front of you on your Facebook app page).

Since voipstutter thinks I should rename this blog "Skype Telephony", I may as well feed into that image by noting that, per the Skype blog, Skype is now 4 years old.  I was not a user then, but judging by the screenshots of the client in 2003, it certainly has come a long way.  Whatever your opinions of Skype, it certainly has caused disruptions in the world of telcom... and it definitely does keep things interesting.  Congrats to the folks at Skype for hitting that birthday.

Given that there was a service called Jajah, I suppose it was inevitable that someone would come up with a competitor called nonoh whose main point seems to be "We are cheaper than Jajah"! Indeed, if Jajah could run a table like this on their website:

You can pretty much expect that a site like nonoh is going to run the inevitable comparison to Jajah:

The race to the bottom and the commoditization of all phone calls (at $0) continues... who will get to the bottom first?  (And what, exactly, will their business plan be?)

Tom Keating has more info about the difference between Nonoh and Jajah.

I noticed late last night that this weblog was added to the "Friends of Skype" blog roll that appears on the side of Skype's blogs.  Thanks, Skype, for adding me... I certainly don't mind the potential for others to find me. I do write about Skype, although often some of that writing can be critical.

I'll note, too, that Skype's list is actually a great compilation of the various bloggers out there who write about VoIP and collaboration.  Many of them would think of themselves as  (and accept the label of) "VoIP bloggers", but others are from the larger blogging space (ex. Neville Hobson, Robert Scoble, Ross Mayfield).

At VoiceCon this week, one of the smaller booths that looked quite interested was that of OnState, a company launching a Live Chat capability for their call center solution based on Skype.  With my schedule at the show, I was unable to spend much time going through a demo, but what I did see looked quite interesting and I thought I might blog about it at some point.

However, it turns out that Jim Courtney over at Skype Journal has written a rather comprehensive post about OnState, with all sorts of links to prior articles, resources, etc.  Now, Jim wrote his story primarily to address issues he found in two other stories about OnState, but the result is a nice profile of what the offering is all about.

I would have to wonder, though, what kind of reaction OnState was getting at VoiceCon given how recently the massive Skype outage was.  Anyway, I would recommend checking out Jim's article and the various links off of it.

Wow!  From the stage here at VoiceCon, Michael Rhodin, General Manager of Lotus Software at IBM just announced that they are acquiring WebDialogs, the makers of Unyte!

IBM just issued a news release announcing this acquisition.

For those of us following the Skype space, Unyte has been known for a long-time as one of the leading business/collaboration extras available for Skype. 

Congrats to the WebDialogics folks and it will be VERY interesting to see where this goes!

In answer to the many questions raised after their announcement yesterday, Skype today issued a clarification FAQ: "The Microsoft connection clarified".  For starters, they clearly stated that there was no connection to Microsoft:

1. Are we blaming Microsoft for what happened?

We don’t blame anyone but ourselves. The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it. And Microsoft has been very helpful and supportive throughout.

The high number of post-update reboots affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources at the time, prompted a chain reaction that had a critical impact.

(more in the FAQ)

In response to the question that many of us raised (for instance, in my post yesterday) about "why this month's update?", Skype wrote:

2. What was different about this set of Microsoft update patches?

In short – there was nothing different about this set of Microsoft patches. During a joint call soon after problems were detected, Skype and Microsoft engineers went through the list of patches that had been pushed out. We ruled each one out as a possible cause for Skype’s problems. We also walked through the standard Windows Update process to understand it better and to ensure that nothing in the process had changed from the past (and nothing had). The Microsoft team was fantastic to work with, and after going through the potential causes, it appeared clearer than ever to us that our software’s P2P network management algorithm was not tuned to take into account a combination of high load and supernode rebooting.

The FAQ continues with answers to more questions.  All in all, a decent set of responses. 

Jim Courtney over at Skype Journal also offers his perspective on this statement.  I join Jim in thanking Skype PR for recognizing that their initial responses were insufficient.  I think a common thread in many responses to the initial communication was that we were looking for more transparency. Stuart Henshall, a very early Skype user and one of the founders of the Skype Journal, posted a good piece about communication and Skype:  "Outing Skype Communications".  Well worth a read.

With that, we can probably wrap up the outage coverage...  it looks like Skype users are already back to using it.

P.S. Phil Wolff just put up a post "Skype and Crisis Communications 2.0" that looks like a good read...

Mashable.com often comes out with various lists, and today they offered "Skype Toolbox: 50+ Enhancements for Skype" which offers a nice list of the various add-ons that have been developed for Skype. Some good ones in here that I know of... a number that I'd not yet heard about.  The Skype ecosystem continues to grow...

(Hat tip to Julian Bond for pointing out the list today.)

Well, the official word is out from Skype and it can be summarized: the reboots from Microsoft patches triggered a previously-undetected condition and crashed out network. 

Skype PR staffer Villu Arak writes in "What happened on August 16":

On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Okay... I can buy that this type of thing could trigger some kind of chain reaction, but I don't understand why this month was different than any other month.  For.. what? two or three years now (more?) Microsoft patches have been coming out like clockwork on the second Tuesday of each month.  Each second Tuesday or Wednesday, the millions of computers set to auto-update do so.  All those zillions of computers restart automatically.  Each and every month.  What was so special about this August that was different from every other month?  Was the number or restarts in a short period of time really that much different from other months? Why? Is the issue that there are so many more Windows Skype users than in previous months and years? Was this just the so-called "tipping point" when there were enough Windows Skype users that the normal restarts triggered this chain reaction?

The issue has now been identified explicitly within Skype. We can confirm categorically that no malicious activities were attributed or that our users’ security was not, at any point, at risk.

In other words, it was not a DDoS by Russian hackers, as one rumor had it (which had actually already been dismissed by every security researcher who looked at the alleged exploit code).

This disruption was unprecedented in terms of its impact and scope. We would like to point out that very few technologies or communications networks today are guaranteed to operate without interruptions.

Fair enough statement - if you are looking at data or web technologies... but the PSTN, to which Skype would seem to like to be compared, is designed to operate without interruptions (or with as minimal as possible).  You know, there is this wee little market for "carrier-grade" equipment/software/etc. that is designed to be highly available without downtime.  If a carrier's network were down for over 48 hours, there would be a zillion lawsuits, intense government inquiries and more.  The carriers that make up what we call the "PSTN" put an incredible effort into ensuring availability.  If Skype wants to play in that game, they have to be ready to play at the same level.

Skype has now identified and already introduced a number of improvements to its software to ensure that our users will not be similarly affected in the unlikely possibility of this combination of events recurring.

Good. We would expect that.

I appreciate that Skype has been as communicative as they have through their blog and heartbeat site.  Thank you, Skype, for communicating - and leaving the comments open.  However, to me the information provided today is still lacking one key piece:

Why were the mass restarts associated with the August 2007 Microsoft updates different from the mass restarts associated with any other month's Microsoft updates?

According to Skype's "Update at midnight GMT", the Skype outage should be resolving over the next few hours:

We are pleased to announce that the situation continues to improve. The sign-on problems have been resolved. Skype presence and chat may still take a few more hours to be fully operational. We know what our faithful users have been going through and we thank you for your patience and kind support.

Skype has come back online for me and the Skype client tells me there are now over 4 million users online, which is the first time I've seen it show that since Wednesday.  Given that it's been going up and down all day, I'll withhold judgement for a while, but that number does seem to be climbing (if we can believe the number, which is an open question).  My contact list is slowly repopulating with its normal list of people.  We'll see.

So the question really is - will Skype see any long-term impact because of this outage?

Certainly in the short-term Skype will have an awful lot of explaining to do.  There are certainly some number of business users who have come to rely on SkypeIn and SkypeOut.  I know a good number of freelance consultants who have cut their landline and give out their SkypeIn number to everyone and use Skype or SkypeOut for their calls.  Countless small startups are "virtual organizations" where Skype is used for all the communication. Skype will definitely need to reassure them that this won't happen again.    I expect that many of them will be looking now at what alternatives they have, if for no other reason than to have a backup.

Skype's competitors are definitely circling like sharks that smell blood in the water.  Sightspeed's CEO came out with a very pointed blog entry today, "All Peer-to-Peer Models are NOT Created Equal", about how their p2p network was better.  Supporters of Damaka seem to be running around leaving glowing praise and inviting people to try Damaka on various blog entries related to the Skype outage (including my own).  There have been any number of blog entries and comments out there extolling the virtues of the Gizmo Project.  Even Jeff Pulver couldn't resist and tossed in a tweet encouraging people who were missing Skype to try out FWD.

Being the open standards geek that I am, I would love to believe that Skype users would try out many of the other services, many or most of which are based on open standards like SIP.  I'd love to see the level of interoperability you have can have with things like (SIP-based) Gizmo.  Skype's walled garden approach does concern me.  I'd love to see massive adoption of some service to which other services could easily interoperate - and which could have peer-reviewed protocols that were definitely secure.  I'd love to see all of that - and that's certainly a potential outcome.

But I'm also a bit too jaded to think it will really happen.  The seductive aspect of Skype is that it makes it very simple and easy to use the product.  It has (at least for the moment) a massive directory of users.  It has APIs that let it be readily integrated with other web sites and services.  It's easy to use... and I expect that probably many if not most users will simply go back to using it as they have always done once it comes back into full operation.  People have short-term memories... 

If there are any beneficiaries of this outage, I would think that they might be AOL, Microsoft, Google or Yahoo!, all of whom have similar consumer services and have voice as an option now.  Skype's certainly been perceived as the leader with regard to voice... but all of those others are there... and they all have simple interfaces and massive directories as well.

What do you think?  Will Skype suffer any major impact because of this?  Will the newer services like Gizmo, Sightspeed and friends win customers?  What about the other consumer IM services?