Disruptive Telephony

Since voipstutter thinks I should rename this blog "Skype Telephony", I may as well feed into that image by noting that, per the Skype blog, Skype is now 4 years old.  I was not a user then, but judging by the screenshots of the client in 2003, it certainly has come a long way.  Whatever your opinions of Skype, it certainly has caused disruptions in the world of telcom... and it definitely does keep things interesting.  Congrats to the folks at Skype for hitting that birthday.

I noticed late last night that this weblog was added to the "Friends of Skype" blog roll that appears on the side of Skype's blogs.  Thanks, Skype, for adding me... I certainly don't mind the potential for others to find me. I do write about Skype, although often some of that writing can be critical.

I'll note, too, that Skype's list is actually a great compilation of the various bloggers out there who write about VoIP and collaboration.  Many of them would think of themselves as  (and accept the label of) "VoIP bloggers", but others are from the larger blogging space (ex. Neville Hobson, Robert Scoble, Ross Mayfield).

At VoiceCon this week, one of the smaller booths that looked quite interested was that of OnState, a company launching a Live Chat capability for their call center solution based on Skype.  With my schedule at the show, I was unable to spend much time going through a demo, but what I did see looked quite interesting and I thought I might blog about it at some point.

However, it turns out that Jim Courtney over at Skype Journal has written a rather comprehensive post about OnState, with all sorts of links to prior articles, resources, etc.  Now, Jim wrote his story primarily to address issues he found in two other stories about OnState, but the result is a nice profile of what the offering is all about.

I would have to wonder, though, what kind of reaction OnState was getting at VoiceCon given how recently the massive Skype outage was.  Anyway, I would recommend checking out Jim's article and the various links off of it.

Wow!  From the stage here at VoiceCon, Michael Rhodin, General Manager of Lotus Software at IBM just announced that they are acquiring WebDialogs, the makers of Unyte!

IBM just issued a news release announcing this acquisition.

For those of us following the Skype space, Unyte has been known for a long-time as one of the leading business/collaboration extras available for Skype. 

Congrats to the WebDialogics folks and it will be VERY interesting to see where this goes!

In answer to the many questions raised after their announcement yesterday, Skype today issued a clarification FAQ: "The Microsoft connection clarified".  For starters, they clearly stated that there was no connection to Microsoft:

1. Are we blaming Microsoft for what happened?

We don’t blame anyone but ourselves. The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it. And Microsoft has been very helpful and supportive throughout.

The high number of post-update reboots affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources at the time, prompted a chain reaction that had a critical impact.

(more in the FAQ)

In response to the question that many of us raised (for instance, in my post yesterday) about "why this month's update?", Skype wrote:

2. What was different about this set of Microsoft update patches?

In short – there was nothing different about this set of Microsoft patches. During a joint call soon after problems were detected, Skype and Microsoft engineers went through the list of patches that had been pushed out. We ruled each one out as a possible cause for Skype’s problems. We also walked through the standard Windows Update process to understand it better and to ensure that nothing in the process had changed from the past (and nothing had). The Microsoft team was fantastic to work with, and after going through the potential causes, it appeared clearer than ever to us that our software’s P2P network management algorithm was not tuned to take into account a combination of high load and supernode rebooting.

The FAQ continues with answers to more questions.  All in all, a decent set of responses. 

Jim Courtney over at Skype Journal also offers his perspective on this statement.  I join Jim in thanking Skype PR for recognizing that their initial responses were insufficient.  I think a common thread in many responses to the initial communication was that we were looking for more transparency. Stuart Henshall, a very early Skype user and one of the founders of the Skype Journal, posted a good piece about communication and Skype:  "Outing Skype Communications".  Well worth a read.

With that, we can probably wrap up the outage coverage...  it looks like Skype users are already back to using it.

P.S. Phil Wolff just put up a post "Skype and Crisis Communications 2.0" that looks like a good read...

Mashable.com often comes out with various lists, and today they offered "Skype Toolbox: 50+ Enhancements for Skype" which offers a nice list of the various add-ons that have been developed for Skype. Some good ones in here that I know of... a number that I'd not yet heard about.  The Skype ecosystem continues to grow...

(Hat tip to Julian Bond for pointing out the list today.)

Well, the official word is out from Skype and it can be summarized: the reboots from Microsoft patches triggered a previously-undetected condition and crashed out network. 

Skype PR staffer Villu Arak writes in "What happened on August 16":

On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Okay... I can buy that this type of thing could trigger some kind of chain reaction, but I don't understand why this month was different than any other month.  For.. what? two or three years now (more?) Microsoft patches have been coming out like clockwork on the second Tuesday of each month.  Each second Tuesday or Wednesday, the millions of computers set to auto-update do so.  All those zillions of computers restart automatically.  Each and every month.  What was so special about this August that was different from every other month?  Was the number or restarts in a short period of time really that much different from other months? Why? Is the issue that there are so many more Windows Skype users than in previous months and years? Was this just the so-called "tipping point" when there were enough Windows Skype users that the normal restarts triggered this chain reaction?

The issue has now been identified explicitly within Skype. We can confirm categorically that no malicious activities were attributed or that our users’ security was not, at any point, at risk.

In other words, it was not a DDoS by Russian hackers, as one rumor had it (which had actually already been dismissed by every security researcher who looked at the alleged exploit code).

This disruption was unprecedented in terms of its impact and scope. We would like to point out that very few technologies or communications networks today are guaranteed to operate without interruptions.

Fair enough statement - if you are looking at data or web technologies... but the PSTN, to which Skype would seem to like to be compared, is designed to operate without interruptions (or with as minimal as possible).  You know, there is this wee little market for "carrier-grade" equipment/software/etc. that is designed to be highly available without downtime.  If a carrier's network were down for over 48 hours, there would be a zillion lawsuits, intense government inquiries and more.  The carriers that make up what we call the "PSTN" put an incredible effort into ensuring availability.  If Skype wants to play in that game, they have to be ready to play at the same level.

Skype has now identified and already introduced a number of improvements to its software to ensure that our users will not be similarly affected in the unlikely possibility of this combination of events recurring.

Good. We would expect that.

I appreciate that Skype has been as communicative as they have through their blog and heartbeat site.  Thank you, Skype, for communicating - and leaving the comments open.  However, to me the information provided today is still lacking one key piece:

Why were the mass restarts associated with the August 2007 Microsoft updates different from the mass restarts associated with any other month's Microsoft updates?

According to Skype's "Update at midnight GMT", the Skype outage should be resolving over the next few hours:

We are pleased to announce that the situation continues to improve. The sign-on problems have been resolved. Skype presence and chat may still take a few more hours to be fully operational. We know what our faithful users have been going through and we thank you for your patience and kind support.

Skype has come back online for me and the Skype client tells me there are now over 4 million users online, which is the first time I've seen it show that since Wednesday.  Given that it's been going up and down all day, I'll withhold judgement for a while, but that number does seem to be climbing (if we can believe the number, which is an open question).  My contact list is slowly repopulating with its normal list of people.  We'll see.

So the question really is - will Skype see any long-term impact because of this outage?

Certainly in the short-term Skype will have an awful lot of explaining to do.  There are certainly some number of business users who have come to rely on SkypeIn and SkypeOut.  I know a good number of freelance consultants who have cut their landline and give out their SkypeIn number to everyone and use Skype or SkypeOut for their calls.  Countless small startups are "virtual organizations" where Skype is used for all the communication. Skype will definitely need to reassure them that this won't happen again.    I expect that many of them will be looking now at what alternatives they have, if for no other reason than to have a backup.

Skype's competitors are definitely circling like sharks that smell blood in the water.  Sightspeed's CEO came out with a very pointed blog entry today, "All Peer-to-Peer Models are NOT Created Equal", about how their p2p network was better.  Supporters of Damaka seem to be running around leaving glowing praise and inviting people to try Damaka on various blog entries related to the Skype outage (including my own).  There have been any number of blog entries and comments out there extolling the virtues of the Gizmo Project.  Even Jeff Pulver couldn't resist and tossed in a tweet encouraging people who were missing Skype to try out FWD.

Being the open standards geek that I am, I would love to believe that Skype users would try out many of the other services, many or most of which are based on open standards like SIP.  I'd love to see the level of interoperability you have can have with things like (SIP-based) Gizmo.  Skype's walled garden approach does concern me.  I'd love to see massive adoption of some service to which other services could easily interoperate - and which could have peer-reviewed protocols that were definitely secure.  I'd love to see all of that - and that's certainly a potential outcome.

But I'm also a bit too jaded to think it will really happen.  The seductive aspect of Skype is that it makes it very simple and easy to use the product.  It has (at least for the moment) a massive directory of users.  It has APIs that let it be readily integrated with other web sites and services.  It's easy to use... and I expect that probably many if not most users will simply go back to using it as they have always done once it comes back into full operation.  People have short-term memories... 

If there are any beneficiaries of this outage, I would think that they might be AOL, Microsoft, Google or Yahoo!, all of whom have similar consumer services and have voice as an option now.  Skype's certainly been perceived as the leader with regard to voice... but all of those others are there... and they all have simple interfaces and massive directories as well.

What do you think?  Will Skype suffer any major impact because of this?  Will the newer services like Gizmo, Sightspeed and friends win customers?  What about the other consumer IM services? 

As of this moment, Skype is back working for me and tells me that 3,181,959 people are online.  However, given the performance this morning, I am not expecting it to stay up.  It's been fading in and out all day.

The last we heard from Skype was about six hours ago in their "Where we are at 1100 GMT" post. The comments (currently 81) to that post are quite fascinating to read.  Some are the typical kind of outrage you expect.  Some are passing along the latest speculation. Some are giving reports of continued outage.  Some include links to news articles.  Some bash Skype.  Some praise Skype.  As is typical, some bash other commenters (like this one apparently from a Skype-using-solider in Iraq).  Some plead for a return.  Some suggest alternatives

In the midst of that (and the other entries with comments: here(183), here(54) and here (64)), you see the comments dealing with the human side:

I miss my friends!!

Thanks for the information, I miss my parents in Mexico, they have the same problem as well, hope today the system comes back

come on guys i need to make an important phone calls plz fix it as soon as possibles

I've not seen any changes in service. Pretty fed up because this is important to bme doing business with my clients.

Now I know not to take a conversation with my boyfriend, who currently is living in Denmark, for granted. Irritating indeed

Hey everyone in the world using SKYPE...wow didn't realize how much we all rely on SKYPE. We use SKYPE for our business. That is how we connect to the world. We connect to all of our contacts using SKYPE. We miss it terribly. All I see is a grey X...need to see the green CK MARK! We basically used our cell service and emails it worked..but made it hard to teleconference...tomorrow is another day Skype..hope we are connected.

I had to plan a flight today with some one in the U.K. this was not very easy sending text messages back and forth on our mobile phones. This problem needs to be fixed soon

Please, please try to resolve this system glitch soon. My business relies so much on Skype voice calls. We are losing business connections every hour if skype isn't availabe.

Hurry it up! This is bad for business!!

You get the idea.  (Kudos to Skype, by the way, for leaving their blog comments open during this situation.)  In another forum where someone was venting, the person somewhat frantically wrote this:

yes but I put my skypein phone # on my resume!!!!

Ouch.

I, too, am impacted to a degree.  Although I am more of a casual user of Skype, i.e. I don't rely on Skype for communication, the number I put in the sidebar to this blog is my SkypeIn number.  Why?  Just because I didn't want to put my home or cell number and figured that it might be a good way to test SkypeIn.  (And in the 7 months I've had it up there I've probably had maybe 2 calls on it resulting from the blog!)  But still, this does cause me to rethink that.  (In fact, I may very shortly change the sidebar number to my GrandCentral number as a layer of redirection.)

The outage looks like it will continue for a while. (I am disconnected again in the time it took me to write this.)  And in the meantime there are real people out there suffering because they have come to depend upon a particular VoIP provider. 

Yes, I work for a company that on one level could be seen as competing with Skype when it comes to business (but we don't run into Skype, really), but an outage of this length isn't good for us as an industry.  Already there are voices out there saying that this shows inherent weaknesses in P2P VoIP (it doesn't, in my opinion).  Perhaps the good news is that people are looking around at alternatives and they are asking these questions now about availability. 

Julian Bond said it perhaps best in an IRC chat room today:

This debacle is proving to be a bit of a shocker. I guess we all got lulled into thinking that Skype was at least as reliable as the Cell networks.

Yes, we did.  Some much more than others (especially those who dropped their landlines and used it for business).

In another post when this is all over, I'll write a bit more from the security side about "lessons learned", but for the moment I think we need to remember that there are real people right now being impacted by all of this.  For better or worse, Skype has become a communication that is (or has been) relied upon by many.  Hopefully the folks in Estonia and other locations can get this fixed soon.

In the beginning, there was IRC. 

Well, okay, not exactly... BITNET Relay was around before that and there were other multi-chat environments in some of the walled garden services (CompuServe, GENIE, etc.) and BBSs... but for most of us who were online from the late 1980s onward, IRC was the place to be for "chat" and realtime IM communication.  Of course, it lived primarily in the geekier side of the Internet.  The "real" Net users used IRC and looked down upon all the "newbies" who were drawn to these new IM services from ICQ, AOL and later MSN and a zillion others.  Sure, they were pretty and had cute emoticons.  Yeah, okay, so they could include videos and knew when other people were typing and had little "toast" popups... all that would just be added to IRC clients at some point.  And, oh yes, I said "clientS" because of course we had many different clients that you could use for IRC from all different platforms.    We had our bots and our "/me". Clients had nick completion and a ton of other features.  We were IRC users and we were vastly superior.

But over the last five years or so I noticed that more and more of the folks with whom I had been communicating on IRC... stopped... using... it.

Myself included.  The last time I seriously used IRC was probably 4 or 5 years ago.

I would put a large part of the blame on corporate firewalls.  Somewhere along the way IRC got the stamp of being a waste of time and a productivity drain (which, like any technology, can be true if mis-used).  It was way too easy to simply block port 6667 on the corporate firewall (and/or the IRC protocol).  As botnets proliferated and used IRC as a control channel, there became a security reason to block the protocol as well.

Many IRC users continued, of course, but to do so from behind a corporate firewall usually meant creating a VPN or ssh tunnel to an external server and running the IRC client there.  Easy enough to do (I did it myself for a while), but not quite as easy as all those consumer IM products that just sat down in your Windows systray and gave you a toast message when someone was contacting you.  Plus, while it was easy for the tech-savvy of us to ssh or VPN out to an external server, many of our less-tech-literate colleagues didn't know how to do that.  So they didn't - but they were the ones with whom we often wanted to communicate.

So over time, we gave in.... and fired up AIM and ICQ and MSN/WLM and Jabber and Skype and.... 

Skype, especially, seemed to have caught on for group chats.  In part perhaps because of the ability to create "public group chats" that were persistent (i.e. they survived logout/login and in fact you basically stay in them forever until you click "Leave").  I've often thought, though, that part of it was also that Skype groupchat is the closest that I've seen to replicating what is in IRC.  It is basically "IRC with a prettier face".  It has "/me" (admittedly a favorite of mine) and many of the other "/" commands.  For IRC users, it is a very easy and seductive change.

But now, with the continued Skype outage, those of us who have come to rely on Skype groupchats as a component of our daily communication are suddenly left without an easy vehicle for the group communication to which we are accustomed.

Jabber, of course, is one option.  Like IRC, it's all about open standards, there are many Jabber servers and a ton of Jabber clients.  But I personally never saw it take off for larger groupchats to the degree that Skype did (or IRC).

So in the end this morning, I dusted off an IRC client I had installed (Miranda) and connected in to good old freenode, where some people with whom I communicate indicated they would be, talking, predictably, about the Skype outage (in "#skype"). 

Will I return to using the Skype groupchats when Skype comes back up?  Probably.  It's way too simple and easy.  Plus, part of what I do is analyze Skype and you can't really do that without participating in it.  But for now... for this moment anyway... I'll return to an old friend.

/me stops reminiscing and returns to work

UPDATE: For those not familiar with IRC, you may want to visit IRC.org and particularly the history page.