Disruptive Telephony

Well, the official word is out from Skype and it can be summarized: the reboots from Microsoft patches triggered a previously-undetected condition and crashed out network. 

Skype PR staffer Villu Arak writes in "What happened on August 16":

On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Okay... I can buy that this type of thing could trigger some kind of chain reaction, but I don't understand why this month was different than any other month.  For.. what? two or three years now (more?) Microsoft patches have been coming out like clockwork on the second Tuesday of each month.  Each second Tuesday or Wednesday, the millions of computers set to auto-update do so.  All those zillions of computers restart automatically.  Each and every month.  What was so special about this August that was different from every other month?  Was the number or restarts in a short period of time really that much different from other months? Why? Is the issue that there are so many more Windows Skype users than in previous months and years? Was this just the so-called "tipping point" when there were enough Windows Skype users that the normal restarts triggered this chain reaction?

The issue has now been identified explicitly within Skype. We can confirm categorically that no malicious activities were attributed or that our users’ security was not, at any point, at risk.

In other words, it was not a DDoS by Russian hackers, as one rumor had it (which had actually already been dismissed by every security researcher who looked at the alleged exploit code).

This disruption was unprecedented in terms of its impact and scope. We would like to point out that very few technologies or communications networks today are guaranteed to operate without interruptions.

Fair enough statement - if you are looking at data or web technologies... but the PSTN, to which Skype would seem to like to be compared, is designed to operate without interruptions (or with as minimal as possible).  You know, there is this wee little market for "carrier-grade" equipment/software/etc. that is designed to be highly available without downtime.  If a carrier's network were down for over 48 hours, there would be a zillion lawsuits, intense government inquiries and more.  The carriers that make up what we call the "PSTN" put an incredible effort into ensuring availability.  If Skype wants to play in that game, they have to be ready to play at the same level.

Skype has now identified and already introduced a number of improvements to its software to ensure that our users will not be similarly affected in the unlikely possibility of this combination of events recurring.

Good. We would expect that.

I appreciate that Skype has been as communicative as they have through their blog and heartbeat site.  Thank you, Skype, for communicating - and leaving the comments open.  However, to me the information provided today is still lacking one key piece:

Why were the mass restarts associated with the August 2007 Microsoft updates different from the mass restarts associated with any other month's Microsoft updates?

According to Skype's "Update at midnight GMT", the Skype outage should be resolving over the next few hours:

We are pleased to announce that the situation continues to improve. The sign-on problems have been resolved. Skype presence and chat may still take a few more hours to be fully operational. We know what our faithful users have been going through and we thank you for your patience and kind support.

Skype has come back online for me and the Skype client tells me there are now over 4 million users online, which is the first time I've seen it show that since Wednesday.  Given that it's been going up and down all day, I'll withhold judgement for a while, but that number does seem to be climbing (if we can believe the number, which is an open question).  My contact list is slowly repopulating with its normal list of people.  We'll see.

So the question really is - will Skype see any long-term impact because of this outage?

Certainly in the short-term Skype will have an awful lot of explaining to do.  There are certainly some number of business users who have come to rely on SkypeIn and SkypeOut.  I know a good number of freelance consultants who have cut their landline and give out their SkypeIn number to everyone and use Skype or SkypeOut for their calls.  Countless small startups are "virtual organizations" where Skype is used for all the communication. Skype will definitely need to reassure them that this won't happen again.    I expect that many of them will be looking now at what alternatives they have, if for no other reason than to have a backup.

Skype's competitors are definitely circling like sharks that smell blood in the water.  Sightspeed's CEO came out with a very pointed blog entry today, "All Peer-to-Peer Models are NOT Created Equal", about how their p2p network was better.  Supporters of Damaka seem to be running around leaving glowing praise and inviting people to try Damaka on various blog entries related to the Skype outage (including my own).  There have been any number of blog entries and comments out there extolling the virtues of the Gizmo Project.  Even Jeff Pulver couldn't resist and tossed in a tweet encouraging people who were missing Skype to try out FWD.

Being the open standards geek that I am, I would love to believe that Skype users would try out many of the other services, many or most of which are based on open standards like SIP.  I'd love to see the level of interoperability you have can have with things like (SIP-based) Gizmo.  Skype's walled garden approach does concern me.  I'd love to see massive adoption of some service to which other services could easily interoperate - and which could have peer-reviewed protocols that were definitely secure.  I'd love to see all of that - and that's certainly a potential outcome.

But I'm also a bit too jaded to think it will really happen.  The seductive aspect of Skype is that it makes it very simple and easy to use the product.  It has (at least for the moment) a massive directory of users.  It has APIs that let it be readily integrated with other web sites and services.  It's easy to use... and I expect that probably many if not most users will simply go back to using it as they have always done once it comes back into full operation.  People have short-term memories... 

If there are any beneficiaries of this outage, I would think that they might be AOL, Microsoft, Google or Yahoo!, all of whom have similar consumer services and have voice as an option now.  Skype's certainly been perceived as the leader with regard to voice... but all of those others are there... and they all have simple interfaces and massive directories as well.

What do you think?  Will Skype suffer any major impact because of this?  Will the newer services like Gizmo, Sightspeed and friends win customers?  What about the other consumer IM services? 

As of this moment, Skype is back working for me and tells me that 3,181,959 people are online.  However, given the performance this morning, I am not expecting it to stay up.  It's been fading in and out all day.

The last we heard from Skype was about six hours ago in their "Where we are at 1100 GMT" post. The comments (currently 81) to that post are quite fascinating to read.  Some are the typical kind of outrage you expect.  Some are passing along the latest speculation. Some are giving reports of continued outage.  Some include links to news articles.  Some bash Skype.  Some praise Skype.  As is typical, some bash other commenters (like this one apparently from a Skype-using-solider in Iraq).  Some plead for a return.  Some suggest alternatives

In the midst of that (and the other entries with comments: here(183), here(54) and here (64)), you see the comments dealing with the human side:

I miss my friends!!

Thanks for the information, I miss my parents in Mexico, they have the same problem as well, hope today the system comes back

come on guys i need to make an important phone calls plz fix it as soon as possibles

I've not seen any changes in service. Pretty fed up because this is important to bme doing business with my clients.

Now I know not to take a conversation with my boyfriend, who currently is living in Denmark, for granted. Irritating indeed

Hey everyone in the world using SKYPE...wow didn't realize how much we all rely on SKYPE. We use SKYPE for our business. That is how we connect to the world. We connect to all of our contacts using SKYPE. We miss it terribly. All I see is a grey X...need to see the green CK MARK! We basically used our cell service and emails it worked..but made it hard to teleconference...tomorrow is another day Skype..hope we are connected.

I had to plan a flight today with some one in the U.K. this was not very easy sending text messages back and forth on our mobile phones. This problem needs to be fixed soon

Please, please try to resolve this system glitch soon. My business relies so much on Skype voice calls. We are losing business connections every hour if skype isn't availabe.

Hurry it up! This is bad for business!!

You get the idea.  (Kudos to Skype, by the way, for leaving their blog comments open during this situation.)  In another forum where someone was venting, the person somewhat frantically wrote this:

yes but I put my skypein phone # on my resume!!!!

Ouch.

I, too, am impacted to a degree.  Although I am more of a casual user of Skype, i.e. I don't rely on Skype for communication, the number I put in the sidebar to this blog is my SkypeIn number.  Why?  Just because I didn't want to put my home or cell number and figured that it might be a good way to test SkypeIn.  (And in the 7 months I've had it up there I've probably had maybe 2 calls on it resulting from the blog!)  But still, this does cause me to rethink that.  (In fact, I may very shortly change the sidebar number to my GrandCentral number as a layer of redirection.)

The outage looks like it will continue for a while. (I am disconnected again in the time it took me to write this.)  And in the meantime there are real people out there suffering because they have come to depend upon a particular VoIP provider. 

Yes, I work for a company that on one level could be seen as competing with Skype when it comes to business (but we don't run into Skype, really), but an outage of this length isn't good for us as an industry.  Already there are voices out there saying that this shows inherent weaknesses in P2P VoIP (it doesn't, in my opinion).  Perhaps the good news is that people are looking around at alternatives and they are asking these questions now about availability. 

Julian Bond said it perhaps best in an IRC chat room today:

This debacle is proving to be a bit of a shocker. I guess we all got lulled into thinking that Skype was at least as reliable as the Cell networks.

Yes, we did.  Some much more than others (especially those who dropped their landlines and used it for business).

In another post when this is all over, I'll write a bit more from the security side about "lessons learned", but for the moment I think we need to remember that there are real people right now being impacted by all of this.  For better or worse, Skype has become a communication that is (or has been) relied upon by many.  Hopefully the folks in Estonia and other locations can get this fixed soon.

In the beginning, there was IRC. 

Well, okay, not exactly... BITNET Relay was around before that and there were other multi-chat environments in some of the walled garden services (CompuServe, GENIE, etc.) and BBSs... but for most of us who were online from the late 1980s onward, IRC was the place to be for "chat" and realtime IM communication.  Of course, it lived primarily in the geekier side of the Internet.  The "real" Net users used IRC and looked down upon all the "newbies" who were drawn to these new IM services from ICQ, AOL and later MSN and a zillion others.  Sure, they were pretty and had cute emoticons.  Yeah, okay, so they could include videos and knew when other people were typing and had little "toast" popups... all that would just be added to IRC clients at some point.  And, oh yes, I said "clientS" because of course we had many different clients that you could use for IRC from all different platforms.    We had our bots and our "/me". Clients had nick completion and a ton of other features.  We were IRC users and we were vastly superior.

But over the last five years or so I noticed that more and more of the folks with whom I had been communicating on IRC... stopped... using... it.

Myself included.  The last time I seriously used IRC was probably 4 or 5 years ago.

I would put a large part of the blame on corporate firewalls.  Somewhere along the way IRC got the stamp of being a waste of time and a productivity drain (which, like any technology, can be true if mis-used).  It was way too easy to simply block port 6667 on the corporate firewall (and/or the IRC protocol).  As botnets proliferated and used IRC as a control channel, there became a security reason to block the protocol as well.

Many IRC users continued, of course, but to do so from behind a corporate firewall usually meant creating a VPN or ssh tunnel to an external server and running the IRC client there.  Easy enough to do (I did it myself for a while), but not quite as easy as all those consumer IM products that just sat down in your Windows systray and gave you a toast message when someone was contacting you.  Plus, while it was easy for the tech-savvy of us to ssh or VPN out to an external server, many of our less-tech-literate colleagues didn't know how to do that.  So they didn't - but they were the ones with whom we often wanted to communicate.

So over time, we gave in.... and fired up AIM and ICQ and MSN/WLM and Jabber and Skype and.... 

Skype, especially, seemed to have caught on for group chats.  In part perhaps because of the ability to create "public group chats" that were persistent (i.e. they survived logout/login and in fact you basically stay in them forever until you click "Leave").  I've often thought, though, that part of it was also that Skype groupchat is the closest that I've seen to replicating what is in IRC.  It is basically "IRC with a prettier face".  It has "/me" (admittedly a favorite of mine) and many of the other "/" commands.  For IRC users, it is a very easy and seductive change.

But now, with the continued Skype outage, those of us who have come to rely on Skype groupchats as a component of our daily communication are suddenly left without an easy vehicle for the group communication to which we are accustomed.

Jabber, of course, is one option.  Like IRC, it's all about open standards, there are many Jabber servers and a ton of Jabber clients.  But I personally never saw it take off for larger groupchats to the degree that Skype did (or IRC).

So in the end this morning, I dusted off an IRC client I had installed (Miranda) and connected in to good old freenode, where some people with whom I communicate indicated they would be, talking, predictably, about the Skype outage (in "#skype"). 

Will I return to using the Skype groupchats when Skype comes back up?  Probably.  It's way too simple and easy.  Plus, part of what I do is analyze Skype and you can't really do that without participating in it.  But for now... for this moment anyway... I'll return to an old friend.

/me stops reminiscing and returns to work

UPDATE: For those not familiar with IRC, you may want to visit IRC.org and particularly the history page.

There's one little feature in my Blackberry 8830 that I just discovered today that I didn't see anywhere in any of the promotional materials about it.  Put simply:

Unified messaging works!

Here's the thing... given that I work for Mitel, I of course have "unified messaging" set up so that whenever someone leaves me a voicemail message, I get an email with the WAV file attached to it.  It's truly a wonderful thing because I never dial in to check to see if I have messages.  I get an email that clues me in to that fact - and generally when I am on my desktop PC, I just play the attached WAV file and listen to the message through my email program.  I don't dial into voicemail to listen.

Before, with my Blackberry 7290 or any of the other earlier models, having this unified messaging feature enabled was really only a "half solution".  Receiving the email clued you in to the fact that you now had a voicemail message... BUT...

you couldn't listen to the WAV file!

The Blackberry operating system couldn't play the WAV file, so you had to dial into the voicemail system to listen to the actual message.  So the nice part about UM was that you got the notification... but you couldn't hear it right there.

Well today a blog post by Russell Shaw over on ZDNet pointed me to this excellent 8830 FAQ over on EVDOInfo.com that contains this little gem:

Q - What are the enterprise advantages of the new 4.2 OS?
A - The biggest 4.2 feature is the ability to listen to WAV files that are attached to an email by a Unified Voice Messaging service. NOTE: The customers BES server must be version 4.1.2 or later.

Naturally I had to try it out so I called my Mitel extension and left a message (a virtue of having multiple phones around). A moment or so after I hung up there was an email msg with a WAV attachment sitting in my Blackberry inbox.  I opened it, scrolled down to the attachment area and chose "Open attachment" and... ta da!  There I was listening to the voicemail message!  (And since the 8830 has a speakerphone I could listen that way which would enable me to easily write things down were it a real message.)  Nice and easy.

Of course, I would be charged for the download of that WAV file but I'm on an "Unlimited Data" plan with Verizon which does, in fact, appear to be for an unlimited amount of data (no fine print that I could see).

So to me this is a wonderful addition to the Blackberry operating system... now I can receive my voicemail messages in my email and listen to them right on my phone.  Very cool!

Very good news here in Mitel-land today - at about 4pm the merger of Mitel and Inter-Tel was completed! At least, in the legal sense... obviously there will be more work to be done on the integration, but for now the celebrations can begin.  The company has now doubled in size, gained a very strong US organization and is all-around well-positioned for growth. As the news release stated:

The company will now be #1 in the North American SMB market¹, #2 in the Western European IP PBX market², the overall leader in the U.K. communications market³ and continues to grow its operations globally. With three trusted brands (Mitel, Inter-Tel and Lake), the company offers customers a broad choice of solutions from the very small to the very large, from IP enabled to pure IP unified communications, from standard solutions to tailored, from single site to multi-site and from outright capital purchase options through sophisticated managed services. 

It should be a fun ride to see where this all winds up!

Somewhat predictably, the blogosphere is going nuts about the Skype outage.   It's currently at the top of Techmeme.  As Tris Hussey notes in his post, Skype has indeed become a key communication tool for many individuals and businesses.  Other VoIP bloggers offering commentary:

• Om Malik: "Does Skype Outage Expose P2P's Limitations?"
• Tom Keating: "Skype Outage" - wonders if it had to do with the Microsoft updates (I would think not because the outage happened to me prior to me installing the updates)
• Andy Abramson: "Skype Suffers Major Outage" - helpfully points people to Gizmo and SightSpeed as alternatives

The Register, of course, chimed in with their normal attitude.  Ars Technica, meanwhile, commented that the outage was surprising mostly because of ""how often it doesn't happen".  Taking another view, Mashable is reporting that eBay's stock has dipped this morning.

Meanwhile, Skype users around the world consider what other IM programs to use and some of them will, maybe, just maybe, have to actually pick up a phone today!

As a Skype user, I was a bit surprised when, after restarting my computer today (due to a crash when I was trying SecondLife with voice, but that's another story), the Skype icon just kept on spinning and spinning saying "Connecting".  And then it would be online... and then it wasn't... and then it was... and then it wasn't....

My first sign that it was something larger than my (sometimes flaky) laptop was a Facebook status update by Jim Courtney saying that Skype might have an outage for 12-24 hours.  About the same time I saw a tweet from Chris Brogan.  The "social networking information sharing network" was hard at work...

A trip to Skype's "Heartbeat" page showed the problem:

Oops.

Nicely, though, Skype staff have posted a status update on their heartbeat page:

UPDATED 14:02 GMT: Some of you may be having problems logging into Skype. Our engineering team has determined that it’s a software issue. We expect this to be resolved within 12 to 24 hours. Meanwhile, you can simply leave your Skype client running and as soon as the issue is resolved, you will be logged in. We apologize for the inconvenience.

So sometime within the next while it will be back up.  It will be curious to see if anyone at Skype will provide technical explanations of what happened.  I note from another status update that they performed some planned maintenance last night.  Perhaps something did not work quite right?

In any event, I would think Skype will need to offer some post-resolution reassurance to all those businesses they are so eagerly courting...

After already publishing my last post about mashups, I came across Jim Courtney's Skype Journal post about the Skype mashup competition, which in turn led me to Thomas Howe's excellent "API of the week" post (got all that?) which had these wonderfully-written words (so much so that I feel compelled to excerpt them here, something I seldom do to this extent):

If you were to ask me, I would say the twenty year old software engineer has a distinct advantage over the older telephone guys (such as me) in the realm of innovation.  Since the barriers to entry to deploying a service provider have fallen through the floor, the larger challenge is not in complex engineering, but is instead in innovation.   The younger engineers are free of the legacy of the PSTN, and many things would occur to an experienced engineer won't to them, and it's not a bad thing. 

<snip>

What does this have to do with telephony? Nothing. What does this have to do with next generation applications? Everything.  Applications that use the Internet as the platform use APIs from a large number of sources, and by and large, these APIs are not telephony. However, nearly every time a telephony API is used, an API such as GoogleMaps, Amazon SQS or DBPedia will be used right alongside it.  As a developer in this market, it makes a lot of sense for you to get to know your neighbors for two reasons. First, the more you can make your API play well with others, the faster the adoption of it will be. Secondly, the more you can understand your customers, their problems and how they need your part for their solution, the better you can make your API for them.   I'm supposing this means that you need to get familiar with APIs like this.

Which leads me back to my original statement.  The twenty-something-don't-know-or-care-about-SS7 engineer will sit down and design their version of the hot-or-not site one day, and use a whole bunch of crazy APIs to put together the application.  Then, they will go have a beer, come back, and say "You know, it would be really cool if you could just call the person you want to hook up with.  Is there an API for that?"  They won't even consider for a minute the words "termination", "LATA" or "CALEA".  They're just writing an application.  They need an API for some function, and it will take a few minutes to integrate it into their application.  And, there are many, many more of these guys than all the telecom engineers that have ever, and will ever, exist.

<snip>

Well, said, Thomas! (Read his original post for the full text.) And to copy a former US President:  "It's the API's, stupid!"

It's very cool to see that one of the leading web sites about "mashups", ProgrammableWeb.com, has announced the launch of a new section focused on "Mobile/Telephony" mashups.  They actually call it a "Market", and there are two others launched today: Mapping and Shopping. 

The obvious focus for this blog, though, is the "Mobile/Telephony" market. The announcement notes that there are currently over 25 different APIs under "Telephony" or "Messaging" and in the actual lists of mashups, 104 are tagged "mobile" and 113 tagged "messaging".  The announcement also notes that Thomas Howe will be assisting with the content of this new section. (Congrats to Thomas!)

The list of telephony APIs (only 11, the other 14+ must be "messaging) is available and several names are probably quite familiar.  Thomas Howe also wrote a piece to explain the different types of APIs and provide a bit of background:  "Telephony & Mobile APIs and Mashups, the Big Picture".

I did find it a bit puzzling that the list of telephony APIs didn't include Skype, given that Skype has a whole developer website set up to support its APIs.  Likewise no mention of Asterisk even though the entire thing is really one big set of APIs.  But hey, the section is only a day old, so we'll have to give them a bit to see how it develops.

The fun thing about the state of VoIP/IP telephony right now is that it really is becoming all about APIs and mashups...  voice is truly just another form of data on the network... and once it is just a bunch of packets you can do really fun stuff with it!  Kudos to the ProgrammeableWeb team and Thomas Howe for helping point the way to some of the interesting stuff people are doing in this area.

Definitely do check out their telephony mashup site.