That's bad.
However, the unqualified apology by DBS' CEO, Mr Piyush Gupta, was truly a breath of fresh air. Not only was it devoid of the typical corporate hedging, but I believe it was the most frank and technically detailed account of a failure that I have ever read in the press.Mr Gupta's response has actually increased my confidence and loyalty in the bank. Which is a rather unusual outcome for a screw-up.
NB: .. to the extent that I actually thought it rather mean-spirited and self-serving for the Monetary Authority of Singapore (MAS) to step forward and emphasize the fact that it was actually the MAS who ".. instructed DBS to give a full account of the incident to the public, including the actions it will take to prevent future recurrence".
I can't help but compare and contrast this to Intuit's woes of late. On 15th June, Intuit experienced catastrophic failures of their Intuit.com, Quicken, QuickBooks, and TurboTax services. Services were only restored more than a day later, and it wasn't until 7pm on the 16th of June that customers had any kind of detailed explanation.
In both cases, the organisations concerned were caught out in the short-term. Customers were affected and actively seeking answers within minutes. But the corporate responses can be measured in hours and days.
It is also worrisome that in both cases, the reported investigations point to failures in standard maintenance procedures: ".. during a routine maintenance procedure.." (Intuit) and ".. IT vendor IBM [..] had used an 'outdated procedure' to carry out the repair.." (DBS).
SaaS providers should take note! Better to learn the lessons from others rather than make our own mistakes. Two take-aways:
What is your realistic response time to a service issue? Can you identify a problem and open a communication channel with customers within minutes or the hour? And does this only really work during normal business hours, or is it 24x7? Prompt and open communication can avert a tsunami of customer complaints in a serious situation. Do you have that insurance in place?
Do you really have production change management under control? My natural assumption is that neither DBS nor Intuit have (a) the staging systems available, or (b) the procedures in place, to exactly and incontrovertibly test production changes. The nett of my years in enterprise computing: "never, ever, make a change in production that you haven't first tested and practised elsewhere". What is your situation, honestly?
Of course, reality bites. Notwithstanding any and all preventative measures you put in place, there's a good chance that one day you will find yourself in a situation as dire as DBS or Intuit (let's not even mention BP, which is a whole different ballpark).
It may not be easy, but the DBS case in particular demonstrates that no matter how bad things are in the short-term, full and frank disclosure - and the balls to take unequivocal responsibility - are essential to getting you back on track. Transparency is not a gamble.Blogarhythm: I think I must have written this post purely for the excuse of linking to Tell me, Tell me by S#arp, who are responsible for saving my sanity during a singular .com project back in the day.