‘This is not supposed to happen’: Experts on DBS, Citi outage caused by data centre failure

BACKUPS, RECOVERY PLANS NEEDED

The importance of the data centre means that banks always have a backup data centre. Some banks also have two data centres that share the workload concurrently. In such cases, if one fails, the other can pick up the slack.

Dr Dennis Khoo, a managing partner at digital consultancy allDigitalFuture, said: “Generally for mission-critical applications like banks, there are multiple levels of redundancy. 

“In most advanced banks, using the latest technologies, the database will be instantly replicated, that means they’ll have a primary site and alternate site and the data is replicated instantly on both sites,” said Dr Khoo. 

The Singapore Computer Society said most data centres are designed and built with a certain level of redundancy and the ability to conduct maintenance in real time. They are also specifically built to meet the exact redundancy requirements of the business, it said.

A common uptime guarantee of a data centre would usually be at 99.982 per cent to mitigate possible disruption to its customers, the society added. 

However, there is still 0.018 per cent of possible downtime. “Thus, the client must establish an efficient Business Continuity Management System and IT Disaster Recovery Plan to allow their critical IT systems and data information to immediately failover to the secondary data centre in the shortest possible time should such an incident occur.”

WHAT CAN BANKS DO WHEN DATA CENTRES ARE DOWN 

That said, should all data centres fail, there is little – if any – service a bank can provide. 

Dr Thng said banks activate what they call “offline mode”, which means they render some services at branch offices using what is available. These transactions are then updated with the mainframe when the data centre is back up. 

These services may include cash withdrawals – as banks have spare cash on hand – cash deposits, payment instructions and credit card transactions. In the case of DBS, it reopened branches to help customers with some services. 

According to Dr Thng, IT incidents, such as a delay in processing, occur daily. Some of these go unnoticed by customers. Still, service outages will negatively impact the bank’s reputation and may have some financial repercussions, for example, if a client incurs late fees from being unable to pay his bill on time due to an outage. 

On reputational impact, Dr Khoo said banks would have “broken” their service commitments to provide around-the-clock service to customers.

“So definitely, in that sense, there will be some reputational damage in terms of your ability to serve customers properly. And with proper design, this is not supposed to happen.”