Massive internet outage traced to Cloudflare glitch

July 03, 2019 15:54
Cloudflare said the massive internet outage resulted from bad software deploy, and was not an attack. Photo: Cloudflare

Internet users in Hong Kong were not able to reach multiple websites late Tuesday night, and the outage was traced to a software problem being experienced by Cloudflare, which provides web content delivery network services worldwide.

The massive internet paralysis raised speculation that Cloudflare servers could have been attacked, but the company explained the problem "was caused by a bad software deploy that was rolled back".

There was no attack, Cloudflare stressed, but noted that the major outage had affected all of its services worldwide.

The outage started at around 9:50 p.m., when local internet users complained on social media that they were unable to access a number of online forums.

The problem lasted about half an hour, affecting popular sites such as Hong Kong Golden Forum, Hong Kong Discuss Forum and Stand News.

Those trying to reach online forums such as LIHKG were greeted with the notification "502 Bad Gateway Cloudflare" while the Cloudflare website itself showed "HTTP 502 Errors".

According to a blog post from Cloudflare, “for about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back.”

“This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred,” it added.

CTO of Cloudflare John Graham-Cumming, Cloudflare's chief technology officer, said the outage resulted from a massive spike in the central processing unit, which caused primary and secondary systems to fall over.

Cloudflare then shut down the service responsible for the massive spike and the service was restored in 30 minutes.

He said all services have resumed normal operations.

The outage came amid a tense political atmosphere in the city.

Massive protests against the government's now-suspended extradition bill have raised fears that mainland authorities could intervene to resolve the political crisis.

Last month, protesters using the messaging app Telegram to communicate with one another reported connectivity issues, which were later blamed on a distributed denial of service (DDoS) attack.

Telegram founder Pavel Durov said the “state actor-sized DDoS” attack was traced to IP addresses in China.

– Contact us at [email protected]