Major Outage
Incident Report for Hero
Postmortem

An outage occurred on December 16th 3:35 PM NZST. End users may have been able to load the application, but pages would have been stuck loading or in a non-responsive state.

We released some software updates a few minutes prior, and unfortunately this allowed for data to be queried from our database in-optimally. This resulted in CPU resources being exhausted on our database cluster, which affected all services ability to save and load data on Hero.

After identifying the root cause of the issue, we rolled back the software update and the platform returned to full functionality at 3:58 PM NZST.

We aim to release large software updates outside of high-use hours, but we apologise if you were affected during the 23 minutes that Hero was unavailable.

We will be looking at ways to ensure we can detect these kinds of issues in our software testing practices going forwards.

Posted Dec 16, 2022 - 08:45 NZDT

Resolved
This incident has been resolved.
Posted Dec 15, 2022 - 16:09 NZDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 15, 2022 - 16:03 NZDT
Identified
We're experiencing a major outage and are actively investigating to isolate the problem.
Posted Dec 15, 2022 - 15:40 NZDT
This incident affected: Hero (Hero App, Hero APIs).