An outage occurred on December 16th 3:35 PM NZST. End users may have been able to load the application, but pages would have been stuck loading or in a non-responsive state.
We released some software updates a few minutes prior, and unfortunately this allowed for data to be queried from our database in-optimally. This resulted in CPU resources being exhausted on our database cluster, which affected all services ability to save and load data on Hero.
After identifying the root cause of the issue, we rolled back the software update and the platform returned to full functionality at 3:58 PM NZST.
We aim to release large software updates outside of high-use hours, but we apologise if you were affected during the 23 minutes that Hero was unavailable.
We will be looking at ways to ensure we can detect these kinds of issues in our software testing practices going forwards.