The backlog has been fully processed and errors and latencies have returned to normal levels. The issue has been resolved and the service is operating normally.

The vast majority of operations continue to operate normally. Our storage partner has implemented two parallel mitigations to improve the speed of processing the backlog of any queued requests.

Our storage partner continues to see a gradual improvement in error rates as the backlog is processed. They are employing mitigations to speed up the processing of the backlog during this event. Once the backlog is resolved, we expect that the error rate will fully recover. The vast majority of operations continue to operate normally.

Our storage provider reports some improvement in the error rates and latencies. A recent update caused increased load within the affected subsystem, which led to increased error rates and latencies. This increased load has now been mitigated and are seeing early signs indicate that the service is recovering. As the sub-system processes the backlog of requests, error rates and latencies will continue to improve.

Our storage partner reports that they have narrowed down the root cause to a specific sub-system and continue to make progress in mitigating the impact to this service but have not yet seen significant improvement. Error rates and latencies have stayed a consistent low level with the vast majority of request being processed within the normal latency levels.

Our storage partner reports that they have identified the subsystem responsible for increased error rates, and are working to isolate the root cause within this subsystem.