Delayed Notifications
Incident Report for CloudWisdom
Postmortem

One of our notification service containers stopped processing notifications. This caused 25% of outgoing notifications to be delayed. Our notification processing alert threshold was set too high, which is why the container was not promptly replaced. We have already greatly lowered this threshold so we are notified about issues with our notification service in a more timely manner.

Posted Nov 13, 2019 - 10:22 EST

Resolved
All notifications have caught up. We will post a summary of the issue on this incident within 24 hours.
Posted Nov 12, 2019 - 11:47 EST
Monitoring
A fix has been implemented and we're monitoring the results.
Posted Nov 12, 2019 - 11:42 EST
Identified
We've identified a delay in some notifications. We are currently resolving and expect notifications to catch up quickly.
Posted Nov 12, 2019 - 11:30 EST
This incident affected: Alerting.