Delayed Notifications

Incident Report for CloudWisdom

Postmortem

One of our notification service containers stopped processing notifications. This caused 25% of outgoing notifications to be delayed. Our notification processing alert threshold was set too high, which is why the container was not promptly replaced. We have already greatly lowered this threshold so we are notified about issues with our notification service in a more timely manner.

Posted Nov 13, 2019 - 10:22 EST

Resolved

All notifications have caught up. We will post a summary of the issue on this incident within 24 hours.

Posted Nov 12, 2019 - 11:47 EST

Monitoring

A fix has been implemented and we're monitoring the results.

Posted Nov 12, 2019 - 11:42 EST

Identified

We've identified a delay in some notifications. We are currently resolving and expect notifications to catch up quickly.

Posted Nov 12, 2019 - 11:30 EST

This incident affected: Alerting.

Subscribe to Incident