Yesterday on 9/23 at 10:45AM Eastern we began having issues with our search database. Due to increased load the database began to return slow results and then return no results at all. This resulted in a UI and query API outage. The database was back online briefly at 10:51AM Eastern, but began returning errors again at 10:53AM. At 11:10AM Eastern the database fully recovered and stayed online.
During the UI and query API outage data processing was delayed as parts of our data processing pipeline requires the search database. Slight delays in some AWS data collection occurred until 12:04PM Eastern. At that point search was back online and AWS collection and data processing was fully recovered.
We're continuing to investigate the reasons why our search database became overloaded so we can better protect both against the quantity and complexity of the queries.