Cortext production incident
Incident Report for Imprivata
Postmortem

On 6/8/2021, starting around 12:00pm Eastern, Cortext server monitoring showed an increase in processing of queue and sessions. The increased load had a downstream impact on database processing which led to much longer reconnection time for the clients. This affected user sign ins, messaging and status changes in all Cortext clients. The team quickly added additional authentication nodes but that only helped for little while before the service degraded again. As the team continued to look through logs and metrics, one of the SQL queries related to Directories was discovered as running extremely slowly. It was determined that the SQL table was corrupted and was rebuilt, which cleared the problem around 6:45p Eastern. Additional monitoring of SQL logs to alert potential slow queries are enabled.

Posted Jun 10, 2021 - 08:02 EDT

Resolved
The issue with Cortext has been resolved and we thank you for your patience. If you continue to see issues with Cortext messaging or status updates, please log out of the client then log back in. A brief incident summary will be published here in 1-2 business days.
Posted Jun 08, 2021 - 19:05 EDT
Monitoring
Imprivata has completed work to help restore functionality to Cortext. Internal testing has proven successful though we continue to monitor on our side. Another update will be sent soon.
Posted Jun 08, 2021 - 18:54 EDT
Update
Imprivata is still working to restore Cortext functionality. Updates will be sent at least every 30 minutes.
Posted Jun 08, 2021 - 18:09 EDT
Update
Imprivata's work continues to resolve the Cortext sign in, messaging and status update issue. We will post another update in 30 minutes.
Posted Jun 08, 2021 - 17:39 EDT
Update
Imprivata is still working to bring Cortext back to full functionality. Sign ins, messaging and status updates continue to be affected. Another update will be sent in 30 minutes.
Posted Jun 08, 2021 - 17:10 EDT
Update
Imprivata is still working to restore Cortext functionality. Updates will continue to be sent at least every 30 minutes as progress is made.
Posted Jun 08, 2021 - 16:41 EDT
Update
Imprivata continues to work to restore functionality to all Cortext clients. Sign ins, messaging and status updates continue to be affected. We will provide another update in 30 minutes.
Posted Jun 08, 2021 - 16:09 EDT
Update
Imprivata is still working to resolve the issue affecting Cortext clients for sign in, messaging and status updates. Another update will be given in 30 minutes.
Posted Jun 08, 2021 - 15:39 EDT
Update
Imprivata's work on the back-end continues. This is being done to restore full functionality to Cortext sign in, messaging and status updates. We will provide another update in 30 minutes.
Posted Jun 08, 2021 - 15:09 EDT
Update
Imprivata is performing back-end work to resolve the ongoing Cortext incident. Sign in, messaging and status update continue to be affected. We will update you on progress in 30 minutes.
Posted Jun 08, 2021 - 14:38 EDT
Update
Imprivata is continuing to work to resolve the Cortext issue affecting sign ins, messaging and status updates. We will keep you updated here every 30 minutes as work progresses.
Posted Jun 08, 2021 - 14:08 EDT
Update
Imprivata is working to fully restore service to Cortext clients. Sign ins, messaging and status updates continue to be affected. Another update will be sent within 30 minutes.
Posted Jun 08, 2021 - 13:39 EDT
Update
Imprivata continues to work on the back end to restore service to all Cortext clients for sign ins, messaging and status updates. We will continue to update here at least every 30 minutes as progress is made.
Posted Jun 08, 2021 - 13:08 EDT
Identified
Imprivata has identified the issues affecting Cortext clients for sign ins, messaging and status updates. Work is continuing to resolve the issue. Another update will be posted within 30 minutes.
Posted Jun 08, 2021 - 12:39 EDT
Investigating
Imprivata is investigating an issue affecting Cortext clients for sign ins, messaging and status updates. Work is underway to remediate. Updates will be posted here at least every 30 minutes.
Posted Jun 08, 2021 - 12:10 EDT
This incident affected: Imprivata Cortext (Messaging, Third Party SMS).