Calls not connecting
Incident Report for Boostlingo
Postmortem

At 5:59 a.m. PT on 6/19/2024, our internal monitoring alerted technical teams of errors that could cause system degradation. Our team immediately jumped in to identify the issue and work towards a resolution.

At 6:05 a.m. PT, our support and account management teams began to receive notifications of long hold times and call failures. These were surfaced to our technical team for help with investigation and mitigation.

Our investigation focused on both identifying the issue and identifying if the delivery assurance plan (DAP) would be effective in mitigating the issue. After thorough investigation and review, our development team initiated our delivery assurance plan at 7:17 a.m. PT.

The delivery assurance plan was in place until 7:29 a.m. PT when our development team implemented a fix that resolved the underlying root cause of the issue, and calls were able to return to normal.

The root cause was an issue with a database table responsible for call routing. There was a counter within the table that reached its maximum value, which prevented records from being added. Our team took action to ensure data retention and performed a reset on the counter, which allowed new records to be added.

This fix will allow calls to flow indefinitely. The Development Team will also add additional development work to strengthen our monitors and technical infrastructure.

If you would like additional information, please get in touch with your account manager directly or our support team at support@boostlingo.com. We will have a more detailed root cause analysis available in the next two weeks, which we can share directly

Posted Jun 21, 2024 - 10:20 PDT

Resolved
This incident has been resolved. We will continue our full investigation into the root cause and follow-up with a debrief in the next 48 hours.
Posted Jun 19, 2024 - 07:48 PDT
Monitoring
A fix has been applied and we are monitoring the results.
Posted Jun 19, 2024 - 07:29 PDT
Investigating
We are currently investigating issues of calls not connecting.
Posted Jun 19, 2024 - 05:59 PDT
This incident affected: Boostlingo Video (Boostlingo Group Rooms, Boostlingo Communication REST API).