OPI Call failures and Silence on opi calls across all servers

Incident Report for Boostlingo

Postmortem

At 8:00 a.m. PST, our team began to see an increase in error messages related to inbound silence across inbound and outbound OPI calls placed over IVR and from the web and mobile applications. Around that same time, we started receiving user reports in response to these error messages.

We quickly identified that this was caused by an incident with our PSTN partner, in which an internal network issue significantly degraded call quality or prevented successful connections. Their incident report can be viewed on their status page here: https://status.twilio.com/incidents/5b8wmb5djj4g

While our incident remained open until 10:35 a.m. PDT, we began seeing increases in connection rates and decreases in error messages much earlier, with our PSTN partners taking action to resolve their network issues. By 8:36 a.m. PDT, we had partial recovery and saw full recovery around 9:20 a.m.

Our team chose not to deploy our Delivery Assurance Plan because it relies on the impacted aspects of the PSTN Partner's platform. This approach ensured the highest number of calls were serviced.

Our Development and Engineering Teams are investigating preventive measures to mitigate and improve future responses to such issues. We are also exploring an iteration of our delivery assurance plan that would rely on less overlapping technology with our primary platform.

We are in contact with our PSTN Partner to ensure they are taking the necessary steps to reduce the risk of a similar incident in the future.

Posted Apr 24, 2024 - 09:24 PDT

Resolved

We've continued to experience stability as we test and monitor call functionality across the platform. Our team will continue to monitor partner updates across their status page here: https://status.twilio.com/incidents/5b8wmb5djj4g and work directly with them to ensure maximum platform stability and continuity.

We are closing this incident as performance and connections have returned to normal and remained stable over the last hour.

We will share a retrospective of this incident on our Status Page in the next 48 hours.

Posted Apr 22, 2024 - 10:35 PDT

Monitoring

Our PSTN Provider has taken steps to mitigate this issue. Here is their most recent update from their status page https://status.twilio.com/incidents/5b8wmb5djj4g

Currently, our baseline call functionality has been restored and is operating normally. Users should experience no issues with standard voice calls. We are also fully recovered on both inbound and outbound Flex calls. Our team is working diligently to resolve any residual issues to minimize any disruption to our users. We will provide another update in 30 minutes or as soon as more information becomes available.

We have seen this in our testing and logging as well and will continue to monitor the situation. We will follow up in an hour or earlier if there are any major updates. At this time, calls should work as expected across all platforms. If you experience any issues, please hang up and try the call again after waiting 30 seconds. Thank you and sorry again for the disruption.

Posted Apr 22, 2024 - 09:33 PDT

Update

We are in conversation with our PSTN Partner. They are actively working towards a fix. This is their most recent update about the work that they are doing:

[Twilio continues] investigating a service interruption affecting Twilio Voice services, impacting both inbound and outbound calls, including calls with Flex Services. Customers may experience degraded audio quality or difficulties connecting new calls. Our engineering team is actively working on identifying and resolving the cause. We aim to provide another update within 1 hour or as soon as more information becomes available.

In our testing, we are seeing intermittent success with OPI calls placed through IVR, the Web, and Mobile platforms. While all calls are still not connecting, we have seen a decrease in errors and an increase in success rates. VRI services remain unimpacted.

We will continue to test, monitor, and provide updates here every 30 minutes or less as we have relevant information to share.

Posted Apr 22, 2024 - 09:07 PDT

Identified

We have confirmed that this is related to the incident reported by our PSTN Partner: https://status.twilio.com/incidents/5b8wmb5djj4g

We have seen an increase in successful calls over the last 15 minutes as our team continues to test and take action alongside our PSTN partner. We will post updates here at least every 30 minutes with any relevant updates.

Posted Apr 22, 2024 - 08:36 PDT

Update

These issues are related to an active incident reported by our PSTN carrier through their status page here: https://status.twilio.com/incidents/5b8wmb5djj4g

[Twilio's] monitoring systems have detected a potential issue with inbound and outbound calls. Our engineering team has been alerted and is actively investigating. We will update as soon as we have more information.

Right now, for languages where VRI is available, we recommend requesting VRI if possible. We will monitor this situation and provide an update once we have one. Thank you.

Posted Apr 22, 2024 - 08:14 PDT

Investigating

We are investigating reports of silence once inbound OPI calls are connected to an interpreter or operator. If you connect to a call and experience silence, we recommend waiting 60 seconds and attempting your call again.

We will follow up with additional information here as soon as we have it. Please subscribe to ensure you receive updates as soon as they are available.

Posted Apr 22, 2024 - 08:04 PDT

This incident affected: Boostlingo Voice (Boostllingo Voice IVR, Boostlingo Network Traversal Service, Boostlingo Speech Recognition), Boostlingo Portal (Boostlingo Interpreter Portal, Boostlingo Requestor Portal), and Twilio Voice, North America.