Between 12:30 UTC and 15:00 UTC on July 23rd 2023 a third party managed websocket service used in Boostlingo went down. The third party determined that an automated OS kernel deployment introduced configuration issues.

The third party mitigated the issue by rolling back to the previous OS image version containing a healthy configuration.

Boostlingo utilizes this websocket service for one-to-many as well as one-to-one communication, namely from the Boost servers to web and mobile applications. The websocket service is an ideal fit with our on-demand calling functionality due to the fact that clients don't need to "poll" for data from an API, the server is able to "push" down updates to clients.

That said, when there is an issue with this service it will impact most real time functionality in web/mobile apps. This issue impacted the ability for interpreters using our apps to receive calls, requestors using our apps to place calls, and impacted push notifications to web/mobile. The last point mentioned could impact things like the async notifications for call log file download and in web toast appointment notifications.

At this time we have outstanding support tickets open with this third party for deeper root cause analysis and information on how they intend to prevent disruptions like this in the future. We have also begun some initial investigation on alternatives to this service, in case issues like this occur in the future. Due to most Boost functionality still being responsive and the date/time (fortunately) being the most "off peek" hours, monitoring triggers did not fire as quickly as we would have liked. We have refined triggers based on this type of error, so we can be more proactive in reaching out with workarounds.

It's important to note that all IVR, Direct Dial, and SIP calls were still able to be placed. If the BPIN was enabled for those accounts, they were also serviced by integrations that do not depend on the websocket service (ie. the calls would have successfully reached an interpreter). We did not even consider DAP since the functionality still available in platform far exceeded the capabilities of DAP. Additionally, all onsite appointment functionality and most of the web portal functionality (other than caller and push notifications mentioned above) was up the whole time.

We are sorry for the inconvenience this caused and for interpreters that were attempting to work during these hours and service calls. We will update this post-mortem when we receive any additional information from the third party managed service

Thanks

Boost Team