4C Communications- RFO - 2012.04.15
Wholesale and Hosted PBX Customers migrated to the new 4C Communications in Business production platform.
Description of issue
Between 7AM and 9:45AM PST the 4C Communications in Business network encountered a service impairment preventing inbound calls from connecting and delays in connect outbound calls.
Incident Time Log
7:10 AM - Monitoring System alerted 4C Communications on call staff of potential problem
7:20 AM - On-call staff tested and confirmed inbound impairment
7:35 AM - On-call staff isolated delays to authentication servers
7:40 AM - Vendor support notified of issue
7:45 AM - Vendor support request acknowledged began analysis
8:25 AM - Vendor identified root cause and began applying service patch
9:00 AM - Service restored to legacy platform
9:30 AM - Vendor support completed update; service restored
Root Cause Analysis
This service impairment was caused by processing delays exceeding the timeout values of
registration and our inbound vendors. The authentication server degradation was resolved with
the implementation of the service patch.