VIP VoIP - Possible issues with calls via Magrathea - Investigating – Incident details

A further update has been provided from our Carrier partner regarding the issue with some calls yesterday.

**********************************************************************************************************

Our technical specialists have continued working through the evening and overnight to closely monitor our network stability whilst also carrying out further investigations into the root cause of yesterdays' outage. 

As you will appreciate, such a serious outage requires thorough investigation and the volume of logs and analysis to be done takes some time.   We have now reached a conclusion on the root cause of the outage..

In summary, we can confirm that there was interruption to some calls between 10:34am and 11:20am on the 20th November, with service returning sporadically during that period.  A core routing device experienced a fault that affected the internal BGP routing, which caused some of our RTP instances to have incorrect routing for UDP traffic. This led to call switching servers failing to respond to requests and forward calls, requiring each affected RTP machine to be restarted. 

The team continue to carry out tests and put plans in place to improve resilience should a similar fault occur in the future, and of course are constantly monitoring the network while we have this vulnerability.  This work will continue throughout the day as a matter of utmost priority.

Following on from the earlier issues with Magrathea we have now had the following further update from them.

We are now confident that we have isolated the problem but the underlying cause is still being investigated.

Our initial findings are, as you can expect, that the failover systems did not meet our expectations in this event and that itself will be taken on as a project by our team to ensure greater resilience going forward.

Meanwhile, our technical team will be fully focussed on proving the underlying cause so we can provide a full and transparent RFO. To ensure this is accurate and useful we ask for your patience but hope to get the document to you no later than tomorrow afternoon

We sincerely apologise for the inconvenience that this disruption will have caused you this morning. Thank you for your support and patience during this period.

Kind regards

Magrathea Telecommunications Ltd

This incident has been resolved.

The issue affecting inbound and outbound calls this morning is now cleared.

The carrier engineers are in the process of investigating and more details will follow as soon as we can.

We sincerely apologise for the disruption this morning.

Please do not reply to this email, if you have any queries please contact us

on support@vipvoip.co.uk

Kind regards

VIP VoIP Support Team

W: www.vipvoip.co.uk

We are still experiencing some issues with both inbound and outbound calls.

Magrathea engineers are in the process of fully restoring service - more updates will follow as soon as they are available. We apologise for the disruption.

We apologise for the disruption.

Please do not reply to this email, if you have any queries please contact us

on support@vipvoip.co.uk

Kind regards

VIP VoIP Support Team

T: 03300 881182

W: www.vipvoip.co.uk