Further update from our Carrier Partner - Magrathea
Following on from the issues that were experienced on Friday 2nd May, please find RFO details below.

We sincerely apologise for the inconvenience that this disruption will have caused you – thank you for your support and patience during this period.

Please do not reply to this email, if you have any queries please contact us on support@vipvoip.co.uk

Post Incident Review

Incident Symptom: Inability to process inbound/outbound SIP calls

Date of Incident: Friday 2nd May 2025

Incident Duration: 14:00 to 19:45 (intermittent and varying impact)

Incident Timeline:

Magrathea customers experienced issues with inbound SIP calls on Friday from approximately 14:15.

Numbers that had failover routing in place continued to function and the fault was intermittent meaning the severity and extent was unclear until approximately 14:44 at which point we issued our first service notice.

Service was initially restored a few minutes later with an all clear notice sent at 14:54. However the issue reappeared and impacted inbound SIP calls again intermittently up to 16:21 when it was cleared.

At approximately 17:50 we received reports of number translation targets being out of sync and a few minutes later reports of outbound calls failing. Both of these issues impacted a smaller number of clients but the same inbound call failures reappeared intermittently alongside. We finally reported the issue fully resolved at 19:48 and continued monitoring before issuing an ‘all clear’ notice at 21:00.

Incident Summary:

The primary cause of this incident was a memory leak condition in the systems responsible for monitoring and reporting errors. This had the effect of depriving operational memory to systems responsible for core network routing decisions causing them to malfunction.

Because the fault occurred in the monitoring systems themselves it frustrated our engineers attempt to quickly resolve the problem and in addition as the fault was intermittent in nature not all of our bank of four independent systems were affected.

These systems have run unchanged for many years without issue and so the issue was completely unexpected. We have taken immediate action to ensure that it cannot reoccur as well as planning additional separate ways to detect this type of issue promptly if it occurs elsewhere in the network.

Separately to this a minor fault occurred in our network during a similar time period which caused changes requested through our numbering API to not propagate through the network correctly. This fault has been isolated to a new network component that was undergoing our final test stages prior to deployment. This was not resolved with our normal speed due to our focus on the more significant fault above.

VIP VoIP - undefined undefined

Possible issues with inbound/outbound calls via carrier (Magrathea) - CLEARED