Good Day,
Network Incident Reference: #878 Current Status Engineers discovered another DHCP storm that started in the early hours of this morning. The root cause of this storm is being investigated. They have made a configuration change to alleviate this and are monitoring. Area/s Affected: Cape Town Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Tue, Feb 12, 2019 at 8:15 AM Vumatel Network Incident < networkincident@vumatel.co.za> wrote:

Good Day,
Network Incident Reference: #878 Current Status Due to the number of suburbs that are being load-shed at a time in Cape Town, we are experiencing longer than normal delays with DHCP request processing when power is restored.
We are looking at various options to decrease the recovery time when power restores. As an interim measure, we have dropped all management traffic to minimise the number of DHCP requests. This means that COS will show a ‘?’ Link status on all most Cape Town services, irrespective of the actual service status.
Our technical team is working continuously to find a permanent solution to ensure that services restore as fast as possible after load-shedding. Area/s Affected: Cape Town Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Tue, Feb 12, 2019 at 9:36 AM Vumatel Network Incident < networkincident@vumatel.co.za> wrote:

Good Day,
Network Incident Reference: #878 Current Status Last night we discovered an issue when customers took longer than normal to re-enter the network. During the troubleshooting process through the night, we discovered another unrelated issue on our core equipment in Cape Town and had to reboot it. It took some time to find this issue as it occurred simultaneously with load-shedding.
Rebooting the equipment reduced the DHCP processing time for the remaining devices in the network that had not yet restored after load-shedding and these remaining customers were up and running shortly after. As at now, services have restored to normal.
We are closely monitoring the network and the processing time of the DHCP requests. The first major power drop that is expected today is from 12h00 to 14h30. We will provide an update thereafter. Area/s Affected: Cape Town Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Tue, Feb 12, 2019 at 7:12 PM Vumatel Network Incident < networkincident@vumatel.co.za> wrote:

Good Day,
Network Incident Reference: #878 Current Status DHCP processing went through very smoothly and services restored in good time since the power restored at 14h30.
The affected areas will be impacted by another power drop between 20h00 – 22h30 tonight.
We will continue to monitor the process and will feedback tomorrow morning.
If any customers’ service is still down in the Cape Town area and not currently impacted by load shedding, please have them reboot both the CPE and the router to restore connectivity. Area/s Affected: Cape Town Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Wed, Feb 13, 2019 at 9:50 AM Vumatel Network Incident < networkincident@vumatel.co.za> wrote:

Good Day,
This is a technical update ONLY. Please do NOT share this with your customers or on social platforms. This is a technical update and should be interpreted and reworked in a customer-friendly way before sharing this information with customers.
Network Incident Reference: #878 Current Status This is a technical update ONLY. Please do NOT share this with your customers or on social platforms. This is a technical update and should be interpreted and reworked in a customer-friendly way before sharing this information with customers.
Although DHCP requests were processed smoothly after power was restored at 14h30 yesterday in Cape Town, we encountered another issue with DHCP processing last night. Power to multiple areas across Cape Town restored within 30 minutes of each other resulting in an influx of DHCP requests causing delays. This issue was fully resolved during the early hours of this morning. Should any of your customers in Cape Town still be down following the restoration of power, please log a ticket for them in Quickbase so that we can investigate further.
During the course of today, we will be installing additional equipment in Cape Town to alleviate the DHCP load being transmitted simultaneously. This will result in downtime for specific areas. We aim to complete this work during load-shedding to minimise the impact on end users. Should load-shedding be canceled, we will communicate the affected areas to you ahead of time. Area/s Affected: Cape Town | Northern Suburbs Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Wed, Feb 13, 2019 at 3:51 PM Vumatel Network Incident < networkincident@vumatel.co.za> wrote:

Good Day,
This is a technical update ONLY. Please do NOT share this with your customers or on social platforms. This is a technical update and should be interpreted and reworked in a customer-friendly way before sharing this information with customers.
Network Incident Reference: #878 Current Status
Due to the large scale load shedding that has been occurring this week, there has been widespread, on-going, disruption of fibre Internet services in Cape Town.

When power is restored there are tens of thousands of devices simultaneously attempting to reconnect to the network and restore their configurations. Some devices connect and restore quickly and others have to attempt several times before they are processed resulting in long delays. Under load, certain configuration processes fail or are stalled resulting in a service that appears to have been restored but is not working.

Vumatel has senior engineers from its supplier and equipment vendor working non-stop to resolve the issue of services not restoring completely. We also made some changes to our network in Cape Town today to alleviate some of the impact and will monitor this closely.
We are focused on reducing the impact of this issue and will continue to make changes to the network where possible until we see better results in the restoration of internet services post-load-shedding.

We apologise for the inconvenience this has caused. Area/s Affected: Cape Town | Northern Suburbs Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Thu, Feb 14, 2019 at 10:02 AM Vumatel Network Incident < networkincident@vumatel.co.za> wrote:

Good Day,
This is a technical update ONLY. Please do NOT share this with your customers or on social platforms. This is a technical update and should be interpreted and reworked in a customer-friendly way before sharing this information with customers.
Network Incident Reference: #878 Current Status We have been closely investigating the network in Cape Town in light of the DHCP storm issues which have occurred this week. While investigating these issues, senior engineers from our supplier and equipment vendor identified and resolved an inter-vrf routing issue on our core equipment in Cape Town last night. This inadvertently caused a complete drop in traffic in Cape Town between 19h00 and 19h35 on 14 February 2019. The resolution of the inter-vrf issue also allowed for all DHCP requests to be processed and services to restore within a reasonable time frame when power restored at 21h45.
We are optimistic that the above has resolved the incident but will continue to monitor the service.
If any customers’ service has not yet restored, please power down the Raycore CPE device and the router. After 10 minutes, power up the Raycore CPE only and wait for the Power and Fx lights to stabilise. When these lights have stabilised, power up the router. After the router has powered up, please force through a new DHCP request and allow this process to run through completely before testing internet connectivity.
If the above fails please use the detailed technical information available to you on COS to confirm the following:
· The operational status of the interface and line is up o GigabitEthernet0/0/11 current state : UP o Line protocol current state : UP · The router has a valid lease and correct IP address on the correct VLAN
If the information on COS is as above, please run through your own diagnostic and troubleshooting process.
If the information on COS is not as above, please log a ticket with us on Quickbase. Area/s Affected: Cape Town | Northern Suburbs Network Type Affected: Active Ethernet Network Incident Description: No connectivity Network Incident Impact: Residents have no connectivity
On Thu, Feb 14, 2019 at 6:56 PM Vumatel Network Incident < networkincident@vumatel.co.za> wrote: