Some clients are reporting issues with CONGESTION on their phone systems. This is a symptom of additional load on lines linking all carriers as a result of staff being sent home to work.
While this incident is extraordinary due to Covid-19, XSTRA is working around these issues as best as possible. Please do not call us on issues of congestion for both inbound and outbound calls.
We will be providing updates here directly from some of our carrier partners as we receive the information.
Thank you for your patience at this time.
______________________________________________________________________________
UPDATE 16: 25/03/2020 11:45AM AEDT
Update on Interconnect Capacity links experiencing congestion since COB yesterday.
Symbio Networks can confirm Inbound traffic has increased by over 80% and forecasting to reach double by the end of the week.
Telstra, Adelaide CCA
Symbio Networks will increase by a total of 57 E1’s for the Adelaide CCA.
Phase One – Symbio have ordered 14x E1’s today. The pre patching and tie cables are ready for this expansion. This has a target implementation date of the end of next week.
Phase Two – Symbio will increase capacity with a further 43 E1’s in this CCA. More information and ETA will be provided on this ASAP.
Further Expansions
Symbio Networks have increased the Optus capacity overnight by a total of 300 channels (150 channels in NSW and 150 channels in QLD). This brings the total national increase to 2,000 channels with Optus since last Monday, 16/03
As per announcements from the other industry carriers, interconnect links between carriers are strained due to the rapid scale of these increases attributed to the COVID-19 coronavirus. We will continue to provide updates on capacity expansions. Unfortunately, we are unable to provide an ETR at this stage.
Next Update: 18:30 Wednesday, 25/03
HISTORICAL UPDATES ON THIS ISSUE
________________________________________
UPDATE 1 : 1:39PM AEDT 16/03/2020
Due to unprecedented call volumes, we are experiencing congestion on our carrier interconnect trunks. Our engineers are investigating. We’ll provide an update once more information becomes available.
________________________________________
________________________________________
UPDATE 2 : 3:13PM AEDT 16/03/2020
Inbound traffic update:
We’re working with our carrier interconnect partners to ensure traffic is being evenly distributed across all trunks and paths.
We are also working with these partners to increase capacity.
Outbound traffic update:
We have negotiated with several customers with high per-second traffic to divert calls away from us temporarily.
We have increased capacity on some outbound carrier trunks where no network upgrade was required, and and we have rerouted traffic within our network to optimise the utilisation of outbound capacity.
The situation should now be improved.
________________________________________
UPDATE 3: 08:35AM AEDT 17/03/2020
Congestion Mitigation:
As per earlier updates, Symbio has implemented a number of network grooming measures throughout the course of today (Monday). In addition, Symbio has identified and is in the process of implementing further actions to manage the observed surge in traffic. Combined, these actions will alleviate overall congestion during peak call volume periods (0830-1300 AEDT).
Earlier congestion issues eased towards the end of the business day as traffic volumes subsided. Whilst the measures already taken will alleviate congestion, Symbio will be closely monitoring overall network performance from the start of business tomorrow, Tuesday, to verify network capacity/utilization. Further status updates will provided by midday local AEDT.
Capacity Increases:
In parallel, Symbio is actively working with our local carrier providers to expedite the delivery of additional Carrier Interconnect (CI) capacity across the network. As the new capacity becomes available it will be added to the network on priority. Symbio support and technology teams are aware of this major incident impacting our customers and will be closely monitoring tomorrow morning at peak times
________________________________________
________________________________________
UPDATE 4: 11:50AM AEDT 17/03/2020
Telcoinabox / Ivox apologizes for the lengthy impact and congestion on our network. Presently, resolving this incident is the number one priority to our business, with all teams and executive management working towards driving a resolution. Symbio networks are working closely with our carrier partners to alleviate the congestion on our network.
What has caused this sudden spike in traffic?
External factors showing unprecedented increase in inbound call logs. This can be attributed to the COVID-19 (or Coronavirus) pandemic being experienced around the globe.
Actions:
Symbio Networks have worked with Vodafone to expedite bringing into production new carrier interconnect capacity under a change request last night between 12-3am.
Yesterday afternoon, Symbio worked with Optus to increase our carrier interconnect capacity. We are continuing to increase this further today by distributing Optus traffic more evenly across our network. Evenly distributing this traffic is now a priority due to the sudden spike in traffic.
Symbio is also working with AAPT to increase call capacity. Executive management have escalated all orders to expedite this process.
Lastly, Symbio are working closely with Telstra to have their core routing updated to allow their traffic to be distributed evenly across the call collection areas. In parallel to this, Symbio networks are escalating all new capacity orders to accelerate the process.
====================================================
UPDATE 5: 17/03/2020 4:50
Dear customer,
Symbio Networks are continuing to work with our carrier partners to minimise congestion on our network. We apologies for the extended impact; please understand we are experiencing a high influx of traffic due to the COVID-19 virus.
This incident is our top priority.
This is a dynamic situation and we are constantly reviewing and, where possible, changing traffic patterns to alleviate the observed congestion.
________________________________________
UPDATE 6: 18/03/2020 1:45PM AEDT
Symbio Networks are continuing to work through the congestion issues being experienced on the carrier interconnect (CI) links between carriers for the past three days with the unprecedented traffic increases. Below is an update on how the industry is working together to alleviate the impact to our customers
Symbio Networks in conjunction with the other interconnect carriers have identified that one service is generating extreme traffic volumes in multiple call collection areas (CCA), resulting in the congestion. We have identified this as the Coronavirus Hotline, and we are working with the end user of this service and the other carriers involved in the call path to have traffic redistributed to CCA’s with spare capacity.
In Sydney and Melbourne, Symbio Networks are analysing calls to identify if there are any other specific high-volume numbers which are causing a significant impact. Targeting these high-volume numbers can allow us to free more capacity for other calls on our network in the short term.
In other measures we are continuing to work with the carrier interconnect partners to expand the capacity between both networks. Currently, Telstra and Symbio are expediting a change request (involving physical works within Telstra exchanges) to increase the capacity on these CI links. This change has many moving parts, but we have a targeted delivery date of the end of next week.
This incident has the full visibility of executive management and has Support and Technology teams working tirelessly to resolve this incident. Over the last 48 hours we have seen the industry come together to ensure voice services, which are still the primary means of communication, remain operational.
________________________________________
UPDATE 7: 19/03/2020 8:00AM AEDT
Here is an update on the ongoing major incident involving congestion issues being experienced on the carrier interconnect (CI) links between carriers.
Symbio Networks was able to further alleviate congestion in Newcastle and Hobart by rerouting the traffic of a high-volume customer to alternate paths within our Network. As a result, we have seen call congestion decrease in these areas.
Our Technology and Support teams continue to work with our Carrier partners to increase capacity for other parts of Australia, with main focus on our Melbourne and Sydney Infrastructure. Given the magnitude of these changes, and the number of parties involved, these works will take time to complete. We are working to reduce the lead times and will keep you updated as new capacity becomes available and is introduced into the network.
We aplogise for the lengthy delays but are working hard with our industry carrier partners to drive resolution.
________________________________________
________________________________________
UPDATE 8: 19/03/2020 12:10PM AEDT
After analysing the peak-time traffic patterns from today, we can confirm the congestion being experienced in Newcastle and Hobart has been relieved.
We are continuing to work with our carrier partners to alleviate the congestion in other major cities, with our main focuses being Sydney and Melbourne. As mentioned in our previous update, given the magnitude of these network changes, and the number of parties involved, these changes will take time to complete.
We apologise for the lengthy delays but are working hard with our industry carrier partners to drive resolution.
===========================================================
Update 9: 19/03/2020 COB
During a further day of substantial traffic increases we are continuing to work closely with our carrier partners to alleviate any areas of interconnection which is experiencing congestion. With recent changes we have alleviated 2 major regional Call Collection areas (Hobart & Newcastle).
There is also no congestion between Symbio and Optus, Vodafone, TPG/AAPT or Vocus.
At peak times today the congestion was isolated to links between Symbio and Telstra within Sydney and Melbourne. Telstra and Symbio have initiatives in place which are progressing well.
We have also kick-started initiatives with the other carriers to ensure headroom for any further potential spikes.
________________________________________
UPDATE 10: 20/03/2020 1:20PM AEDT
Update of Interconnect Capacity links experiencing Congestion.
Melbourne – Telstra / Symbio IC
Successfully completed patching works within one Melbourne exchange, awaiting Telstra to provision capacity. Works are being expedited.
Scheduled work on second Melbourne exchange set to be completed Tuesday, 24th March.
Symbio continue to work closely with Telstra on further mitigation steps and upgrades. Further updates will be sent as confirmed.
Next update prior to 21/03/2020
________________________________________
UPDATE 11: 20/03/2020 5:20PM AEDT
Update for the ongoing incident – Interconnect Capacity links experiencing congestion.
Telstra, Melbourne Call Collection Area (CCA)
Telstra Exchange, Windsor: Pre patch complete, awaiting Telstra to provision the capacity. These works are being expedited by Symbio Networks
Telstra Exchange, Lonsdale: Awaiting pre patch from Symbio Networks. These works are scheduled for the 24th of March
Telstra, Sydney CCA
Symbio Networks are working to expedite our capacity expansion which was scheduled for later this year. We are completing the initial design work before engaging contractors to complete the pre-patching. Symbio Networks will advise of expansion dates when they become available.
Telstra, other CCA’s: No congestion
Other bi-lateral Interconnect Partners: No congestion
Further expansions
Symbio Networks have added a further 1,000 channels with Vodafone, making it a total of 5,000 channels increased this week.
________________________________________
UPDATE 12: 23/03/2020 12:30PM AEDT
We are also aware of intermittent call failures into Telcoinabox service providers Inbound/1300 numbers , faults are being raised with upstream (Optus) to determine the root cause , but likely will also be due to congestion.
If your 1300 number is affected by this , please raise a ticket to the SDA team such that it can be added to the parent case.
Update on Interconnect Capacity links experiencing congestion.
Symbio Networks can confirm a further 20% increase in traffic today – taking Inbound traffic increases to over 80% and forecasting to reach double by the EOW.
Telstra, Melbourne Call Collection Area (CCA)
Telstra Exchange, Windsor: Pre patch complete, awaiting Telstra to provision the capacity. These works are being expedited by Symbio Networks. No further update on this.
Telstra Exchange, Lonsdale: Pre patch works were brought forward and were completed over the weekend. Symbio Networks are now working with Telstra to have the approval processes expedited.
Telstra, Sydney CCA
Symbio Networks is working to expedite our capacity expansion which was scheduled for later this year. We are completing the initial design work before engaging contractors to complete the pre-patching. Symbio Networks will advise of expansion dates when they become available.
Telstra, other CCA’s: No congestion
Other bi-lateral Interconnect Partners: No congestion
Further expansions
As well as an additional 5,000 channels with Vodafone added last week, Symbio Networks have added a further 1,000 channels with Optus to assist with the congestion issues being faced.
UPDATE 13: 23/03/2020 6:20 AEDT
Update on Interconnect Capacity links experiencing congestion.
Symbio Networks can confirm a further 20% increase in traffic today – taking Inbound traffic increases to over 80% and forecasting to reach double by the end of the week.
Telstra, Melbourne Call Collection Area (CCA)
Telstra Exchange, Windsor: Pre patch complete, awaiting Telstra to provision the capacity. These works are being expedited by Symbio Networks. No further update on this.
Telstra Exchange, Lonsdale: Pre patch works were brought forward and were completed over the weekend. Symbio Networks are now working with Telstra to have the approval processes expedited.
Telstra, Sydney CCA
Symbio Networks is working to expedite our capacity expansion which was scheduled for later this year. We are completing the initial design work before engaging contractors to complete the pre-patching. Symbio Networks will advise of expansion dates when they become available.
Telstra, Launceston CCA
Symbio Networks have added additional capacity in Launceston by provisioning a further 8x E1’s (250 channels).
Telstra, Hobart CCA
Symbio Networks have scheduled upgrades for Hobart to turn on an additional 520 channels which are due to be completed by tomorrow morning at 9am.
Telstra, other CCA’s: No congestion
Other bi-lateral Interconnect Partners: No congestion
Further expansions
As well as an additional 5,000 channels with Vodafone added last week, Symbio Networks have added a further 1,000 channels with Optus to assist with the congestion issues being faced.
Outage Today
Symbio Networks attempted to increase capacity on Optus today. Due to the sudden increase in load, instability was caused on some network elements. This required an emergency restart to restore services between 13:00 and 14:00.
Symbio Networks apologises for the continued congestion. Please understand that this congestion is being experienced industrywide and can be attributed to the COVD-19 coronavirus. The high influx of traffic on the interconnect links is unprecedented. Unfortunately, we are unable to provide an ETR at this stage.
________________________________________
UPDATE 14: 24/03/2020 12:00PM AEDT
Update on Interconnect Capacity links experiencing congestion.
Symbio Networks can confirm Inbound traffic has increased by over 80% and forecasting to reach double by the end of the week.
Telstra, Melbourne Call Collection Area (CCA)
Telstra Exchange, Windsor: Pre patch complete, awaiting Telstra to provision the capacity. These works are being expedited by Symbio Networks. No further update on this.
Telstra Exchange, Lonsdale: Pre patch works were brought forward and were completed over the weekend. Symbio Networks are now working with Telstra to have the approval processes expedited.
Telstra, Sydney CCA
Symbio Networks is working to expedite our capacity expansion which was scheduled for later this year. We are completing the initial design work before engaging contractors to complete the pre-patching. Symbio Networks will advise of expansion dates when they become available.
Telstra, Launceston CCA
Symbio Networks have added additional capacity in Launceston by provisioning a further 8x E1’s (250 channels).
Telstra, Hobart CCA
Symbio Networks have scheduled upgrades for Hobart to turn on an additional 520 channels which are due to be completed by COB Wednesday, 25/03.
Telstra, Adelaide CCA
Symbio Networks are working closely with Telstra to add additional capacity. Tie Cables are ready. Further updates to be provided as soon as possible
Telstra, other CCA’s: No congestion
Other bi-lateral Interconnect Partners: No congestion
Further expansions
As well as an additional 5,000 channels with Vodafone added last week, Symbio Networks have added a further 1,000 channels with Optus to assist with the congestion issues being faced.
As per announcements overnight interconnect links between carriers are strained due to the rapid scale of these increases attributed to the COVID-19 coronavirus. We will continue to provide updates on capacity expansions. Unfortunately, we are unable to provide an ETR at this stage.
________________________________________
UPDATE 15: 24/03/2020 18:30 AEDT
Update on Interconnect Capacity links experiencing congestion.
Symbio Networks can confirm Inbound traffic has increased by over 80% and forecasting to reach double by the end of the week.
COB Update:
Telstra, Adelaide CCA
Symbio Networks are working with Telstra to provision a further 30x E1’s (approximately 900 channels). Further information will be provided as soon as possible.
Midday Update:
Telstra, Melbourne Call Collection Area (CCA)
Telstra Exchange, Windsor: Pre patch complete, awaiting Telstra to provision the capacity. These works are being expedited by Symbio Networks. No further update on this.
Telstra Exchange, Lonsdale: Pre patch works were brought forward and were completed over the weekend. Symbio Networks are now working with Telstra to have the approval processes expedited.
Telstra, Sydney CCA
Symbio Networks is working to expedite our capacity expansion which was scheduled for later this year. We are completing the initial design work before engaging contractors to complete the pre-patching. Symbio Networks will advise of expansion dates when they become available.
Telstra, Launceston CCA
Symbio Networks have added additional capacity in Launceston by provisioning a further 8x E1’s (250 channels).
Telstra, Hobart CCA
Symbio Networks have scheduled upgrades for Hobart to turn on an additional 520 channels which are due to be completed by COB Wednesday, 25/03.
Telstra, Adelaide CCA
Symbio Networks are working closely with Telstra to add additional capacity. Tie Cables are ready. Further updates to be provided as soon as possible
Telstra, other CCA’s: No congestion
Other bi-lateral Interconnect Partners: No congestion
Further expansions
As well as an additional 5,000 channels with Vodafone added last week, Symbio Networks have added a further 1,000 channels with Optus to assist with the congestion issues being faced.
As per announcements overnight interconnect links between carriers are strained due to the rapid scale of these increases attributed to the COVID-19 coronavirus. We will continue to provide updates on capacity expansions. Unfortunately, we are unable to provide an ETR at this stage.
Post your comment on this topic.