| NOTICE TIME ZONE | Brisbane GMT+10 |
|---|---|
| STATUS (Open/Closed) | Closed |
| INCIDENT START DATE | 20210808 |
| INCIDENT START TIME (HH:MM) | Approx: 17:20pm |
| INCIDENT CAUSED BY A 3RD PARTY? | No |
| IF YES, NAME OF 3RD PARTY | |
| OUTAGE DURATION | 1 hour |
| ESTIMATED TIME TO RESOLUTION | N/A |
| XCRM TICKET NUMBER | Not Applicable |
| BRAND | XRACK |
| PRIORITY | P2 |
| CLIENTS AFFECTED | X16, X40, X70 & X81 |
| DESCRIPTION OF INCIDENT | During routine maintenance a host server entered critical low disk space and required rebooting |
| DESCRIPTION IMPACT | Clients unable to access servers |
| Primary Effect | Host server go to 2% available diskspace on C drive and hypervisor manager became unresponsive |
| Secondary Effect | Performance on servers became slow and eventually had no access due to reboot |
| Residual Effect | None expected once the problem is resolved |
| EVENT TIMELINE | |
| Sunday 8th August 2021 | |
| 16:45 | XSTRA’s monitoring system alerted diskspace entered critical state |
| 16:46 | Technicians investigated issue |
| 16:55 | Technicians identified cause and commenced resolving |
| 17:15 | Hyper-Visor manager became unresponsive and unable to continue with the clean-up |
| 17:20 | Host server rebooted |
| 17:25 | Host server back online and all virtual servers left turned off and clean-up continued |
| 17:45 | Clean-up completed, commenced bringing servers back online |
| 18:20 | Servers all back online and all checked to ensure connectivity is returned |
| RECOVERY & RESOLUTION | Removed replication on numerous servers due large file changes resulting in a backlog. replication was recreated once storage space was recovered. |
| ROOT CAUSE | Numerous customer virtual servers had a large and sudden change in files resulting in DRP systems consuming larger than normal storage space. This occurred at the same time routine maintenance was being performed that also consumes a large amount of space. |
| CORRECTIVE & PREVENTATIVE MEASURES | N/A |
Revision:
1
Last modified:
Aug 10, 2021
Post your comment on this topic.