| NOTICE TIME ZONE | Brisbane GMT+10 |
|---|---|
| STATUS (Open/Closed) | Closed |
| INCIDENT START DATE | 20200826 |
| INCIDENT START TIME (HH:MM) | Approx: 18:00 |
| ESTIMATED TIME TO RESOLUTION | N/A |
| INCIDENT END DATE | 20200826 |
| INCIDENT END TIME (HH:MM) | Approx: 18:05 |
| OUTAGE DURATION | Approx. 5 Minutes |
| INCIDENT CAUSED BY A 3RD PARTY? | No |
| IF YES, NAME OF 3RD PARTY | N/A |
| XCRM TICKET NUMBER | 12826 |
| BRAND | XRACK |
| PRIORITY | P5 – Very Low |
| CUSTOMERS AFFECTED | X40 |
| DESCRIPTION OF INCIDENT | A virtual machine for X40 (A-40-0-VS27) contains a non-critical PostGreSQL installation and this service restarted as a result of an auto-upgrade to the PostgreSQL packages. |
| PRIMARY IMPACT EFFECT | Access to the databases hosted on that server would have been inaccessible for a few minutes |
| SECONDARY IMPACT EFFECT | An automated script based data download and import into a database on the system may have been interrupted |
| EVENT TIMELINE | |
| WEDNESDAY 26TH AUGUST 20020 | |
| 18:00 | XSTRA’s monitoring system, PRTG, advising that balances tables is empty after extraction |
| 19:06 | Email sent to database and support team informing that the table was empty |
| 19:08 | Randolph (X40) asked for further investigation. Hao (X40) began investigating |
| 20:12 | Hao informed that it would not affect current reporting. The table affected was not important for business reporting |
| 21:00 | Chris Farrell (XSTRA) began investigating |
| 21:38 | Chris notified his findings from the postgres logs – a shutdown to postgres had occurred |
| 22:44 | Chris notified that he had completed a manual update of the table |
| RECOVERY & RESOLUTION | Chris Farrell downloaded and extracted the table then uploaded the data to the database |
| ROOT CAUSE | Ubuntu applied a security patch to PostgreSQL and restarted PostgreSQL whilst the data was being populated. Log extract is as follows: 2020-08-26 06:11:33.829 UTC,,,1282,,5f2cc8fb.502,4,,2020-08-07 03:22:35 UTC,,0,LOG,00000,“received fast shutdown request”,,,,,,,,,”“ 2020-08-26 06:11:33.838 UTC,,,1282,,5f2cc8fb.502,5,,2020-08-07 03:22:35 UTC,,0,LOG,00000,“aborting any active transactions”,,,,,,,,,”“ 2020-08-26 06:11:33.842 UTC,“postgres”,“kidsoft”,170431,”[local]”,5f45d83f.299bf,5,“COPY”,2020-08-26 03:34:23 UTC,14/1043,1321687,FATAL,57P01,“terminating connection due to administrator command”,,,,,“COPY balances, line 67510085: |
| CORRECTIVE & PREVENTATIVE MEASURES | PostgreSQL has been added to the auto-update blacklist in Ubuntu |
| RESIDUAL EFFECT | None |
Revision:
6
Last modified:
Aug 27, 2020
Post your comment on this topic.