NOTICE TIME ZONE Brisbane GMT+10
STATUS (Open/Closed) Closed
INCIDENT START DATE 20200826
INCIDENT START TIME (HH:MM) Approx: 18:00
ESTIMATED TIME TO RESOLUTION N/A
INCIDENT END DATE 20200826
INCIDENT END TIME (HH:MM) Approx: 18:05
OUTAGE DURATION Approx. 5 Minutes
INCIDENT CAUSED BY A 3RD PARTY? No
IF YES, NAME OF 3RD PARTY N/A
XCRM TICKET NUMBER 12826
BRAND XRACK
PRIORITY P5 – Very Low
CUSTOMERS AFFECTED X40
DESCRIPTION OF INCIDENT A virtual machine for X40 (A-40-0-VS27) contains a non-critical PostGreSQL installation and this service restarted as a result of an auto-upgrade to the PostgreSQL packages.
PRIMARY IMPACT EFFECT Access to the databases hosted on that server would have been inaccessible for a few minutes
SECONDARY IMPACT EFFECT An automated script based data download and import into a database on the system may have been interrupted
EVENT TIMELINE
WEDNESDAY 26TH AUGUST 20020
18:00 XSTRA’s monitoring system, PRTG, advising that balances tables is empty after extraction
19:06 Email sent to database and support team informing that the table was empty
19:08 Randolph (X40) asked for further investigation. Hao (X40) began investigating
20:12 Hao informed that it would not affect current reporting. The table affected was not important for business reporting
21:00 Chris Farrell (XSTRA) began investigating
21:38 Chris notified his findings from the postgres logs – a shutdown to postgres had occurred
22:44 Chris notified that he had completed a manual update of the table
RECOVERY & RESOLUTION Chris Farrell downloaded and extracted the table then uploaded the data to the database
ROOT CAUSE Ubuntu applied a security patch to PostgreSQL and restarted PostgreSQL whilst the data was being populated. Log extract is as follows:
2020-08-26 06:11:33.829 UTC,,,1282,,5f2cc8fb.502,4,,2020-08-07 03:22:35 UTC,,0,LOG,00000,“received fast shutdown request”,,,,,,,,,”“
2020-08-26 06:11:33.838 UTC,,,1282,,5f2cc8fb.502,5,,2020-08-07 03:22:35 UTC,,0,LOG,00000,“aborting any active transactions”,,,,,,,,,”“
2020-08-26 06:11:33.842 UTC,“postgres”,“kidsoft”,170431,”[local]”,5f45d83f.299bf,5,“COPY”,2020-08-26 03:34:23 UTC,14/1043,1321687,FATAL,57P01,“terminating connection due to administrator command”,,,,,“COPY balances, line 67510085:
CORRECTIVE & PREVENTATIVE MEASURES PostgreSQL has been added to the auto-update blacklist in Ubuntu
RESIDUAL EFFECT None

Feedback

Was this helpful?

Yes No
You indicated this topic was not helpful to you ...
Could you please leave a comment telling us why? Thank you!
Thanks for your feedback.

Post your comment on this topic.

Please do not use this for support questions.
https://x.direct/1/en/topic/welcome

Post Comment