>This chassis was located in Melbourne and it was part of the mobile core timing architecture and was designed to provide stratum 2 and stratum 3 NTP services. When this server went back on line (at 2:50am), the GPS card in the chassis had reset during the change and began sending the incorrect date from 2006 into the network. The GPS card was actively used in that chassis and was missing a critical firmware update that was required to prevent this shift in time due to a known GPS rollover that was published in bulletins by the supplier. While the planned change of the NTP server chassis was the trigger point to begin the Outage, the steps leading to that point go back many years, as far back as
2020.
The core issue however was accumulation of technical debt (architectural changes in how timing systems were managed were not being tracked) and not recognising NTP as a sovereign system that needed to be properly managed.
>This chassis was located in Melbourne and it was part of the mobile core timing architecture and was designed to provide stratum 2 and stratum 3 NTP services. When this server went back on line (at 2:50am), the GPS card in the chassis had reset during the change and began sending the incorrect date from 2006 into the network. The GPS card was actively used in that chassis and was missing a critical firmware update that was required to prevent this shift in time due to a known GPS rollover that was published in bulletins by the supplier. While the planned change of the NTP server chassis was the trigger point to begin the Outage, the steps leading to that point go back many years, as far back as 2020.
The core issue however was accumulation of technical debt (architectural changes in how timing systems were managed were not being tracked) and not recognising NTP as a sovereign system that needed to be properly managed.