Atomic time source failure at NIST Gaithersburg campus

Posted by dpcx 4 hours ago

Comments

Comment by floatin 3 hours ago

I wonder if it’s a response to this https://www.reuters.com/world/china/china-accuses-us-cyber-b...

Comment by CaliforniaKarl 3 hours ago

Huh, I guess it's best to think of each site's NIST NTP servers as 'load-balancers' in front of a single 'application server'.

Fun fact: Per [0], if you provide enough servers, the NTP client can detect a "falseticker" that is not providing accurate time. The number of NTP servers required is `2n+1` where `n≥1`.

Of course, that requires each NTP server use its own time source.

So, note for me: If I want NTP redundancy and I'm using NIST's servers, pick one NTP server from each of NTP's three sites.

[0]: https://support.ntp.org/Support/SelectingOffsiteNTPServers#U...

Comment by metaphor 2 hours ago

> So, note for me: If I want NTP redundancy and I'm using NIST's servers, pick one NTP server from each of NTP's three sites.

System robustness hazard that won't tolerate just querying time.nist.gov at 4-sec or greater intervals?

From the cow's mouth[1]:

>> The global address time.nist.gov is resolved to all of the server addresses below in a round-robin sequence to equalize the load across all of the servers.

[1] https://tf.nist.gov/tf-cgi/servers.cgi

Comment by RossBencina 3 hours ago

-10ms, no redundant clocks, and they're leaving most of the servers up with that amount of skew. Wow. I am astonished that NIST does not have multiple clocks over multiple distributed sites with robust ability to detect and bypass individual failures.

Comment by metaphor 2 hours ago

> I am astonished that NIST does not have multiple clocks over multiple distributed sites with robust ability to detect and bypass individual failures.

They may not operate redundant clocks at a single site, but ITS redundancy posture[1] doesn't look bad at all:

>> Servers at the Boulder and WWV/Ft. Collins campuses are independent and unaffected.

[1] https://tf.nist.gov/tf-cgi/servers.cgi

Comment by ianburrell 2 hours ago

They do have multiple clocks and sites. The primary clock is in Boulder. Only the Maryland time servers are affected, the Colorado ones should be fine. They mention switching to another atomic clock, but that probably has to be setup.

The email explains why they haven't shut down, cause haven't hit the threshold. And talks about maybe shutting them down manually.

Comment by jedimastert 2 hours ago

> I am astonished that NIST does not have multiple clocks over multiple distributed sites with robust ability to detect and bypass individual failures.

Is this sarcasm? I can't tell.

Per the email:

> Servers at the Boulder and WWV/Ft. Collins campuses are independent and unaffected.

Comment by octaane 2 hours ago

Good to know they have multiple backup clocks across the continental US.