Oriel — AI & technology worth noticing, every day.

A false smoke alarm almost drowned Jane Street's only NYC datacenter

Web page · https://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter

First seen by Oriel on .

AI summary

A Jane Street systems engineer recounts how a false smoke alarm disabled datacenter cooling, nearly triggering heat-sensitive sprinklers before he manually powered down every device.

  • Contractor dust triggered the alarm, pressurizing dry sprinkler pipes and shutting down the CRAC units even though no fire existed.
  • The engineer unplugged roughly 100 server cabinets as the room temperature climbed, stopping the rise at 143.6 F before the 150 F sprinkler threshold.
  • Because major NYC markets were closed, downtime was minimal, and Jane Street restored operations the next morning.
  • The incident exposed weak monitoring: Hong Kong noticed only because many servers simultaneously displayed rising temperatures.
  • The team built an OCaml monitor that polls every Linux server’s onboard sensors each minute, averages temperatures by location, and alerts the systems team when elevated.

#datacenter operations #incident response #ocaml #infrastructure monitoring

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 583, latest observation ; +576 over observed period since
  • Lobsters: 86, latest observation ; +83 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 275, latest observation ; +275 over observed period since
  • Lobsters: 17, latest observation ; +17 over observed period since

Aliases

SourceExternal IDURL
Hacker News50455394https://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter/ Read discussion
Lobstersii1gzphttps://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter/ Read discussion