A false smoke alarm almost drowned Jane Street's only NYC datacenter
Web page · https://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter
First seen by Oriel on .
AI summary
A Jane Street systems engineer recounts how a false smoke alarm disabled datacenter cooling, nearly triggering heat-sensitive sprinklers before he manually powered down every device.
- Contractor dust triggered the alarm, pressurizing dry sprinkler pipes and shutting down the CRAC units even though no fire existed.
- The engineer unplugged roughly 100 server cabinets as the room temperature climbed, stopping the rise at 143.6 F before the 150 F sprinkler threshold.
- Because major NYC markets were closed, downtime was minimal, and Jane Street restored operations the next morning.
- The incident exposed weak monitoring: Hong Kong noticed only because many servers simultaneously displayed rising temperatures.
- The team built an OCaml monitor that polls every Linux server’s onboard sensors each minute, averages temperatures by location, and alerts the systems team when elevated.
#datacenter operations #incident response #ocaml #infrastructure monitoring
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 583, latest observation ; +576 over observed period since
- Lobsters: 86, latest observation ; +83 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 275, latest observation ; +275 over observed period since
- Lobsters: 17, latest observation ; +17 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50455394 | https://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter/ Read discussion |
| Lobsters | ii1gzp | https://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter/ Read discussion |