False Smoke Alarm Nearly Destroyed Jane Street's Datacenter
A false smoke alarm triggered when contractors kicked up dust in an adjacent prep room nearly destroyed Jane Street’s datacenter in 1 New York Plaza. The smoke detector automatically shut down the CRAC cooling units to prevent fires from spreading, but with cooling offline, servers began rapidly heating toward the 150°F threshold where overhead sprinklers would activate and flood the machine room—potentially business-ending damage.
The author, a systems engineer living nine minutes away on foot, was called after Hong Kong staff detected unusual temperature spikes across multiple servers. Running through the Financial District, he yanked power cables from every device row by row. Temperature stabilized at 143.6°F, just below the trigger point. With NYC markets closed for the day, downtime was minimal and systems were restored by morning.
Subsequent meetings confirmed the sprinkler system would stay while procedures were tightened. The incident revealed that ad-hoc monitoring had caught the crisis only by accident—prompting the author to design a dedicated system monitor to query all servers and detect generalized cooling failures, replacing alerts tuned to individual machines.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.