New York Data Centers coming back to life

From Mark Hachman at Slashdot:

NYC Data Centers Struggle to Recover After Sandy
Problems in New York�s data centers persisted through Wednesday morning, with hosting companies and other facilities racing against time to keep generators humming as water was pumped out of their facility basements.

The outages had taken several sites off line, most notably the Gawker network, whose hosting service, Datagram, suffered outages.

At about 7 PM on Oct. 29, local power utility ConEd began proactively shutting off power to parts of lower Manhattan, in an attempt to remove the load from the energy grid. Many data centers shifted to backup power as a result. Unfortunately, the storm�s storm surge was higher than expected, and the basements housing the generators flooded, rendering them inoperable. For the most part, these data centers then turned to alternate sources of power, typically generators that they hauled in themselves. (ConEd has provided a map of outages and the expected times when service is restored; Telx, one company affected by the outage with a facility at 60 Hudson St., said ConEd would restore power within 2 to 4 days.)

The fight now is to keep those generators fueled while pumps clear the basement areas, allowing the standard backup generators to begin operating. It�s also unclear whether the critical elements of infrastructure (power and communications) will both be up and running in time to restore services.

From host SquareSpace:

Update [5pm ET]
Terrific progress this afternoon.

We have a few hundred gallons of spare fuel on the roof along with a full tank, which will take us well into the night. On top of teams from Peer1, Fog Creek, and Squarespace, we were able to hire additional help from Brooklyn and Queens. Special thanks to Mike Mazzei and his team at Peer1 NYC for unprecedented dedication these past days.

We have a big crew teaming up tonight for a midnight bucket brigade, and another in the morning. Traffic is terrible with gridlock everywhere and no public transport, so we are scheduling carefully.

We are cautiously optimistic that fuel will be running to the roof by end of day tomorrow or the day after. Water is being successfully pumped from the basement at a rate of one foot per hour. Fifteen feet of water remain. We have increasing confidence that we may avoid downtime. Of course, there are no guarantees and we are working hour-by-hour.

From Fog Creek Software:

Outlook promising for all Fog Creek services
Trello is running happily on AWS and all other Fog Creek services continue to run in the data center without problems.

The fuel supply at ground level is sufficient for the foreseeable future. Additional labor has been added to the bucket brigade and the building is making progress on securing a fuel pump to replace the brigade. This all suggests the generator is at little to no risk of running out of fuel.

In a pleasant development, after previous ineffective efforts at pumping water from the basement, we are now removing one foot an hour, with about 15 feet left.

Too many damn politicians angling for the media these days. When will he hear an interview from a lineman or any of those intreppid sysadmins at these data centers.