Helium outage

June 26, 2013 - 3:00 PM - all nodes are back up

June 26, 2013 - 12:45 PM - 13 racks of nodes up.

June 26, 2013 - 12:15 AM - 12 racks back up. The nodes in the remaining two racks will be up soon.

June 25, 2013 - 4:00 PM - 10 racks are back up. All 12-core nodes are back in production.

June 25, 2013 - 11:15 AM - 6 racks are now back in service.

June 25, 2013 - 7:45 AM - 4 racks of servers are back in production. Some jobs are running.

June 24, 2013 - 5:30PM - One rack of compute nodes has been returned to service. Additional nodes will come online over the next 24-36 hours.

June 24, 2013 - 4:30PM - The compute nodes on the Helium system are currently being rebuilt. This process is taking longer than expected and at this time we do not expect full system functionality until sometime on June 25th. Some nodes will become available during the evening of June 24th.

June 24, 2013 - 9:50am - The Helium cluster went down due to a power failure. Power was restored approximately five minutes later.