Back to overview
Degraded

NYCM2 Downtime

Oct 11 at 02:12pm BST
Affected services
New York City (US)

Resolved
Oct 12 at 01:32pm BST

It has been ~12 hours since the node has been online again and we have not noticed any issues, we will retain a copy of the VM imgs for few more days. Please submit a ticket if you have any issues.

Updated
Oct 12 at 02:35am BST

All servers were successfully restored, please submit a ticket if you have any issues.

Updated
Oct 12 at 01:20am BST

We have began the restore process, your VM will be automatically booted up again once completed. Control panel access will remain disabled whilst we monitor.

Updated
Oct 11 at 11:20pm BST

It's likely the thermal shutdown and reboots caused data corruption, we have had the broken fans replaced and NVMe out of caution. We will now begin reinstalling the node.

Updated
Oct 11 at 07:03pm BST

As the server has gone down again we will proceed with replacing the hardware components and restore from last available backup we have.

Updated
Oct 11 at 04:45pm BST

This looks to be an issue with the NVMes, we have been able to reboot the node.

We will likely need to replace the NVMe & reinstall the host node. We will attempt to copy the VM disks over.

We recommend all clients take backups whilst still online.

Created
Oct 11 at 02:12pm BST

NYCM2 is currently down, we are investigating the root cause. Updates to follow.