Back to overview
Degraded

NYCM2 Downtime

Oct 11, 2023 at 01:12pm UTC
Affected services
New York City (US)

Resolved
Oct 12, 2023 at 12:32pm UTC

It has been ~12 hours since the node has been online again and we have not noticed any issues, we will retain a copy of the VM imgs for few more days. Please submit a ticket if you have any issues.

Updated
Oct 12, 2023 at 01:35am UTC

All servers were successfully restored, please submit a ticket if you have any issues.

Updated
Oct 12, 2023 at 12:20am UTC

We have began the restore process, your VM will be automatically booted up again once completed. Control panel access will remain disabled whilst we monitor.

Updated
Oct 11, 2023 at 10:20pm UTC

It's likely the thermal shutdown and reboots caused data corruption, we have had the broken fans replaced and NVMe out of caution. We will now begin reinstalling the node.

Updated
Oct 11, 2023 at 06:03pm UTC

As the server has gone down again we will proceed with replacing the hardware components and restore from last available backup we have.

Updated
Oct 11, 2023 at 03:45pm UTC

This looks to be an issue with the NVMes, we have been able to reboot the node.

We will likely need to replace the NVMe & reinstall the host node. We will attempt to copy the VM disks over.

We recommend all clients take backups whilst still online.

Created
Oct 11, 2023 at 01:12pm UTC

NYCM2 is currently down, we are investigating the root cause. Updates to follow.