NYCM2 Downtime
Resolved
Oct 12, 2023 at 12:32pm UTC
It has been ~12 hours since the node has been online again and we have not noticed any issues, we will retain a copy of the VM imgs for few more days. Please submit a ticket if you have any issues.
Affected services
Updated
Oct 12, 2023 at 01:35am UTC
All servers were successfully restored, please submit a ticket if you have any issues.
Affected services
Updated
Oct 12, 2023 at 12:20am UTC
We have began the restore process, your VM will be automatically booted up again once completed. Control panel access will remain disabled whilst we monitor.
Affected services
Updated
Oct 11, 2023 at 10:20pm UTC
It's likely the thermal shutdown and reboots caused data corruption, we have had the broken fans replaced and NVMe out of caution. We will now begin reinstalling the node.
Affected services
Updated
Oct 11, 2023 at 06:03pm UTC
As the server has gone down again we will proceed with replacing the hardware components and restore from last available backup we have.
Affected services
Updated
Oct 11, 2023 at 03:45pm UTC
This looks to be an issue with the NVMes, we have been able to reboot the node.
We will likely need to replace the NVMe & reinstall the host node. We will attempt to copy the VM disks over.
We recommend all clients take backups whilst still online.
Affected services
Created
Oct 11, 2023 at 01:12pm UTC
NYCM2 is currently down, we are investigating the root cause. Updates to follow.
Affected services