Incidents reported on status page for INIZ https://status.iniz.com/ https://d1lppblt9t2x15.cloudfront.net/logos/02639576347a8064f709c7fb5651206b.png https://status.iniz.com/ en https://status.iniz.com/ Mon, 28 Apr 2025 14:42:09 +0000 https://status.iniz.com/#8dc72c47888365a2628809d8ac20eb2a9e134ab30e0fecfa73ec90c759e59e2b Dallas (US) recovered https://status.iniz.com/ Mon, 28 Apr 2025 14:35:33 +0000 https://status.iniz.com/#8dc72c47888365a2628809d8ac20eb2a9e134ab30e0fecfa73ec90c759e59e2b Dallas (US) went down https://status.iniz.com/ Fri, 07 Mar 2025 16:35:59 +0000 https://status.iniz.com/#684f4abbdd3d65387985a2492845715c726558416ca80d58e52c287905409a03 Amsterdam (NL) recovered https://status.iniz.com/ Fri, 07 Mar 2025 16:29:15 +0000 https://status.iniz.com/#684f4abbdd3d65387985a2492845715c726558416ca80d58e52c287905409a03 Amsterdam (NL) went down https://status.iniz.com/ Fri, 07 Mar 2025 16:24:21 +0000 https://status.iniz.com/#9bd7ca16259a1b05ee8a85fa84f3ba1319d21b688bc9d3e7689226128bf47aac Amsterdam (NL) recovered https://status.iniz.com/ Fri, 07 Mar 2025 16:08:15 +0000 https://status.iniz.com/#9bd7ca16259a1b05ee8a85fa84f3ba1319d21b688bc9d3e7689226128bf47aac Amsterdam (NL) went down https://status.iniz.com/ Fri, 07 Mar 2025 09:03:01 +0000 https://status.iniz.com/#4f241c981308c99ccf5249ad64a53c4e757f530d8acaeb7fa782f160e40679e1 Amsterdam (NL) recovered https://status.iniz.com/ Fri, 07 Mar 2025 02:25:49 +0000 https://status.iniz.com/#4f241c981308c99ccf5249ad64a53c4e757f530d8acaeb7fa782f160e40679e1 Amsterdam (NL) went down https://status.iniz.com/incident/374995 Sun, 26 May 2024 15:18:00 -0000 https://status.iniz.com/incident/374995#80b74d7d8043ff2cbe17453c237c534945a1d7214a0a82fa78fe0cc81121fe06 Power (and nodes) have been restored over an hr ago and appears to be stable whilst we continue to monitor closely and await details of the outage. https://status.iniz.com/incident/374995 Sun, 26 May 2024 12:15:00 -0000 https://status.iniz.com/incident/374995#5fb16e0e18ac2b7b86373d84fbc8331611d3bbd2d6d1eddd6c6e35128005ba42 Our DC in Dallas has lost power and we're awaiting further updates from our provider. https://status.iniz.com/incident/295433 Wed, 06 Dec 2023 14:36:30 -0000 https://status.iniz.com/incident/295433#033b515569eb7a6848bcd14051782b75a6fe3dcff08a08bc3d7d746969b9a1ca As per our recent notification we are migrating customers over from older hardware utilising our old backend to new hardware and upgraded control panel. https://status.iniz.com/incident/295433 Tue, 05 Dec 2023 16:51:53 -0000 https://status.iniz.com/incident/295433#033b515569eb7a6848bcd14051782b75a6fe3dcff08a08bc3d7d746969b9a1ca As per our recent notification we are migrating customers over from older hardware utilising our old backend to new hardware and upgraded control panel. https://status.iniz.com/incident/295433 Tue, 05 Dec 2023 16:00:00 +0000 https://status.iniz.com/incident/295433#217096b9b3c2a682922db62871d0dd41c4ae69001728415bf38a04ef3b82d447 Maintenance completed https://status.iniz.com/incident/295433 Mon, 04 Dec 2023 00:50:09 -0000 https://status.iniz.com/incident/295433#033b515569eb7a6848bcd14051782b75a6fe3dcff08a08bc3d7d746969b9a1ca As per our recent notification we are migrating customers over from older hardware utilising our old backend to new hardware and upgraded control panel. https://status.iniz.com/incident/295433 Sun, 03 Dec 2023 16:00:00 -0000 https://status.iniz.com/incident/295433#033b515569eb7a6848bcd14051782b75a6fe3dcff08a08bc3d7d746969b9a1ca As per our recent notification we are migrating customers over from older hardware utilising our old backend to new hardware and upgraded control panel. https://status.iniz.com/incident/290512 Wed, 22 Nov 2023 02:20:00 -0000 https://status.iniz.com/incident/290512#2b6938cbed5b35c05386f8d9371008ddcfc5d84383972e492a3b30d5fa61658a The CPU cooler was replaced and node is back online. VMs should be booting within the next few minutes. https://status.iniz.com/incident/290512 Wed, 22 Nov 2023 01:13:00 -0000 https://status.iniz.com/incident/290512#3d5edaf5000fd2906751fb3655cf57e057aa307ad6ceef096250322193408a0c We have seen a thermal trip on the server causing it to shut down, we are awaiting for our DC to check and replace any failed fans/cooling issues. https://status.iniz.com/incident/277397 Thu, 26 Oct 2023 12:33:00 -0000 https://status.iniz.com/incident/277397#23c539156d7f512ae91337a375c5fd58f4d8cbdcaed1445a4685a07a244f9b44 This is now resolved, the root cause was redundant fibre rings from our provider were both cut however we established they had these from the same provider and will implement additional connectivity. https://status.iniz.com/incident/277397 Wed, 25 Oct 2023 18:34:00 -0000 https://status.iniz.com/incident/277397#9b868cdd610a14402a7ff083391fe0e874ad97f2ce17b9c3265ae58e3441c449 We see connectivity being restored and awaiting final confirmation the issue is resolved. https://status.iniz.com/incident/277397 Wed, 25 Oct 2023 16:14:00 -0000 https://status.iniz.com/incident/277397#fb327c7502cdc69100583ce56779ce21d50807dba5e3cc611b22d62a66853f1b Unfortunately we do not have any further updates apart from the one posted here: https://tranzact.zayo.com/#!/case-detail/5004z00001tVV9sAAG/TTN-0007089358/ We must stress there is no loss of data or any power outage, it is purely a network outage affecting connectivity and once restored your VM will be back online in the existing state. https://status.iniz.com/incident/277397 Wed, 25 Oct 2023 09:34:00 -0000 https://status.iniz.com/incident/277397#6bb74cfc3c31c263d717b8cc3d9a9ef4c88b19162a0a23db4fa6d4e8405f2646 We have been informed the maintenance to resolve this has been extended till 10:30 Eastern Time. Direct updates can be followed via https://tranzact.zayo.com/#!/case-detail/5004z00001tVV9sAAG/TTN-0007089358/ https://status.iniz.com/incident/277397 Wed, 25 Oct 2023 07:35:00 -0000 https://status.iniz.com/incident/277397#45e79481d31f4dad3ed3394d6985dc7d98f961cb15bcc9fd3dc6def76339fc2c Due to a fibre break between major DC facilities there is currently large outage affecting our NYCM data centre. This is purely network issue which we have received estimate time to resolve within the next 2 hours. https://status.iniz.com/incident/271292 Thu, 12 Oct 2023 12:32:00 -0000 https://status.iniz.com/incident/271292#4f5e5da838300193e33f2a29119b3ec92ab00b4eb156f7dfabd1eef844112fe1 It has been ~12 hours since the node has been online again and we have not noticed any issues, we will retain a copy of the VM imgs for few more days. Please submit a ticket if you have any issues. https://status.iniz.com/incident/271292 Thu, 12 Oct 2023 01:35:00 -0000 https://status.iniz.com/incident/271292#09b51e657d97d6c4115c62a6bf6519a5c6f36f44ad68867207a097519bf649ed All servers were successfully restored, please submit a ticket if you have any issues. https://status.iniz.com/incident/271292 Thu, 12 Oct 2023 00:20:00 -0000 https://status.iniz.com/incident/271292#e34ebc22fe916b0d999eb4aa916d96a34d1e5bb51bf509bf353392db4d1781a2 We have began the restore process, your VM will be automatically booted up again once completed. Control panel access will remain disabled whilst we monitor. https://status.iniz.com/incident/271292 Wed, 11 Oct 2023 22:20:00 -0000 https://status.iniz.com/incident/271292#44739a0809297dc27be4139f1eb35ea07de1ea9fbd04db46594458d637574777 It's likely the thermal shutdown and reboots caused data corruption, we have had the broken fans replaced and NVMe out of caution. We will now begin reinstalling the node. https://status.iniz.com/incident/271292 Wed, 11 Oct 2023 18:03:00 -0000 https://status.iniz.com/incident/271292#c6aadb902357f2e2a2fbeb7b4f77c2d727831b0d16ae612b83f4a175c9386cef As the server has gone down again we will proceed with replacing the hardware components and restore from last available backup we have. https://status.iniz.com/incident/271292 Wed, 11 Oct 2023 15:45:00 -0000 https://status.iniz.com/incident/271292#871ce594885fd54dec24c55eda543f622e8a7c41a13609a6c64e05d0af71513c This looks to be an issue with the NVMes, we have been able to reboot the node. We will likely need to replace the NVMe & reinstall the host node. We will attempt to copy the VM disks over. We recommend all clients take backups whilst still online. https://status.iniz.com/incident/271292 Wed, 11 Oct 2023 13:12:00 -0000 https://status.iniz.com/incident/271292#35202c507f46f8b834af94b99ebadf5c76a40aa5334996ef2ae810a2211453d3 NYCM2 is currently down, we are investigating the root cause. Updates to follow. https://status.iniz.com/incident/255722 Thu, 07 Sep 2023 11:26:00 -0000 https://status.iniz.com/incident/255722#aa8e3314670a179d9399711152a85455c9f0209048a311144686383beb5e5e18 The issue returned in the afternoon yesterday however we have been able to locate the cause and resolved this. The node has been stable for the last 12+ hours now. https://status.iniz.com/incident/255722 Wed, 06 Sep 2023 11:36:00 -0000 https://status.iniz.com/incident/255722#9682c8682c7f9824b5dc1180b241ba96cb8b373631e6b8ba2bcd8113be14526f There was issues with the LACP ports which have been resolved, this should fix the intermittent network issues. We will monitor this closely over the week. https://status.iniz.com/incident/255722 Wed, 06 Sep 2023 11:18:00 -0000 https://status.iniz.com/incident/255722#4086a9d481af2485e2bf4f281370a88c379a307428835e3a0fcbe4284bee9abd We have issued reboot on the node to update kernel and packages, all VMs should be back online within the next 15 minutes. https://status.iniz.com/incident/255722 Wed, 06 Sep 2023 10:00:00 -0000 https://status.iniz.com/incident/255722#5d6016d155aaf4af69fbd78158ce2d2675b49f19c448643ccde7ea2da700ac81 Over the last 48 hours we have noticed sporadic latency causing issues with response times on AMSSDVZ6 which we are investigating. https://status.iniz.com/incident/227239 Sun, 02 Jul 2023 17:52:12 +0000 https://status.iniz.com/incident/227239#1f2425c0637ee58755ec9e7b672226cae924444a588206f2fe15060f761071b8 Maintenance completed https://status.iniz.com/incident/227239 Sun, 02 Jul 2023 17:52:12 -0000 https://status.iniz.com/incident/227239#830d629e7b4e09f383e9e2801548a0795f4ada976cbe72f873a0400eeda7fb57 As per the email sent out on 26th Jun 2023 we are performing migration of our LA customers to a new DC which will result in several hours downtime. There are no changes to your IPv4 address and we will attempt to migrate your VM data over. https://status.iniz.com/incident/227239 Sun, 02 Jul 2023 14:54:41 -0000 https://status.iniz.com/incident/227239#830d629e7b4e09f383e9e2801548a0795f4ada976cbe72f873a0400eeda7fb57 As per the email sent out on 26th Jun 2023 we are performing migration of our LA customers to a new DC which will result in several hours downtime. There are no changes to your IPv4 address and we will attempt to migrate your VM data over. https://status.iniz.com/incident/227239 Sun, 02 Jul 2023 10:00:00 -0000 https://status.iniz.com/incident/227239#830d629e7b4e09f383e9e2801548a0795f4ada976cbe72f873a0400eeda7fb57 As per the email sent out on 26th Jun 2023 we are performing migration of our LA customers to a new DC which will result in several hours downtime. There are no changes to your IPv4 address and we will attempt to migrate your VM data over. https://status.iniz.com/incident/221904 Mon, 19 Jun 2023 15:20:00 -0000 https://status.iniz.com/incident/221904#f37167728c44c823b6fddc63a7f1d6928cb3dca020ff5c7c4923c27b0fa43e99 This has been completed. https://status.iniz.com/incident/221904 Mon, 19 Jun 2023 10:43:00 -0000 https://status.iniz.com/incident/221904#134bbe22672b3f108600101ab23a01e6a644cab5de7c46b806bc4aee889181b0 We have migrated >80% of the VMs from this node and look to complete by evening. https://status.iniz.com/incident/221904 Sun, 18 Jun 2023 22:00:00 -0000 https://status.iniz.com/incident/221904#84bd9764831f8323b03e314e602ff716da2dde9b920906f6d4b2deb0e304e1a9 We have noticed potential hardware failure and ssds failing as a result we have began swift migration of VMs from AMSVZ5 to other nodes. If you have a VM on this node you will see error when trying to access client area/control panel until the VM has been migrated which will take up to ~30 minutes depending on the disk space used. https://status.iniz.com/incident/183373 Thu, 16 Mar 2023 17:36:01 +0000 https://status.iniz.com/incident/183373#5863cba9644623f514fd00dae7c6fa9200ce0b53747c4d07d93d2edd50945fc6 Maintenance completed https://status.iniz.com/incident/183373 Thu, 16 Mar 2023 17:36:01 -0000 https://status.iniz.com/incident/183373#8d7033154929abd7297dc225ace71e8560f71e3f2bf683868134f575d1f2b316 Our DC will be relocating our nodes to different rack space within the DC, we expect each server to go offline for approx. 20 minutes. https://status.iniz.com/incident/183373 Thu, 16 Mar 2023 16:23:49 -0000 https://status.iniz.com/incident/183373#8d7033154929abd7297dc225ace71e8560f71e3f2bf683868134f575d1f2b316 Our DC will be relocating our nodes to different rack space within the DC, we expect each server to go offline for approx. 20 minutes. https://status.iniz.com/incident/183373 Thu, 16 Mar 2023 15:28:00 -0000 https://status.iniz.com/incident/183373#8d7033154929abd7297dc225ace71e8560f71e3f2bf683868134f575d1f2b316 Our DC will be relocating our nodes to different rack space within the DC, we expect each server to go offline for approx. 20 minutes. https://status.iniz.com/incident/183373 Thu, 16 Mar 2023 14:00:00 -0000 https://status.iniz.com/incident/183373#8d7033154929abd7297dc225ace71e8560f71e3f2bf683868134f575d1f2b316 Our DC will be relocating our nodes to different rack space within the DC, we expect each server to go offline for approx. 20 minutes. https://status.iniz.com/incident/169146 Sat, 28 Jan 2023 10:30:00 -0000 https://status.iniz.com/incident/169146#2ac2dbdd2f0b85f0bc1f9194f9359d967c55c9de04bfdcd6094c038e1c9027e9 These have been replaced and the node is booting back up. https://status.iniz.com/incident/169146 Sat, 28 Jan 2023 09:50:00 -0000 https://status.iniz.com/incident/169146#8784752ff23ceab36474185b89c300a446c8d3ccc0aa67cbea64d9a6b20b2096 We are investigating issues with AMSSDVZ3 which appears to be thermal shutdown due to 2 fan failures and are awaiting for the DC to replace these. https://status.iniz.com/incident/167368 Mon, 23 Jan 2023 16:06:00 -0000 https://status.iniz.com/incident/167368#2b9ba38a7892080c95fabffc438fa7b0e3298c3d8186107a7d904fd67e701486 Faulty PSU has been replaced and network is back online. https://status.iniz.com/incident/167368 Mon, 23 Jan 2023 15:30:00 -0000 https://status.iniz.com/incident/167368#6e74a784ba1339f4bea555528d297fec28b93ea758c86c608bec12e2aec9785a We're investigating downtime in Los Angeles which appears to be due to bad switch PSU. As we have spare on site we're awaiting for our DC to swap this over to debug further. https://status.iniz.com/incident/149419 Fri, 30 Dec 2022 22:00:51 -0000 https://status.iniz.com/incident/149419#ecf71b934cf13b62812d3b533e3483eb10a6d09367e246057584b657aabe6415 Planned hardware maintenance for all UK VM nodes starting at 9 PM GMT. https://status.iniz.com/incident/149419 Fri, 30 Dec 2022 21:30:00 +0000 https://status.iniz.com/incident/149419#3928ddf2e5e3268842079ebabb2b139fc50149168e46619e8bc744b250a96f77 Maintenance completed https://status.iniz.com/incident/149419 Fri, 30 Dec 2022 20:55:00 -0000 https://status.iniz.com/incident/149419#ecf71b934cf13b62812d3b533e3483eb10a6d09367e246057584b657aabe6415 Planned hardware maintenance for all UK VM nodes starting at 9 PM GMT. https://status.iniz.com/incident/139180 Sat, 26 Nov 2022 04:47:20 +0000 https://status.iniz.com/incident/139180#db3efbe2b2f9d7a7b55329bba8a09fbc1a78b4fd45a1e515a0fcd080dd8aa1a6 Maintenance completed https://status.iniz.com/incident/139180 Sat, 26 Nov 2022 04:47:20 -0000 https://status.iniz.com/incident/139180#d08de2e44642947fb90cf756af335ca147c20378468e2abcffa3968bf1519389 As per the notification sent via email you may track progress here of our UK datacentre migration affecting all UK VPS nodes. We have shut down the nodes at 00:30. https://status.iniz.com/incident/139180 Sat, 26 Nov 2022 04:10:35 -0000 https://status.iniz.com/incident/139180#d08de2e44642947fb90cf756af335ca147c20378468e2abcffa3968bf1519389 As per the notification sent via email you may track progress here of our UK datacentre migration affecting all UK VPS nodes. We have shut down the nodes at 00:30. https://status.iniz.com/incident/139180 Sat, 26 Nov 2022 00:25:00 -0000 https://status.iniz.com/incident/139180#d08de2e44642947fb90cf756af335ca147c20378468e2abcffa3968bf1519389 As per the notification sent via email you may track progress here of our UK datacentre migration affecting all UK VPS nodes. We have shut down the nodes at 00:30. https://status.iniz.com/incident/115401 Mon, 05 Sep 2022 03:45:00 -0000 https://status.iniz.com/incident/115401#545e6ae02c3fdc31620ec3b4ca0821d8f5fa4b9a1e09dc3aa24ca8925148082d Remaining OpenVZ node is back online and VMs are booting up. https://status.iniz.com/incident/115401 Mon, 05 Sep 2022 02:05:00 -0000 https://status.iniz.com/incident/115401#3462f1e3a1c33932100cd68a223c84bf2da4d37d74ad51a6137b7da0afd3b54a We have been able to restore all KVM nodes however OpenVZ LA-SSDVZ2 still has high temperature warnings. Our DC provided us the below unfortunate update that is causing these issues: "We are sorry for the inconvenience but one of our cooling units is down and is being repaired." https://status.iniz.com/incident/115401 Mon, 05 Sep 2022 00:15:00 -0000 https://status.iniz.com/incident/115401#215feecefafbeec52a2c0c9526f13f75e24d0259599d5445655c752861e33ce0 We're still awaiting updates from our DC regarding cooling which is not allowing us to keep the server(s) online due to high temperature warnings. This now affects multiple nodes. https://status.iniz.com/incident/115401 Sun, 04 Sep 2022 23:00:00 -0000 https://status.iniz.com/incident/115401#2f32208e3de283490dbe12a448c22858d7928308bc32b1b49eac20d201e71306 We are currently investigating issues on LA-SSD-VZ2 which has shut down due to high ambient temperatures. The node is currently being rebooted. We're awaiting updates from our DC. Currently there is a heatwave in Los Angeles causing sub optimal cooling. [1] https://ready.lacounty.gov/heat/ https://status.iniz.com/incident/112689 Thu, 25 Aug 2022 22:05:00 -0000 https://status.iniz.com/incident/112689#26338b4b4a6b5aa5af4ce197c3bd4b3dd2d6dc4b539c7fe37d09ca97640f85d1 This appears to be resolved now, we're awaiting RFO from our provider. https://status.iniz.com/incident/112689 Thu, 25 Aug 2022 21:45:00 -0000 https://status.iniz.com/incident/112689#91d7c6a0928c84962dba0dfc75b0fcbe38a8eefd1c2c5b0fa2613f5554794079 Our network provider is currently investigating issue in our London, UK location resulting in packet loss. https://status.iniz.com/incident/106021 Sun, 31 Jul 2022 21:02:00 -0000 https://status.iniz.com/incident/106021#c9ecb4bc919345fede3b6881f30417fb0bc2e708c7d913367ead70972ac2765b The node and VMs have successfully rebooted and we will continue to debug and monitor closely. https://status.iniz.com/incident/106021 Sun, 31 Jul 2022 20:55:00 -0000 https://status.iniz.com/incident/106021#3c847927d301c84ec423a3e3040ceb8f2f29614086186baac4544596f25cdcf4 We are currently investigating the AMS-SSD-VZ2 node which has gone down. https://status.iniz.com/incident/103776 Sat, 23 Jul 2022 15:32:00 -0000 https://status.iniz.com/incident/103776#e982a3970f7941aaf0475a00954c8355740a7856b208710162a1f7e308cd709d Node has been back up for the last ~40 minutes and all fan checks are now passing. We hope this is now resolved but will keep monitoring closely. https://status.iniz.com/incident/103776 Sat, 23 Jul 2022 14:38:00 -0000 https://status.iniz.com/incident/103776#f07801bcb01640232b684875bfacc281a3c5af7ab23f0c7e4fdc8384fc4787cd We are doing emergency maintenance on replacing the remaining defected fan. Server is currently being shutdown. https://status.iniz.com/incident/103776 Fri, 22 Jul 2022 10:23:00 -0000 https://status.iniz.com/incident/103776#ae6cf7240be6addc45425e9af8643849f4b7da209fefa68f8df7b94546b7a1d2 The server is online for past hour and we are continuing to monitor with only one failure and temperatures being low we can keep the server online. We will perform emergency maintenance when we are able to obtain additional replacement. https://status.iniz.com/incident/103776 Fri, 22 Jul 2022 09:13:00 -0000 https://status.iniz.com/incident/103776#842df893a0e3eb734ecb42bb12fbed5811a99f38645311e9a479654bddd3573f The affected parts were replaced however we are still having issues with it now giving warnings for other fans, we are investigating further. https://status.iniz.com/incident/103776 Thu, 21 Jul 2022 20:37:00 -0000 https://status.iniz.com/incident/103776#df57fdb56be56930309b00bb5402e773dcf44c34c61c1f8ed9d770d968977bc5 ---Email sent to clients--- Unfortunately, the system shutdown this evening due to two system fans failing (2/6) and we have just got it to boot temporarily though we suspect it may shutdown again as the HP node won't allow 2 or more fan failures.  Our datacentre has informed us the next available engineer will be tomorrow morning, as we always keep spares on site we hope to have these replaced then. We wouild also like to request clients to make backups immediately whilst we have gotten the server online and as always ensure they make their own backups on a regular basis. As of writing this email, the node was able to boot again though its likely it may shutdown again due to the fan failure. We will post any updates on our status page and no further emails will be sent: https://status.iniz.com/incident/103776 ---Email sent to clients--- Control panel access will remain disabled until tomorrow. https://status.iniz.com/incident/103776 Thu, 21 Jul 2022 19:12:00 -0000 https://status.iniz.com/incident/103776#e5dc5ce9099f813823b13417ede46e8c84b41e60e9736258e2e09a85bbb04892 At this time unfortunately we have no further updates, an engineer is scheduled to replace the affected fans tomorrow morning however the system is unable to boot due to 2 fans being down. We're looking at way to bring system up as safely as possible temporarily if possible. https://status.iniz.com/incident/103776 Thu, 21 Jul 2022 17:55:00 -0000 https://status.iniz.com/incident/103776#0693196e10e3d1fe945b0d6953bb017787eb3e3da249b3d217b7d3752e3bed92 We are investigating issues with the AMS-SSDVZ-3 node currently which looks likely due to high temperatures and potential fan failure. https://status.iniz.com/incident/101578 Wed, 13 Jul 2022 00:32:00 -0000 https://status.iniz.com/incident/101578#f9d35eb9562c6d9901b8840e4831386a9a38877286e0873de6041161ad870158 Our provider in UK has performed unscheduled network maintenance resulting in temporary routing issues which should now be resolved. https://status.iniz.com/incident/71220 Wed, 16 Feb 2022 15:03:00 -0000 https://status.iniz.com/incident/71220#c4d5c257c46d1a00f5edd70f5902748f5f3f2c426112dbefcedc39095ac7ab36 We received the below and hope the issue is now permanently resolved. --- An update from our network engineer, the issue has been resolved, and it should no longer be a problem. One of our core routers had a problem with internal routing that led us to schedule a system failover, but due to the high level of connections and traffic, we had to move traffic first to clear resources to make a move. --- https://status.iniz.com/incident/71220 Wed, 16 Feb 2022 14:40:00 -0000 https://status.iniz.com/incident/71220#dfaab2d8b633e8b9227ddad2f7bbfe7df32b76590b38b1b3f0d2ebd2e632cd68 This now appears resolved, the downtime and time to resolve issues twice in such a short period is of course unacceptable and we are now seeking further explanation. https://status.iniz.com/incident/71220 Wed, 16 Feb 2022 12:45:00 -0000 https://status.iniz.com/incident/71220#d37e1641433ef9ed5081e0f7951642a08605b0e92bf480882b3bb6ae8bff3743 The issue has been escalated to the senior network admin but we are yet to receive any further updates. https://status.iniz.com/incident/71220 Wed, 16 Feb 2022 11:00:00 -0000 https://status.iniz.com/incident/71220#7ed3a26a22a32427bf92f00e1658d78abcafe78ef1a2834127a019df8d88da2f Our IPv4 BGP is down again in Ashburn and looks to be issue from our provider end again, we are awaiting updates from them. Servers are still online/accessible via IPv6. https://status.iniz.com/incident/69976 Wed, 09 Feb 2022 02:05:00 -0000 https://status.iniz.com/incident/69976#dc5f91db9fb09d6ad720c12a9ed14fbaba9e7c0b4a1872fb71a9b69809877594 We received an update this was escalated to senior network team and has now been resolved. https://status.iniz.com/incident/69976 Wed, 09 Feb 2022 00:13:00 -0000 https://status.iniz.com/incident/69976#9c3e617bbf52263e6842eb96385a80ff39e277b6427b1a99761d338534506fac We can confirm no servers are down and IPv6 connectivity is working as normal. Awaiting updates from network provider regarding IPv4. https://status.iniz.com/incident/69976 Tue, 08 Feb 2022 23:58:00 -0000 https://status.iniz.com/incident/69976#37ef219daf5d733a88590600e2234b91265249d15f1003a54638d2dadb4cc035 We have been alerted to our BGP session going down in Ashburn, resulting in our subnets to stop routing. We have not made any changes on our end and are awaiting updates from our network provider. https://status.iniz.com/incident/63876 Wed, 29 Dec 2021 22:29:00 -0000 https://status.iniz.com/incident/63876#ca3b0484f099434e793fcf0f57af83a856bb14bae3f45ac96accb742aaba93fa We continue to process all requests same day and have had positive feedback from clients who've been able to recover from the backups we provided. Please ensure all requests are made by 7th Jan 2022. After that we will destroy all data held from the previous node. https://status.iniz.com/incident/63876 Wed, 29 Dec 2021 00:02:00 -0000 https://status.iniz.com/incident/63876#a325730c591cc4b92698f991e60e7afbd4e4118000c0e84b32268460a6364a50 We have been working all evening ensuring clients get a copy of the available data quickly and have completed 95%+ of the requests that came through. Some clients have posted positive feedback on state of the copy and were able to recover important data. https://status.iniz.com/incident/63876 Tue, 28 Dec 2021 17:11:00 -0000 https://status.iniz.com/incident/63876#44cb30c979dc72b46cd676a34fc43699f28c91513aed65c58a1741c834132220 Unfortunately, the restore attempts have been unsuccessful for your affected VPS(s) on the AMS-SSDVZ-5 node.   > What you should do now: - We have set up a new node and you may reinstall with a new OS from the control panel now, your VM will show offline in the client area and you must reinstall. Attempting to boot will not work.   > Partial (corrupt) data is available: - The data we were able to retrieve is partially corrupted and so VMs are not successfully booting, our test VMs from the node have partial data and we believe most (but not all) clients could possibly retrieve some data that we were able to copy over. To request a copy of this (if we have it available), please open a ticket. Please be patient as we go through the requests as we will need to mount & compress the VM HDDs.  It is very rare and unfortunate that this has occured and we have not previously had such fatal failure ever. This is a friendly reminder that you must retain your own backups at all times, if you are hosting critical data it is well advised to have multiple off site recovery points. Majority of the clients on this node have been with us for several years and we have emailed on numerous occasions (eg. OpenVZ. 6>7 and HDD>SSD migrations) that you must retain your own and that we do not take backups.  We thank you for being patient and apologise sincerely for the inconvenience this may have caused.  https://status.iniz.com/incident/63876 Tue, 28 Dec 2021 15:59:00 -0000 https://status.iniz.com/incident/63876#e081e633e42c28b0dce972285e74a982a5952af5da65c725043b0d8c9002db33 We have began migrating data we recovered yesterday to the new node, we will then attempt to restart the VMs. https://status.iniz.com/incident/63876 Tue, 28 Dec 2021 14:42:00 -0000 https://status.iniz.com/incident/63876#baa0baf3433e7f3b4ee2ee8b1812265241459bef4ca29920c99c71d41513dfe9 The failed drives have been replaced and we are now reinstalling the node. https://status.iniz.com/incident/63876 Mon, 27 Dec 2021 21:14:00 -0000 https://status.iniz.com/incident/63876#2c7cf55c94dd0b360f42935d742495234d0c8f6025f321c8ac2c875875eee4a2 We have been able to access some data and recover approx. 2TB. We are unsure of the state of the ploop hdd of your VM until we begin restoring tomorrow evening. As a result we have paused creating new VMs, we hope that most data is in recoverable state to bring your VM back online. If you have your own backups and would like to get back online ASAP, then please simply submit a ticket to have a new VM created with the same IPs. Once again we apologise for any inconvenience this may have caused and thank you for your patience. https://status.iniz.com/incident/63876 Mon, 27 Dec 2021 14:18:00 -0000 https://status.iniz.com/incident/63876#99cef9dab4f7d5a504ca5be3be322aab2d44fd34b2ae45b05de4303b8956f216 We are attempting to recover data from the array, we may be successful. We have paused creating new VMs. If you have your own backups and need to be online ASAP then please submit ticket so we can create new VM (same IP). https://status.iniz.com/incident/63876 Mon, 27 Dec 2021 10:31:00 -0000 https://status.iniz.com/incident/63876#f9076486d5a2c625362584ccbcf4e14cc305227a55b881e3eb5ddf568b8d1556 We have began creating new VMs and emailing clients new details, we believe the data is not recoverable. Should this change tomorrow, we will inform clients immediately however you should begin plans to restore from your own available backups. At the same time we will also be extending all affected VMs due dates by +1 month. This is our first major incident in several years, we apologise for any inconvenience this may have caused. https://status.iniz.com/incident/63876 Mon, 27 Dec 2021 08:30:00 -0000 https://status.iniz.com/incident/63876#b0dc2d2e86382e233342380957de4faf2aa0440b5d7bf742c8f45cd31e90107e We are currently investigating two drive failure on the same RAID10 span and we believe it will mean the data is lost. Unfortunately due to COVID restrictions and holidays, our datacentre has informed us no engineer is available to investigate on site and replace the failed drives till tomorrow. This is the only node down in NL.