Earlier today, we experienced a failure on one of our block storage arrays, which caused partial downtime and disconnection to some production environments.
The issue was related to a faulty array controller (CT1), which resulted in NVMe connection drops from affected nodes. As a result, the nodes mounted the filesystem in read-only mode, preventing further write operations.
To restore functionality, a reboot of the affected nodes was performed, allowing them to reconnect to the storage controller successfully.
The issue has now been fully resolved, and all impacted environments are back online. We are conducting a root cause investigation to determine the underlying reason for the controller failure.
We apologize for any inconvenience this may have caused and appreciate your patience. If you have any concerns or need further assistance, please reach out to our support team.