Impacted services: pbs001.ede
[13:41] Since yesterday morning we are experiencing an increased amount of disk health errors and disconnections. The issue is being investigated.
[13:50] An engineer will be onsite to perform emergency maintenance. The machine will temporarily not be available to create back-ups.
[14:19] The machine has been powered-off and taken into the workshop, we're investigating what the possible cause could be.
[14:37] All cables related to disk connection have been reseated, we will verify the behavior in the workshop.
[15:28] The machine has been racked again and will be closely monitored.
We expect that the machine will be available for users again before 16:00.
[15:58] The PBS is available for users again.
We have held open logs for the past 30 minutes which produced 0 errors, we deem the situation stable for now.
The machine will be closely monitored to ensure the problem does not return.
Re-opening incident:
[21:09] An engineer has deemed that the issues from this afternoon have returned.
To ensure integrity of the system it will temporarily become unavailable again while we perform emergency maintenance again.
An engineer will be on-route to the datacenter shortly. Updates will be made.
[21:30] An engineer has arrived on-site and will perform maintenance, from this moment on the pbs will temporarily not be available.
[22:11] The engineer has decided to temporarily change the datastores from read-only to offline in order to export the datapool and re-attach it.
Because the datastore is offline there will be a 'gap' in your graphs, which Proxmox may complain about.
[22:35] At this moment the data-pool is recovering, due to our 3-mirror design back-ups can be continued whilst one of the disks is rebalancing.
We will keep a close eye on the machine and monitor the situation.
Our most sincere apologies for the inconvenience that this incident may have caused.
