Resolved -
This incident has been resolved.
Nov 25, 12:03 CET
Monitoring -
The problem started approximately 05:14 CET when the connection from the control-plane placed in HEL DCs to FSN DCs was interrupted. Our loadbalancer didn’t have all the nodes configured as backend, so when the router is moved across the cluster automatically in case of failure, HAproxy didn’t know about that. The last node was placed in FSN DC but because of that network connectivity problem, OpenShift moved it away to a new node, which was not in this configuration.
A fix has been implemented and we keep closely monitoring the systems.
Nov 25, 09:16 CET
Update -
We are continuing to investigate this issue.
Nov 25, 08:37 CET
Investigating -
We are currently experiencing a major service disruption due to an outage with our data center operator, Hetzner. Our team is actively working both internally and in collaboration with Hetzner to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience.
Nov 25, 08:37 CET