The problem started approximately 05:14 CET when the connection from the control-plane placed in HEL DCs to FSN DCs was interrupted. Our loadbalancer didn’t have all the nodes configured as backend, so when the router is moved across the cluster automatically in case of failure, HAproxy didn’t know about that. The last node was placed in FSN DC but because of that network connectivity problem, OpenShift moved it away to a new node, which was not in this configuration.
A fix has been implemented and we keep closely monitoring the systems.
Posted Nov 25, 2024 - 09:16 CET
Update
We are continuing to investigate this issue.
Posted Nov 25, 2024 - 08:37 CET
Investigating
We are currently experiencing a major service disruption due to an outage with our data center operator, Hetzner. Our team is actively working both internally and in collaboration with Hetzner to resolve the issue as quickly as possible. We apologize for the inconvenience and appreciate your patience.
Posted Nov 25, 2024 - 08:37 CET
This incident affected: Appmixer Hosted (Studio, Engine, API, Backoffice), Appmixer Connector Repository, Appmixer Image Registry, Authentication Hub, and Signup.