HelloID login currently not possible for some users

Incident Report for HelloID

Postmortem

Post Mortem incident 25th of January

On Wednesday the 25th a major outage occurred on the HelloID platform. The outage caused long loading times, multiple redirects or failures to load pages.

The outage was caused by an issue at one of our cloud partners Microsoft. A detailed description of the issue can be found at: Azure status history | Microsoft Azure

How did we respond?

Tools4ever engineers started to investigate the issue at 6.15 AM UTC. This process was hindered due to loading problems with the administrative interface of the Microsoft cloud environment. Several communication issues between HelloID services were identified and mitigation actions were implemented. At 8 AM UTC Microsoft confirmed that they were experiencing a major incident causing communication issues between Microsoft cloud services.

At 10 AM UTC most services were restored. However several databases remained unavailable until 13:45 UTC due to the connection issue in the Microsoft datacenters.

At 13:45 UTC all issues were resolved.

Posted Jan 31, 2023 - 15:08 UTC

Resolved

This incident has been resolved.
We keep monitoring the situation.

Posted Jan 26, 2023 - 09:34 UTC

Monitoring

It seems that the remaining services are restored and recovered. We are monitoring the situation.

Please reach out to support at your local office if something is not recovered for your service.

Posted Jan 25, 2023 - 14:30 UTC

Update

Most of the HelloID services recovered, but some are still troublesome. Microsoft is working to recover the impacted services.

A part from https://status.azure.com/en-us/status/history/ :

Mitigation: We identified a recent change to WAN as the underlying cause and have rolled back this change. Networking telemetry shows recovery from 09:00 UTC onwards across all regions and services, with the final networking equipment recovering at 09:35 UTC. Most impacted Microsoft services automatically recovered once network connectivity was restored, and we worked to recover the remaining impacted services.

Go to the URL for the full article.

Posted Jan 25, 2023 - 13:32 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jan 25, 2023 - 08:18 UTC

Identified

Microsoft 365 has an outage. It seems this causes problems for HelloID.
We keep investigating.

Posted Jan 25, 2023 - 08:18 UTC

Investigating

Currently, it is not possible to log into HelloID for some users. We are investigating the issue.

Posted Jan 25, 2023 - 07:14 UTC

This incident affected: HelloID West Europe (Access Management), HelloID West America (Access Management), and HelloID Training (Access Management).