SA Agent connection issues
Incident Report for HelloID
Postmortem

Service Automation Agent connection issues were a series of (combined) problems. The main problem lays in having every customer’s agent receive their information fast and just their information. To ensure this we’ve implemented an additional check that verifies only a single agent installation is connected. Although this greatly increases the stability and security; it unfortunately also shed light at some connection problems. As these problems only occurred occasionally this meant we required additional time to fix these problems. One of the most helpful fixes was making sure the customer’s agent always reconnects. As we’ve updated the installed agent clients and did various improvements on our own side, we expect things to behave in a more stable manner. Adding to these fixes and improvements there are now more insights in the agent connection status. As we continue to monitor how the agents perform; this is going to be of great help to prevent future outages.

Posted Nov 01, 2022 - 09:00 UTC

Resolved
After releasing the fix regarding agent connection issues we've monitored the situation closely.
The issue has been resolved.
Posted Oct 28, 2022 - 13:54 UTC
Update
We are continuing to work on a fix for this issue.
Posted Sep 23, 2022 - 14:26 UTC
Update
We will be performing an update on Monday, September 26 to further improve SA agent stability.
The update will take place at 08:00 UTC for WUS and at 18:00 UTC for WE.

Note: This update will also release a new agent-version!!
Posted Sep 23, 2022 - 14:19 UTC
Identified
Currently we are experiencing issues with the SA Agent connections. We have been able to identify the issue after extensive reproduction.
A few enhancements have already been rolled out, more are being tested as we speak en will be rolled out as soon as possible.
In the mean time, if you experience issue, a restart of the SA Agent service resolves the issue for the time being.
Posted Sep 21, 2022 - 08:00 UTC
This incident affected: HelloID West Europe (Service Automation) and HelloID West America (Service Automation).