This article is contributed. See the original author and article here.
Final Update: Sunday, 20 February 2022 16:57 UTC
We’ve confirmed that all systems are back to normal as of 02/20, 16:52 UTC. Our logs show the incident started on 02/20, 07:34 UTC and that during the duration of incident customers in South UK using Azure Log Analytics, Log Search Alerts & Azure Sentinels services may have experienced intermittent log data gaps and/or incorrect alert activation.
We’ve confirmed that all systems are back to normal as of 02/20, 16:52 UTC. Our logs show the incident started on 02/20, 07:34 UTC and that during the duration of incident customers in South UK using Azure Log Analytics, Log Search Alerts & Azure Sentinels services may have experienced intermittent log data gaps and/or incorrect alert activation.
- Root Cause: The failure was due to a recent deployment to one of the backend instances of our service to become unhealthy leading to a backlog of data to be processed.
- Incident Timeline: 02/20, 07:34 UTC through 02/20, 16:52 UTC
-chandar
Update: Sunday, 20 February 2022 16:25 UTC
Root cause has been isolated to an outage in a backend system which caused ingested data to back up and not be processed. To address this issue, the backend team has rolled back the deployment Azure Log Analytics is now working as expected , though ingestion data is still backed up and data sent to the system which got stuck. Some customers using Azure Log Analytics, Log Search Alerts & Azure Sentinels services may experience intermittent log data gaps and/or incorrect alert activation and/or latency. We estimate 2 hours before all log data gaps and/or incorrect alert activation and/or latency is addressed.
Root cause has been isolated to an outage in a backend system which caused ingested data to back up and not be processed. To address this issue, the backend team has rolled back the deployment Azure Log Analytics is now working as expected , though ingestion data is still backed up and data sent to the system which got stuck. Some customers using Azure Log Analytics, Log Search Alerts & Azure Sentinels services may experience intermittent log data gaps and/or incorrect alert activation and/or latency. We estimate 2 hours before all log data gaps and/or incorrect alert activation and/or latency is addressed.
- Work Around: none
- Next Update: Before 02/20 18:30 UTC
Update: Sunday, 20 February 2022 12:30 UTC
We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers using Azure Log Analytics, Log Search Alerts & Azure Sentinels services in UK-South region continue to experience intermittent log data gaps and/or incorrect alert activation and/or latency. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02/20 07:34 UTC. We currently have no estimate for resolution.
We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers using Azure Log Analytics, Log Search Alerts & Azure Sentinels services in UK-South region continue to experience intermittent log data gaps and/or incorrect alert activation and/or latency. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02/20 07:34 UTC. We currently have no estimate for resolution.
- Work Around: none
- Next Update: Before 02/20 16:30 UTC
Initial Update: Sunday, 20 February 2022 09:55 UTC
We are aware of issues within Log Analytics and are actively investigating. Azure Log Analytics, Log Search Alerts & Azure Sentinels customers using workspaces in UK-South-0 may experience intermittent log data gaps and/or incorrect alert activation and/or latency .
-Surya
We are aware of issues within Log Analytics and are actively investigating. Azure Log Analytics, Log Search Alerts & Azure Sentinels customers using workspaces in UK-South-0 may experience intermittent log data gaps and/or incorrect alert activation and/or latency .
- Work Around: none
- Next Update: Before 02/20 13:00 UTC
-Surya
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.
Recent Comments