Breaking changes November 2023

by | Nov 23, 2023 | Technology | 0 comments

This article is contributed. See the original author and article here.

Description


Report CreateFile operations during Event Grid ingestion from Adlsv2 as errors


 


When uploading a file with Azure Data Lake SDK, there is an initial CreateFile event with size of 0.


Another FlushAndClose event is sent to Azure Data Explorer when setting the close parameter to ‘true’ on the SDK’s file upload option.


This event indicates that the final update has been made and the file stream is ready for ingestion.


 


Current Behavior:


Azure Data Explorer ignores the CreateFile event during data ingestion and no monitoring of these errors is reflected to users.


 


Expected behavior:


Azure Data Explorer will treat CreateFile events as an error of empty blob.


As a result, users are expected to get the following permanent errors for these events:



  • Failure ingestion metrics:

    • “IngestionResult” of permanent bad request

    • “BlobsDropped” from the relevant event grid data connection.

    • “EventsDropped” from the relevant event grid data connection.



  • Failed ingestion logs in case of turning on the failed ingestion diagnostic logs


 


Required Change:


Filter out CreateFile events from the event grid subscription.


This filtering reduces the traffic coming from Event Grid and optimizes the ingestion of events into Azure Data Explorer.


You can read more about how to use the SDK correctly and avoid empty file errors here.


 


Schedule & plan 


Step 1: Existing clusters which do not use the functionality today will get the change immediately.


Step 2: Clusters created after end of December 2023 will get the change.


Step 3: Current flow users as well as new clusters created until end of December 2023 will receive the changes after end of February 2024.


 



  1. Deprecating the metric “Events Processed (for Event/IoT Hubs)”


 


This metric represents the total number of events read from Event Hubs/ IoT hub and processed by the cluster. These events can be split by the status: Received, Rejected, Processed.


 


Required Change


Users can use the metrics “Event received”, “Events processed” and “Event dropped” to get the number of events that were received, processed, or dropped from each data connection respectively.


You can read more about these metrics here.


To ensure continuity of monitoring, please move any automated process that uses the old metric to use the new metrics instead.


 


Schedule & plan 


The changes above are planned to take place at the end of February 2024.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Submit a Comment