Understanding Pipeline Failures and Error Handling 2

by | Apr 29, 2021 | Technology | 0 comments

This article is contributed. See the original author and article here.

In our previous installment of Understanding Pipeline Failures and Error Handling, we introduced the difference between TRY-CATCH logic and DO-IF-ELSE logic.


 


Just a memory refresher, DO-IF-ELSE logic defines both Upon_Success and Upon_Failure paths


Understanding Error Message 04 WillFail Definition.png


 If we execute this pipeline, with WillFail activity failing, the whole pipeline will be marked as Failed, as expected


Understanding Error Message 01 IF-ELSE.png


 


Sometimes users may want a more elegant error handling in their pipeline: despite branching activity (WillFail in this case) failed, I do not want to mark the whole pipeline as Failed, maybe because



  • I do not want to block downstream dependent tumbling windows, in case of tumbling window dependency

  • branching activity is used to look up a control table, and sometimes it is expected to return empty results and throw an error


To implement the desired behavior, customers may consider DO-IF-SKIP-ELSE block: adding a dummy UponSkipped to the end of the UponSuccess path, like this


Understanding Error Message 03 MarkSuccess Definition.png


 


If we run this pipeline, WillFail will still fail, but the pipeline will be marked as SUCCESS


Understanding Error Message 05 IF-ELSE-Success.png


 


In the monitoring view, you can see the difference a dummy UponSkipped activity can make. Notice that PipelineErrorHandlingSuccess is defined with the addition dummy activity, while PipeErrorHandling is not. (And the additional pipeline duration comes from the extra 5 seconds in the dummy wait activity)


Understanding Error Message 02 Run Status.png


 


Technical reasons for the difference is that, Azure Data Factory defines pipeline success and failures as follows:



  • Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead

  • Pipeline result is success if and only if all leaves succeed


 


Here is an expanded table summarizing the difference:


 





























Approach Error Handling Defines When Activity Succeeds When Activity Fails
TRY-CATCH Only Upon Failure path Pipeline shows Success Pipeline shows Success
DO-IF-ELSE

Upon Failure paths


+


Upon Success path


 Pipeline shows Success Pipeline shows Failure
DO-IF-SKIP-ELSE

Upon Failure paths


+


Upon Success path (with a dummy Upon Skipped activity at the end)


 


 Pipeline shows Success Pipeline shows Success

 

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Submit a Comment