This article is contributed. See the original author and article here.
Another day another case.
It took me a while to follow this step by step by using the new Synapse APIs. So this post has the intention to make it easier by example.
You will need:
- A Synapse workspace
- An ADF pipeline that you want to start from Rest API.
Doc references:
https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough
https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/createpipelinerun
That is my pipeline code. It is a simple one it just creates Spark Database using a notebook.
%%spark
spark.sql("CREATE DATABASE IF NOT EXISTS DB_example")
As you can see in figure 1. That is my pipeline:
Figure 1 Pipeline
The name of my pipeline is User_not_test.
I can run successfully this pipeline from Synapse Studio. But I want to run it from the Rest API, actually, that is the post idea.
Step by Step
The first step consists in using this documentation to register my pipeline/workspace as an application:
(https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough)
$subscriptionId = "{azure-subscription-id}"
$resourceGroupName = "{resource-group-name}"
# Authenticate to a specific Azure subscription.
Connect-AzAccount -SubscriptionId $subscriptionId
# Password for the service principal
$pwd = "{service-principal-password}"
$secureStringPassword = ConvertTo-SecureString -String $pwd -AsPlainText -Force
# Create a new Azure AD application
$azureAdApplication = New-AzADApplication `
-DisplayName "My Azure Monitor" `
-HomePage "https://localhost/azure-monitor" `
-IdentifierUris "https://localhost/azure-monitor" `
-Password $secureStringPassword
# Create a new service principal associated with the designated application
New-AzADServicePrincipal -ApplicationId $azureAdApplication.ApplicationId
# Assign Reader role to the newly created service principal
New-AzRoleAssignment -RoleDefinitionName Reader `
-ServicePrincipalName $azureAdApplication.ApplicationId.Guid
Fill the gasp. For example, suppose my workspace was named as: synapseworkspace_Demo. So this part of the script you should add your workspace name as my example: https://synapseworkspace_Demo.dev.azuresynapse.net
Take note of the password that you create and define here: $pwd = “{service-principal-password}”
# Create a new Azure AD application
$azureAdApplication = New-AzADApplication `
-DisplayName "APP_synapseworkspace" `
-HomePage "https://synapseworkspace_Demo.dev.azuresynapse.net" `
-IdentifierUris "https://synapseworkspace_Demo.dev.azuresynapse.net" `
-Password $secureStringPassword
After that, you will execute some steps to actually invoke the API. The example is described here:
https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api
But the Synapse APIs are here:
https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/createpipelinerun
My script will run with the following:
https://synapseworkspace_Demo.dev.azuresynapse.net/pipelines/User_not_test/createRun?api-version=2018-06-01
Do not forget to add the password that you defined:
$pwd = “Password that you defined on the previous step”
$pwd = "Password that you defined on the previous step"
$azureAdApplication = Get-AzADApplication -IdentifierUri "https://YourWorkspaceName.dev.azuresynapse.net"
$subscription = Get-AzSubscription -SubscriptionId $subscriptionId
$clientId = $azureAdApplication.ApplicationId.Guid
$tenantId1 = $subscription.TenantId
$authUrl = "https://login.microsoftonline.com/${tenantId1}/oauth2/token"
$cred = New-Object -TypeName Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential -ArgumentList ($clientId, $pwd)
$AuthContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]$authUrl
$result = $AuthContext.AcquireTokenAsync("https://dev.azuresynapse.net", $cred).GetAwaiter().GetResult()
# Build an array of HTTP header values
$authHeader = @{
'Content-Type'='application/json'
'Accept'='application/json'
'Authorization'=$result.CreateAuthorizationHeader()
}
$request = "https://YourWorkspaceName.dev.azuresynapse.net/pipelines/YourPipelineName/createRun?api-version=2018-06-01"
$body = @"
{
"name": "YourWorkspaceName",
"location": "Region",
"properties": {},
"identity": {
"type": "SystemAssigned"
}
}
"@
$response = Invoke-RestMethod -Method POST -Uri $request -Header $authHeader -Body $body
$response | ConvertTo-Json
$runId = $response.runId
If you face this error:
Invoke-RestMethod: {“error”:{“code”:”Unauthorized”,”message”:”The principal ‘some number display’ does not have the necessary permissions to perform this operation. “}}
Go back to you synapse studio -> open Monitoring -> access control and be sure of 2 things:
1) The user that will start the rest API needs Workspace admin permission
2)The APP that you register needs workspace admin permissions and to satisfy this requisite: Copy the number displayed on the error and add the permission like figure 2:
Figure 2 Permission
Note: Use the number of the principal service id, not the APP name to get this permission done.
You could also monitor the pipeline execution adding this piece of code:
while ($True) {
$response = Invoke-RestMethod -Method GET -Uri "https://YourWorkspaceName.dev.azuresynapse.net/pipelineruns/${runId}?api-version=2018-06-01" -Header $authHeader
Write-Host "Pipeline run status: " $response.Status -foregroundcolor "Yellow"
if ($response.Status -eq "InProgress") {
Start-Sleep -Seconds 15
}
else {
$response | ConvertTo-Json
break
}
}
Here is the execution – Figure 4:
Figure 4 Execution
Thanks, Dan Rosales for help during this process.
That is it!
Liliam
UK Engineer
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.
Recent Comments