Unleash the power of predictive analytics in Azure Synapse with machine learning and AI

by Contributed | Dec 8, 2020 | Azure, Microsoft, Technology

This article is contributed. See the original author and article here.

We are really excited to introduce the preview of new machine learning experiences in Azure Synapse Analytics, to make it easier for data professionals to enrich data and build predictive analytics solutions.

AI and machine learning is an important aspect of any analytics solution. By integrating Azure Synapse Analytics with Azure Machine Learning and Azure Cognitive Services, we are bringing together the best of two worlds, to empower data professionals with the power of predictive analytics and AI.  Data engineers working in Azure Synapse can access models in Azure Machine Learning’s central model registry, created by data scientists. Data engineers can also build models with ease in Azure Synapse, using the code-free automated ML powered by Azure Machine Learning and use these models to enrich data.

Linking workspaces to enable collaboration between Data professionals and ML professionals

Linked services can be created to enable seamless collaboration across an Azure Synapse and an Azure Machine Learning workspace. Linked workspaces allow data professionals in Synapse to leverage new machine learning experiences aiming to make it easier to collaborate across Synapse and Azure ML.

Create the Azure Machine Linked service in your Synapse workspace

Seamlessly access Azure Machine Learning models from Synapse

Data professionals working in Azure Synapse can collaborate seamlessly with ML professionals who create models in Azure Machine Learning. These models can be shared and deployed directly in Azure Synapse for enrichment of data.

In the model scoring wizard, enrich with an existing model

By supporting the portable ONNX model format, users can bring a variety of models to Synapse for performant batch scoring, right where the data lives. This removes the need for data movement and ensures that the data remains within the security boundaries defined by Azure Synapse. Columns containing predicted values can easily be appended to the original views and tables that are used to populate your Power BI reports.

Enrich data with Azure Cognitive Services pre-trained models

Fully integrated data enrichment capabilities powered by Azure Cognitive Services allow Synapse users to enrich data and gain insights by leveraging state of the art pre-trained AI models. The first two models available through the Synapse workspace are Text Analytics (Sentiment Analysis) and Anomaly detector. In the future you’ll see more pre-trained models available for use.

Leverage Azure Cognitive Services in Azure Synapse for sentiment analysis

Leverage Azure Cognitive Services in Azure Synapse for Anomaly detection

Train models in Synapse using Automated ML powered by Azure Machine Learning

Data professionals can also build models with ease in Azure Synapse, using code-free automated ML powered by Azure Machine Learning. These Automated ML runs will be executed on Synapse serverless Apache Spark pools and tracked in the Azure Machine Learning service.

Select the task type for your Automated ML run in Azure Synapse

All the machine learning experiences in Azure Synapse produce code artifacts such as PySpark Notebooks or SQL scripts, that allow users of all skill levels to easily operationalize their work in data integration pipelines, to support end-to-end analytics flows from a single unified Synapse experience.

Get started today

We are expanding Azure Synapse to bring together the best in big data analytics and machine learning so you can leverage the full power of Azure. These new experiences in Synapse Studio will streamline the way data teams collaborate and build predictive analytics solutions. A large number of our customers are already taking advantage of predictive analytics solutions. Learn more about how you can get started on your journey with ML experiences in Azure Synapse by using the links provided below.

Resources:

AzureSynapse ML Docs overview: https://aka.ms/synapseMLDocs

Azure Synapse Automated ML tutorial: https://aka.ms/SynapseMLDocs_AutoML_tutorial

Azure Synapse model scoring tutorial: https://aka.ms/SynapseMLDocs_Scoring_Tutorial

Azure Synapse Cognitive Services tutorial: https://aka.ms/SynapseML_Docs_Cognitive_Services

Link Azure Synapse workspace to Azure ML workspace:https://aka.ms/SynapseMLDocs_link_AML

Azure Synapse TechCommunity: Check this blog daily to see a roundup of all the new tutorial blogs that will be posted for the next two weeks.

How to setup a Canarytoken and receive incident alerts on Azure Sentinel

by Contributed | Dec 8, 2020 | Azure, Microsoft, Technology

This article is contributed. See the original author and article here.

With Azure Sentinel you can receive all sorts of security telemetry, events, alerts, and incidents from many different and unique sources. Those sources can be firewall logs, security events, audit logs from identity and cloud platforms. In addition, you can create digital trip wires and send that data to Azure Sentinel. Ross Bevington first explained this concept for Azure Sentinel in “Creating digital tripwires with custom threat intelligence feeds for Azure Sentinel”. Today you can walkthrough and expand your threat detection capabilities in Azure Sentinel using Honey Tokens or in this case Canarytokens.

What is a Honey Token? A honey token is a digital artifact like a Word Document, Windows Folder, or JavaScript file that when opened or accessed will act as a digital trip wire and alert you to being used. When used the honey token might make a GET HTTP call to a public facing URL or IP. The so that an attacker would want to investigate and exfiltrate the artifact but also ensuring you reduce false positives from normal users. One way to do this is creating a separate folder from the normal directory structure. This could take the form of naming a Word document High Potential Accounts.docx. And then placing it in a Sales share but inside two more nested directories.

The other key is to make the digital artifact searchable or easily found, you want the attacker to see the token and access it. You can also sprinkle these honey tokens through out the network and in different use cases beyond. The key here is ensuring that the honey token is in a visible location and can directory searched upon by normal user credentials.

As with most things a balanced approach should be taken with honey token names and placement. Think through where in the cyber kill chain you want the digital trip wire, and ways to make the token enticing to an attacker but will also reduce false positives from normal employees and routines.

Honey Tokens are not a new concept but the following approach described to use a service called Canarytokens is a bit newer. Canarytokens is a free service provided by Thinkist that generates different types of tokens and provides the back end trip wire logging and recording. The service allows you to focus on the naming and placement specific to your industry and buisness rather then building a Public facing URL that logs and collects the tokens being tripped. Thinkist also has a paid service as well that includes many useful features.

In the below example you will walk through creating a free Canarytoken (honey token as described) but through a Canary service and use it to update Azure Sentinel when it is triggered.

To begin with you can deploy a Logic App Ingest-CanaryTokens here. The Logic App will act a listener and will provide a URL you can use in the Canarytoken generation.

To Deploy the Logic App fill in your Azure Sentinel Workspace ID and Key.

Once deployed go to the Logic App and in the Overview click on the blue link: See trigger history

Copy the URL from the following field: Callback url [POST]

With this LogicApp and a Callback listening URL you can now generate a Canarytoken.

To create the Canarytoken go to the following website: Canarytokens

1. Choose Microsoft Word Document


2. Fill out your email address and enter a <SPACE> and paste the Logic App Callback URL


3. In the final field enter a description, – see below

You will use description to also host your Entities for Azure Sentinel. You can use a comma as a separator between the entity information you want to capture upon tripping the wire.

Be sure to be descriptive to what ServerShare or OneDrive the Canarytoken will be placed. Because you will generate several different tokens the descriptive notes will come in the alert that is triggered ensuring you will be able to dive further on that Server or Service to investigate further activity of the attacker.

In this example you could use:

*EXAMPLE:

FS01,42.27.91.181,10.0.3.4,T:departmentssaleshipospecials,token placed on FS01 available to all corporate employees and vendors

     4. Once Completed click Create my Canarytoken

Check out the further use cases for the Canarytokens to be placed. Go ahead and Download your MS Word file.

Notice the file name that downloads is the Canarytoken id itself. This word document name really is not that compelling for an attacker to discover, exfiltrate, and investigate. You should rename the file immediately to something more compelling.

You want to grab the attention of the attacker searching for valuable information. Remember the overall arching goal for most attackers is obtaining key corporate data. The Canarytoken is helping alert to the violation of confidentiality, integrity and availability of key corporate data. Names like Project Moonshot placed in NextGeneration folder could help entice. Document name like High Potential Account List in a Sales team folder may also do the trick. Be creative to your industry and business as to what data could be valuable.

In this example we used White Glove Customer Accounts.docx

To make the document seem more legitimate you can use a website Mockaroo – Random Data Generator and API Mocking Tool | JSON / CSV / SQL / Excel to generate random and fictious data easily. Here you can create what appears to be a customer account list with account numbers and email addresses.

Once you fill out the fields you want go ahead and download a CSV sample by clicking Download Data green button. Open with Excel and be sure to manipulate the Rows and Columns to make it nicely formatted. With the table looking presentable copy the content in Excel and Open the Word Document Canarytoken and paste the content in and save the document.

You now have a Canarytoken that looks authentic and hopefully will not arouse the suspicion of the attacker but will be visible and entice them greatly to exfiltrate and open it. Continue to examine Mockaroo and the data you can generate it is a very easy to use and helpful tool.

Now find a home for the word document in a File Share on a File Server, or as an email attachment in your executives mailbox – again think back to the description you gave it and follow that to where it is placed so in the worst case you are attacked this can tip you off to where on your network to focus your investigation further in Azure Sentinel’s logs and events you are collecting.

To test this open the Word Document on your computer or on another server or computer with word. When Microsoft Word opens a .1 by .1 header and footer image with a open URL will execute a GET HTTP call to the appropriate CanaryToken endpoint you created earlier. Once this occurs you will receive an email with details like below.

Be sure to also check out the More info on this token here link, which will provide more geo information on the public ip that opened the document and also if it came off a known Tor browser or not.

You can also download a JSON or CSV file of the detailed information found in the Incidents generated when the Canarytoken was opened.

In addition to the email the Logic App listener will be invoked which will take the Incident Data and enrich it a little further and send it to Azure Sentinel into a custom logs table named CanaryTokens_CL.

Some of those enriched fields include geo information on the public ip address that triggered the Canarytoken. There is also parsed information from the memo field to include specifics around the Canarytokens placement in your environment and objectives and some logic to tell you if the canary was triggered on host. Finally string fields for URLs have been populated for you to review the management and history of the Canarytoken if you need to pivot from Azure Sentinel to the Canarytoken specifically while investigating.

You can now use Azure Sentinel to raise a High Priority incident and work the incident with case management. You can also correlate logs and data with other Azure Sentinel data collected further helping you investigate the incident.

An example Scheduled query rule in Azure Sentinel you can use following along this walkthrough. Step by step instructions Here

id: 27dda424-1dbe-4236-9dd5-c484b23111a5
name: Canarytoken Triggered
description: |
  'A Canarytoken has been triggered in your enviroment, this may be an early sign of attacker intent and activity, 
    please follow up with Azure Sentinel logs and incidents accordingly along with the Server this Canarytoken was hosted on.
    Reference: https://blog.thinkst.com/p/canarytokensorg-quick-free-detection.html'
severity: High
requiredDataConnectors:
  - connectorId: Custom
    dataTypes:
      - CanaryTokens_CL
queryFrequency: 15m
queryPeriod: 15m
triggerOperator: gt
triggerThreshold: 0
tactics:
  - Discovery
  - Collection
  - Exfiltration
relevantTechniques:
query: |
CanaryTokens_CL
| extend Canarydata = parse_csv(memo_s)
| extend CanaryHost = tostring(Canarydata[0]), CanaryPublicIP = tostring(Canarydata[1]), CanaryPrivateIP = tostring(Canarydata[2]), CanaryShare = tostring(Canarydata[3]), CanaryDescription = tostring(Canarydata[4])
| extend CanaryExcutedonHost = iif(CanaryPublicIP == src_ip_s, true, false)
| extend timestamp = TimeGenerated, IPCustomEntity = src_ip_s //,AccountCustomEntity = user_s, HostCustomEntity = computer_s
entityMappings:
  - entityType: IP
    fieldMappings:
      - identifier: Address
        columnName: IPCustomEntity

Once you have created the rule, open the Canarytoken word document one more time to generate an alert.

Within 15 minutes or so a new Azure Sentinel Incident for the Canarytoken being trigged will appear, your SOC can now use the Logs fed into Azure Sentinel to correlate and investigate further.

In addition the Investigate Graph is also populated with the Public IP Address of where this was triggered.

Please tweak the Custom Entities to your liking. Another way is to point where the Canarytoken was placed to bolster the pivot of the Investigation graph. The above alert sample parses the memo field you added early with commas when generating the initial Canarytoken.

In this article you learned about honey tokens and a Canary service and how to use Canarytokens in your environment and integrate the enriched alerts into Azure Sentinel raising awareness of a potential attacker and data exfiltration that may have occurred.

You have just scratched the surface with the concept of honey tokens. If you are interested in learning more in depth I highly recommend Chris Sander’s book Intrusion Detection Honeypots which is a excellent resource.

Special thanks to:

@Ofer Shezaf for reviewing this post

@Chris Sanders for inspiration and information on the topic of Honey Tokens

CI CD in Azure Synapse Analytics Part 1

by Contributed | Dec 7, 2020 | Azure, Microsoft, Technology

This article is contributed. See the original author and article here.

I was helping a friend earlier today with their Azure Synapse Studio CI / CD integration.  They had followed our Docs page Source control in Azure Synapse Studio and then they shared errors they were seeing in their release pipeline during deployment. 

We took a step back to discuss what they wanted to do, and it looked like they were too far in the weeds for ADO.  So I walked through creating an Azure DevOps Project, connecting Git to my Azure Synapse Studio, and then creating a branch and pushing some changes.  We’ll push changes in a follow up blog post.  Today we cover the basics.

First let’s navigate to Projects – Home (azure.com).  We will create a New Project and title it Azure Synapse Studio CI CD.  I’m going to mark this repo private because it’s just for us.

Now I will click on the Repos menu.

Next I will go to the bottom of the page.  I want to select Initialize main branch with a README or gitignore.  I will click Initialize.

At this point I have a Repo that is initialized.

Now we can connect this to our Azure Synapse Studio.  Let us travel over to https://web.azuresynapse.net/ and log into our Azure Synapse Studio.  After we login we need to navigate to the Manage screen.  If you are not on the Git configuration page, navigate there.  

Next we want to click on Set up code repository.  You can select Azure DevOps Git or GitHub.  For this blog we will be selecting Azure DevOps Git.  Then select your organizations Azure Active Directory tenant. *a quick side note, make sure the AAD account you are using to connect to Azure DevOps is the same account that has permissions to your Azure Synapse Studio workspace.

Then click Continue.

Select the Azure DevOps Account that our organization is using. The Project and Git repository name are the same, and are the Project Name we created earlier.

My collaboration branch is main, my Publish branch is workspace_publish, my Root folder is the default, I have checked import existing resources to repository.

As this is my initial commit I want to commit this to my main branch.

Then I click Apply.

*Another note: your company will have a DevOps environment, and specific rules on how you want things to connect.  If I’m doing anything that makes you scream from a developer philosophy, please find me on Twitter under BuckWoody_MSFT …. also don’t tell Buck I did this …. I’m not Buck.

At this point in time your Azure DevOps Git Rep should be connected.

If we go to our Azure DevOps Repo we should see that it is populated with objects from our Azure Synapse Studio.

Back in Azure Synapse Studio, we can navigate to the develop pane, create a new branch to ensure any changes I make will not be automatically deployed against my main version of Azure Synapse Studio.

This is what we will tackle next time.