A Real Example: How to debug Azure Stream Analytics jobs locally using job diagram

by Scott Muniz | Sep 16, 2020 | Azure, Technology, Uncategorized

This article is contributed. See the original author and article here.

Azure Stream Analytics is a full managed service for real-time analytics. In addition to the experience in Azure portal we have developer tools which make development and debugging easier. This blog will introduce a new debugging feature in Azure Stream Analytics tools extension for Visual Studio Code. 

Have you ever faced a situation where your streaming job produces no result or unexpected results and you don’t know how to trouble shoot which parts go wrong? We are happy to announce the roll out of the newest debugging feature – job diagram debugging in Visual Studio Code extension for Azure Stream Analytics. This feature brings together job diagrams, metrics, diagnostic logs, and intermediate results to help you quickly isolate the source of a problem. You can not only test your query on your local machine but also connect to live sources such as Event Hub, IoT Hub. The job diagram helps you to understand how data flows between each step, and you can view the intermediate result set and metrics to debug issues. You can iterate fast because each local test of the job only takes seconds. 

This blog will use a real example to show you how to debug an Azure Stream Analytics job using job diagram in Visual Studio Code.  

Note 

This job diagram only shows the data and metrics for local testing in a single node on your own machine. It should not be used for performance and scalability tuning. 

Step 1 Install the tools 

• Install Visual Studio Code and Azure Stream Analytics tools extension for Visual Studio Code. 
• Install .NET core SDK and restart Visual Studio Code. 
• Sign-in Visual Studio Code using your Azure account. 

Step 2 Open your job in Visual Studio Code 

Please go to step 3 if you already have your Azure Stream Analytics project opened in Visual Studio Code.  

Open the Azure Stream Analytics job you want to debug on the Query Editor on Azure portal. Select from drop down Open in Visual Studio and choose Visual Studio Code. Then choose Open job in Visual Studio Code. The job will be exported as an Azure Stream Analytics project in Visual Studio Code. 

Step 3 Run job locally 

Since the credentials are already auto-populated, only thing you need to do is to open the script and Run locally. Make sure the input data is sending to your job input sources. 

Step 4 Debug using job diagram 

The job diagram shown on the right window of the editor shows how data flows from input sources, like Event Hub or IoT Hub, through multiple query steps to output sinks. You can also view the data as well as metrics of each query step in each intermediate result set to find the source of an issue. 

Now, let’s look at a real example job below. We have a job receiving stock quotes for different stocks. In the query there are filters on a few stocks but one output does not have any data. 

We run the job locally against live input stream from Event Hub. Through job diagram we can see that the step ‘msftquotes’ does not have data flowing in.    

To troubleshoot that, let’s zoom in the diagram and select the upstream step ‘typeconvertedquotes’ to see if there is any output. In the node it shows that there are 3135 output events. Also, from the Result tab below it is easy to find out that there are data been output from this step with symbol ‘MSFT’. 

Then we select the step ‘msftquotes’ and locate to the corresponding script to take a closer look.  

Now we find the root cause – there is a typo in the script, ‘%MSFT%’ is mistakenly typed as ‘%MSFA%’.  

MSFTQuotes AS ( 

    SELECT typeconvertedquotes.* FROM typeconvertedquotes 

    WHERE symbol like ‘%MSFA%’ 

    AND bidSize > 0 

), 

Let’s fix the typo, stop the job and run again. 

Look, the data is flowing into step ‘msftquotes’ and the other downstream steps. 

Other than checking the result for each step, you can also view logs and metrics for the job.  

Step 5 Submit to Azure

When local testing is done, submit the job to Azure to run in the cloud environment and further validate the job status in a distributed manner. 

Hope you find these new features helpful, and please let us know what capabilities you’re looking for with regard to job debugging in Azure Stream Analytics! 

Internet of Things – Event Learning Path (Learn to Build IoT Solutions on Microsoft Azure)

by Scott Muniz | Sep 16, 2020 | Azure, Technology, Uncategorized

This article is contributed. See the original author and article here.

The Internet of Things Event Learning Path is designed for Solution Architects, Business Decision Makers, and Development teams that are interested in building IoT Solutions with Azure Services. The content is comprised of 5 video based modules that approach topics ranging from IoT device connectivity, IoT data communication strategies, use of artificial intelligence at the edge, data processing considerations for IoT data, and IoT solutioning based on the Azure IoT reference architecture.

Each session includes a curated selection of associated modules from Microsoft Learn that can provide an interactive learning experience for the topics covered and may also contribute toward preparedness for the official AZ-220 IoT Developer Certification.

This content may be reused as-is across partner, field, and third party events or modified to suit custom audiences. The video resources and presentation decks are open-source and can be found on GitHub @ https://aka.ms/iotlp.  To pull down a local copy of all included slide decks and video presentations simply clone the GitHub repo with:

git clone https://github.com/microsoft/Internet-of-Things-Event-Learning-Path.git

We have packaged this content in a way that makes it very easy to reuse either as-is or to modify with your own personal touch.  Each session includes a 15 minute introduction session and a longer 45 minute deep dive.  To make these sessions your own, you can modify the existing presentations to your liking.  We recommend delivering in a full-hour format by presenting the 15 minute introductions in a live format where you can then switch to a playback of the 45 minute video recording.  In this way, you can genuinely introduce the content to your audience then allow for playback of the more advanced demos in a recorded fashion where they can’t fail (barring unforeseen ability to playback the video recordings)!  Let us know in the comments if you have any ideas or questions around using this content for your own events.

If you are interested in sharing or viewing the content right away, we have hosted the recordings on the IoT Developer YouTube where they can be viewed on-demand in a curated Internet of Things – Event Learning Path Playlist.  

Modules

IOT10: Connecting Your Physical Environment to a Digital World – A Roadmap to IoT Solutioning

With 80% of the world’s data collected in the last 2 years, it is estimated that there are currently 32 billion connected devices generating said data. Many organizations are looking to capitalize on this for the purposes of automation or estimation and require a starting point to do so. This session will share an IoT real world adoption scenario and how the team went about incorporating IoT Azure services.

IOT20: Deciphering Data – Optimizing Data Communication to Maximize Your ROI

Data collection by itself does not provide business values. IoT solutions must ingest, process, make decisions, and take actions to create value. This module focuses on data acquisition, data ingestion, and the data processing aspect of IoT solutions to maximize value from data.

As a device developer, you will learn about message types, approaches to serializing messages, the value of metadata and IoT Plug and Play to streamline data processing on the edge or in the cloud.

As a solution architect, you will learn about approaches to stream processing on the edge or in the cloud with Azure Stream Analytics, selecting the right storage based on the volume and value of data to balance performance and costs, as well as an introduction to IoT reporting with PowerBI.

IOT30: Adding Intelligence – Unlocking New Insights with AI & Machine Learning​

For many scenarios, the cloud is used as a way to process data and apply business logic with nearly limitless scale. However, processing data in the cloud is not always the optimal way to run computational workloads: either because of connectivity issues, legal concerns, or because you need to respond in near-real time with processing at the Edge.

In this session we dive into how Azure IoT Edge can help in this scenario. We will train a machine learning model in the cloud using the Microsoft AI Platform and deploy this model to an IoT Edge device using Azure IoT Hub.

At the end, you will understand how to develop and deploy AI & Machine Learning workloads at the Edge.

IOT40: Big Data 2.0 IoT as your New Operational Data Source

A large part of value provided from IoT deployments comes from data. However, getting this data into the existing data landscape is often overlooked. In this session, we will start by introducing what are the existing Big Data Solutions that can be part of your data landscape. We will then look at how you can easily ingest IoT Data within traditional BI systems like Data warehouses or in Big Data stores like data lakes. When our data is ingested, we see how your data analysts can gain new insights on your existing data by augmenting your PowerBI reports with IoT Data. Looking back at historical data with a new angle is a common scenario. Finally, we’ll see how to run real-time analytics on IoT Data to power real time dashboards or take actions with Azure Stream Analytics and Logic Apps. By the end of the presentation, you’ll have an understanding of all the related data components of the IoT reference architecture.

IOT50: Get to Solutioning – Strategy & Best Practices when Mapping Designs from Edge to Cloud

In this session we will explore strategies for secure IoT device connectivity in real-world edge environments, specifically how use of the Azure IoT Edge Gateway can accommodate offline, intermittent, legacy environments by means of Gateway configuration patterns. We will then look at implementations of Artificial Intelligence at the Edge in a variety of business verticals, by adapting a common IoT reference architecture to accommodate specific business needs. Finally, we will conclude with techniques for implementing artificial intelligence at the edge to support an Intelligent Video Analytics solution, by walking through a project which integrates Azure IoT Edge with an NVIDIA DeepStream SDK module and a custom object detection model built using CustomVision.AI to create an end-to-end solution that allows for visualization of object detection telemetry in Azure services like Time Series Insights and PowerBI.

Azure Marketplace at Microsoft Ignite 2020

by Scott Muniz | Sep 16, 2020 | Azure, Technology, Uncategorized

This article is contributed. See the original author and article here.

Microsoft Ignite banner

The Azure Marketplace team is excited for the upcoming Microsoft Ignite conference! The event is all digital and all free with tons of interesting sessions for customers and partners. We’ll be sharing insight into how IT organizations can use Azure Marketplace to remain agile while maintaining security, compliance, and budgets.

Join the live session or watch on-demand for announcements, engage the team at Ask The Experts, and schedule consultations to go deeper one-on-one.

Register for Microsoft Ignite:

https://myignite.microsoft.com/

Sessions and consultations

Find, try, and buy solutions with ease in Azure Marketplace

On-demand viewing anytime

Learn how to find an application in Azure Marketplace or in the Azure Portal. We will show you how to find, try, and buy solutions aligned to your needs and deploy applications in your environment. After that, we will walk through how you can manage applications purchased through Marketplace.

https://myignite.microsoft.com/sessions/6e0827f9-1e21-43cd-b2ea-472d007ee981

Simplify software procurement and management with Azure Marketplace

Tuesday, September 22 | 1:00 PM – 1:30 PM PDT

Want to accelerate your organization’s access to IT, developer, and business solutions while thrilling your procurement and legal teams? Confidently buy and deploy software solutions and discover services offerings from Microsoft and Microsoft partners in Azure Marketplace. Learn about new governance and management functionality built into the Azure Portal plus how buying through Microsoft simplifies contracting and billing.

https://myignite.microsoft.com/sessions/0e22c4ff-63c4-452d-b502-42083ac46c27

Simplify software procurement and management with Azure Marketplace

Tuesday, September 22 | 9:00 PM – 9:30 PM PDT

https://myignite.microsoft.com/sessions/868cbac4-b741-4270-b970-3f4a6afb45e1

Simplify software procurement and management with Azure Marketplace

Wednesday, September 23 | 5:00 AM – 5:30 AM PDT

https://myignite.microsoft.com/sessions/1e7a9a47-3c2a-445f-8a6a-29c6c86f4165

Ask the Expert: Simplify software procurement and management with Azure Marketplace

Wednesday, September 23 | 5:45 AM – 6:15 AM PDT

Join Microsoft experts to discuss how you can confidently buy and deploy software solutions and discover services offerings from Microsoft and Microsoft partners in Azure Marketplace.

https://myignite.microsoft.com/sessions/8c056d2e-559a-43fb-83e1-ed3dd444ff24

One-on-one consultation

Schedule a 45 minute engagement

Organizations of all sizes and industries find innovative IT and developer tools in Azure Marketplace and business and industry solutions in Microsoft AppSource. Join individual consultations with specialized tech and industry category managers to help you find software from Microsoft partners, or meet with architects and program managers to learn how you can commercialize your organization’s IP to sell through Microsoft.

https://myignite.microsoft.com/app-consult