Integration of Azure Cosmos DB with Azure Data Explorer for near real-time analytics

Integration of Azure Cosmos DB with Azure Data Explorer for near real-time analytics

This article is contributed. See the original author and article here.

Many of you must be wondering if building near real-time analytical solution is really possible or just another buzz in the world of analytics. In traditional data analytics solutions, you have to go through the time consuming curation processes to bring data in a shape to be consumed by the end users.

 

Now with the advanced systems and technologies, it is practically possible to ingest and query raw operational data in near real-time without impacting OLTP(Online transaction processing) system. Integration of Azure Cosmos DB with Azure Data Explorer makes it real with the following solution architecture –

 

RefArch.png

 

Benefits of this architecture

  • Readily available operational data for analysis as opposed to waiting for days to get the data.
  • Querying data without impacting the OLTP system’s performance.
  • Quick and fast interactive queries over fresh and large data sets.
  • Drill down from analytic aggregates always point to the fresh data.

 

Use cases 

There are numerous potential benefits of this architecture from business growth perspective. Just to give you an idea on its value proposition which is applicable to most organizations across diverse industries, sharing few examples of near real-time scenarios –

  • In e-commerce system, contextual recommendations and promotions based on customer’s current purchase enriched with historical purchase, consumer behavior trend analysis.
  • In manufacturing industry, analysis of data from IoT devices to respond to operational events, predictive maintenance, improvement in production safety.
  • In energy and utilities industry, analysis of meter readings from smart meters, decisions to instantly buy and sell capacity, power generation analysis.

Similarly in health, finance and many other industries, heaps of scenarios where you could make better business decisions if you get an ability to analyse data in near real-time.

 

Cost optimization

The next obvious question would be around data redundancy and cost impact of this solution. You could optimize the cost of this solution by managing the data retention policies across all the services. For example, Cosmos DB is an operational hot store where data is stored only for few days, Azure Data Explorer is an analytical warm store where frequently accessed data is stored, export rest of the old data to cold storage which is Azure storage in this solution. It is very easy to configure data retention in all these services so you could easily change it depending on your requirements.

 

Demonstration of solution with hands on lab

To help you understand the end to end flow of near real-time analytics solution, hands on lab with step by step guidance has been put together along with working code samples so you can try and test it on your own with the simulated data. Brief on what is being covered in this lab –

  • Simulate data using data generator component which will pump the data into Cosmos DB.
  • Leverage Cosmos DB change feed feature to trigger Azure function to push every change in Cosmos DB.
  • Use streaming capability of Azure Data explorer to ingest the streamed data via Azure Event Hub.
  • Run interactive queries using KQL(Kusto query language) with the glimpses on the advanced scenarios like forecasting, anomaly detection and time series analysis.
  • In last module of lab, you will have lot of fun building near real-time dashboard using ADX dashboards.

The lab is publicly available here at GitHub.

Try it out and share your feedback!

 

Note

Near real-time analytics solution can be built in multiple ways using different azure services, this lab describes one of the possible scenarios. Similar outcomes can be achieved using other azure services which are not covered in this lab.

 

 

 

 

Gwynn’s Of Mt Pleasant

Gwynn’s Of Mt Pleasant

[vc_row css_animation=”” row_type=”row” use_row_as_full_screen_section=”no” type=”full_width” angled_section=”no” text_align=”left” background_image_as_pattern=”without_pattern” css=”.vc_custom_1540223703471{padding-bottom: 15px !important;}” z_index=””][vc_column][vc_column_text]Congratulations to Gwynns of Mount Pleasant on the launch of their new website! You had a fabulous fashion make over. We look forward to helping you get your eCommerce site up and running as well.

[/vc_column_text][/vc_column][/vc_row][vc_row css_animation=”” row_type=”row” use_row_as_full_screen_section=”no” type=”full_width” angled_section=”no” text_align=”left” background_image_as_pattern=”without_pattern”][vc_column width=”1/2″][vc_gallery type=”image_grid” images=”2273,2274″ column_number=”2″ grayscale=”no” images_space=”gallery_with_space”][/vc_column][vc_column width=”1/2″][/vc_column][/vc_row]

What is Microsoft Teams

What is Microsoft Teams

Microsoft Teams Videos:

Welcome to Teams

Teams & Channels Overview

Create Instant Meetings

[/vc_column_text][/vc_column][vc_column width=”3/4″][vc_column_text]Microsoft Teams is a platform that combines workplace chat, meetings, notes, and attachments. The service integrates with the company’s Office 365 subscription office productivity suite, including Microsoft Office and Skype, and features extensions that can integrate with non-Microsoft products. Microsoft announced Teams at an event in New York, and launched the service worldwide on 14 March 2017. Here’s a brief overview of the features:

Teams

Teams allow communities, groups, or teams to join through a specific URL or invitation sent by a team administrator or owner. Teams for Education allows admins and teachers to set up specific teams for classes, professional learning communities (PLCs), staff members and everyone.[18]

Messaging

Within a team, members can set up channels. Channels are topics of conversation that allow team members to communicate without the use of email or group SMS (texting). Users can reply to posts with images, GIF’s and custom made memes.

Direct messages allow users to send private messages to a specific user rather than a group of people.

Connectors are third party services that can submit information to the channel, some connectors include MailChimp, Facebook Pages, Twitter, Yammer, SharePoint, Dynamics, SalesForce, Bing News, and many others.

Calling
  •     Instant messaging
  •     Voice over IP (VoIP)
  •     Video conferencing inside the client software

Teams supports public switched telephone network (PSTN) conferencing allowing users to call phone numbers from the client.

Meeting

Meetings can be scheduled or created ad-hoc and users visiting the channel will be able to see that a meeting is currently in progress. Teams also has a plugin for Microsoft Outlook to invite others into a Teams meeting.[19]

Education

Microsoft Teams allows teachers to distribute, provide feedback, and grade student assignments turned-in via Teams using the Assignments tab, available to Office 365 for Education subscribers.[20] Quizzes can also be assigned to students through an integration with Office Forms.[21]

Clients

As of November 2017, the following Microsoft Teams clients are available:

  •     Windows and MacOS: Included with Office 365
  •     iOS: Microsoft app in iTunes app store[22]
  •     Android: Microsoft app in Google Play[23]
  •     Windows 10 Mobile and Windows Phone 8.1[24]

For a deeper dive please watch the videos in this article. You can see the full Wikipedia article here.

*** CALL TO ACTION GOES GERE >>> For a free trial or demo![/vc_column_text][/vc_column][/vc_row]

Office 365 Update September 2018

Office 365 Update September 2018

[vc_row css_animation=”” row_type=”row” use_row_as_full_screen_section=”no” type=”full_width” angled_section=”no” text_align=”left” background_image_as_pattern=”without_pattern” css=”.vc_custom_1537382242845{padding-top: 10px !important;padding-bottom: 10px !important;}” z_index=””][vc_column width=”1/3″][/vc_column][vc_column width=”1/3″][/vc_column][vc_column width=”1/3″][/vc_column][/vc_row][vc_row css_animation=”” row_type=”row” use_row_as_full_screen_section=”no” type=”full_width” angled_section=”no” text_align=”left” background_image_as_pattern=”without_pattern”][vc_column width=”1/2″][vc_video link=”https://vimeo.com/290756741″][vc_column_text]

Introductory Comments

Welcome to the Office 365 update for September of 2018. In the next few minutes I’ll be giving you a quick rundown of the latest Office 365 updates, with the goal of helping you get the most out of the service.[/vc_column_text][vc_column_text]

PowerPoint

Source:    Office Support: What’s New in Office 365 for Mac— The font you choose is the font they see

Office Support: How to embed fonts in PowerPoint

Some of you may have experienced the situation after spending hours creating a PowerPoint presentation on a computer that uses one set of fonts. The presentation looks different if you open it on another computer that has a different set fonts installed. Let’s just say it’s happened to me a few, very inopportune, times.

I didn’t know this, but on Windows devices, you can ensure your fonts display as intended by embedding the fonts in the PowerPoint deck when you save it. Simply click on Save As, then click Browse to open the Save As dialog box, then click on Save Options in the Tools dropdown menu next to the Save button. Clicking the ‘Embed fonts in file’ check box ensures that the fonts you’ve used in your designs will be the fonts your viewers will see when you share your PowerPoint deck.

This font embedding open has been exclusive to Windows devices—until now. PowerPoint for Mac just released this same feature to Office Insiders. If you’re an Insider, simply click on Preferences under the PowerPoint menu, then on the Save icon. You’ll find the new Font Embedding option in the dialog box.

Now you can ensure that the fonts you design into your PowerPoint presentations display correctly not only on other Windows computers, but on the Mac platform as well.

Access

Source:    Office Support: Create a chart on a form or report

Access Blog: Introducing new and modern chart types

Several respondents to our recent viewer survey asked for more on Microsoft Access, so consider your request granted.

When talking with customers who rely on Access for their day-to-day business, a common theme is the key role that Access reports play, enabling decision makers to draw powerful insights from their business data. One of the most frequent requests in these conversations, as well as the UserVoice site, is for Access to provide more modern data visualizations, as well as creating a more streamlined process for creating those visualizations.

Recently, the Access team fulfilled that request with the introduction of a new set of easier-to-create charts that provide more ways to visualize your data and present it in a new and modern way. The list of available reports includes both updated versions of traditional report, such as column, bar, line, and pie charts, as well as new variations, such as the stacked column and stacked bar graphs.

The key here is that adding graphs to your reports and forms in Access is now easier than ever, with more intuitive controls. The new reports can be customized to fit your specific needs using a new Chart Settings pane as well as the familiar Access Property Sheet.

The Access team realizes that many existing customer created Access-based solutions rely on the current, ActiveX-based charts, and those solutions will continue to work. Access power users and developers can use the new charts alongside the classic charts within the same solution, transitioning to the new charts whenever they’d like.

There are great step-by-step instructions on how to use these new charts in the blog post I link to in the transcript and resources document. If you’re an Access aficionado, be sure to check it out!

Microsoft Teams

Source:    Microsoft Teams Blog: What’s new in Microsoft Teams – August round up

Microsoft Teams announced several feature enhancements in the August 2nd news roundup. Here are a couple of features I’m now using in my day-to-day work.

When concentrating on focus work, I’ve often wanted to set my status in Teams to Do Not Disturb. However, I’ve been reluctant to use that status setting because I want certain people, like my boss, to be able to reach me. Now, I’m able to set exceptions to my DND status. I just click on my avatar, click on Settings, then Privacy, and I’m able to give priority access to select individuals. That means I’ll receive notifications for chats, calls, and @mentions from the people I choose, even when my status is set to Do No Disturb.

Another new Teams feature I’m using is the new Wiki app. I access it from the More Apps ellipsis on the left-hand navigation panel. The Wiki app enables me to keep my own private notes on the Personal tab, see all the wiki pages I’m part of on the All tab, or get back to ones I’ve used recently on the Recent tab.

There’s also a handy wiki slash command for when you want to jot down a note while in a different area of Teams. In the command box at the top of the screen, simply type /wiki, then press Enter and type a note. Press Enter again when you’re finished, and the note will go right to the Personal tab in the Wiki app.

In a separate August 24th blog post, Microsoft announced they had completed their roadmap for bringing Skype for Business Online features and functionality into Teams. That means that Microsoft Teams is now ready to meet your messaging, meeting, and calling needs.

This new functionality in Microsoft Teams includes many new features, including:

  • Rich messaging features such as unified presence, federated chat, and in-line translation capabilities that allow team members to converse in their native language.
  • The ability to host and record cloud-based meetings with up to 250 participants;
  • and robust calling features including delegate support, call queues, auto-attendant, consultative transfer, and out of office support.

There are lots of additional details about this breakthrough announcement in the August 24th Microsoft Teams blog post. I’ve provided a link in the transcript and resources document.[/vc_column_text][/vc_column][vc_column width=”1/2″][vc_column_text]

Power BI

Source:    Power BI Blog: Power BI Desktop August 2018 Feature Summary

Power BI Blog: Ask A Question GA’s in Power BI Desktop

While Power BI reports and dashboards look great online, the Power BI team heard loud and clear that you needed an option to easily print or email copies of your work reports to others both inside and outside the organization. In fact, this was the number one feature request on the Power BI UserVoice site.

With the August Power BI Desktop update, you’ll find a new Export to PDF command under the File menu. Just give it a click and it will generate a pixel-perfect rendering of each page in your Power BI Report, opening it automatically in your default PDF viewer. From there, you can save it wherever you like, email it as an attachment, or print it!

Note that the Export to PDF option is only available in the desktop version of Power BI. There’s a download link as well as additional Power BI feature enhancements in the August 8th Power BI Blog post.

Speaking of the Power BI desktop, the team has made the ‘Ask a Question’ feature inside the Power BI Desktop globally available in the client, which allows you to directly place the ask a question box within your reports.

As an analyst: you can use natural language to easily explore your data and quickly author new charts and visualizations.

If you wish to allow consumers of the report to perform ad hoc analysis, you can place a button within the report which brings up the ‘Ask a Question’ experience. This experience offers the consumer of the report the ability to ask their own questions of the data or pick from a list of suggested questions.

Flow

Source:    Flow Blog Name: Introducing Microsoft Flow integration in Excel

The Microsoft Flow for Excel add-in enables you to connect your data to a wide range of services such as SharePoint, Outlook 365, Dynamics 365, Microsoft Teams, Visual Studio Online, Twitter, and more. With this integration that uses the ‘For a selected row trigger’ and the Flow launch panel, you can create and trigger on-demand flows for selected rows in any Excel table on spreadsheets hosted in SharePoint or OneDrive for Business.

The announcement of this new capability may cause you to wonder why someone would want to kick off a Flow from within an Excel spreadsheet in the first place. Well, you can satisfy your curiosity by visiting the blog post we link to in the transcript and resources guide and following along the scenario you’ll find there.

SharePoint

Source:    SharePoint Community Blog: Updates to SharePoint self-service site creation

Beginning in late July of 2018, the SharePoint team began rolling out several improvements to the SharePoint Online site creation experience, with the goal of empowering users while also providing the right controls to admins.

Up until recently, if Office 365 Groups creation was disabled for a set of users, those users were only able to create ‘classic’ SharePoint team sites. Now, those same users are enabled to create both modern SharePoint team sites and communication sites. SharePoint admins and Office 365 global admins can enable or disable the ability for users to create these SharePoint team sites. They can also specify which type of sites users can create.

The new sites users create will have the modern user experiences for the home page, lists, libraries, and, if they’re not connected with a Group, will function as a standalone site collection.

From an admin perspective, IT establishes both who can create Office 365 Groups and who can create SharePoint sites. With this change, admins can now allow more users to create modern team sites even if those users are not enabled to create Office 365 Groups, all the while keeping your governance plan intact.

Security

Source:    Security, Privacy and Compliance Microsoft Cloud App Security Updates – Microsoft Mechanics

Moving to the cloud increases flexibility for employees and reduces IT cost, but it also introduces new challenges and complexities for keeping your organization secure. To be able to get the full benefit of cloud applications, an IT team must find   the right balance of supporting access while maintaining control, to protect you data.

Our research shows that more than 30% of the apps used in organizations can be classified as, to use a technical term, risky. Taking into consideration more than 70 factors, Microsoft Cloud App Security evaluates every app against regulatory certifications, and compliance and industry specific standards, to provide customers with valuable insights to help them make informed decisions on how to manage cloud apps in their organization.

Our discovery advancements include the addition of 15 new risk factors that evaluate every app for its compliance with GDPR and a new C-level report that provides high-level summaries of the key findings, as well as recommendations on how to improve your organization’s security posture.

Lastly, with cyberthreats on the rise, we continue to invest in our threat detection capabilities and leverage end-user behavior analytics to minimize the impact on your organization. Microsoft Cloud App Security is available as part of Enterprise Mobility + Security E5 or as a standalone service.

We link to a great Microsoft Mechanics video on all the new enhancements, and of course, we’ll cover the updates at our Microsoft Ignite conference taking place September 24-28. If you’re unable to join us in Orlando, we’re live streaming all the sessions or you can watch them on demand via the Microsoft Tech Community.

Close

That’s it for this month. Remember, you can download the transcript and resources document that accompanies this video at aka.ms/o365update-blog.

I’m Jim Naroski, thanks for watching, and I’ll see you again soon![/vc_column_text][/vc_column][/vc_row]

Disaster Preparation and Recovery Plan

Disaster Preparation and Recovery Plan

[vc_row css_animation=”” row_type=”row” use_row_as_full_screen_section=”no” type=”full_width” angled_section=”no” text_align=”left” background_image_as_pattern=”without_pattern”][vc_column][vc_column_text]Hurricane season is here again and, if you haven’t already, it is time for your company to review your disaster plan. The list below is intended to help you put your business’ plan in place. It might seem overwhelming so please Contact us if you need assistance!

Protecting important information and documents

  1. Make a list of crucial organization contacts important (such as a list of employees, your insurance company, bankers, suppliers, lawyers, accountants, etc.) to your organizations and secure duplicate copies (paper, smart phone, cloud storage, etc.)
  2. To avoid water damage, back-up (to multiple sources) vital records and documents (i.e. accounting statements, insurance documents, legal contracts, tax returns) that are not easily reproduced. Store paper copies in a water and wind-proof location; store electronic copies off-site or in cloud storage.

Hardware and equipment

Before the storm inventory all hardware and software.
When it is determined the stores will be closed or there is an evacuation:

  1. Move all registers, printers, routers, floor mounted displays, and other non-mounted hardware to the place with the highest elevation.
  2. Wrap each piece of equipment in a trash back and make sure it is air tight.
  3. Staff should take laptops, tablets, phones with them if they evacuate to a safe place.
  4. Document the location of each piece of equipment and where it’s from.
  5. No equipment should be left plugged in.

Advance preparations

  1. Arrange for cleanup, repair, and restoration services for your facility with reputable firms outside of your immediate area, well in advance. This is very important since many organizations will probably be calling upon the same local clean-up services; you’ll be well-served by planning for your organizations clean-up beforehand.
  2. Establish a hurricane preparedness and response plan customized to your organization’s unique location, construction, operations and resources.
  3. Consider establishing a mutual aid agreement with an organization similar to yours that is located outside the storm’s impact area to share resources and serve as a possible evacuation site.
  4. Photograph the exterior and interior of your building as well as the contents. Store photos in a protected location, or email them to yourself to help document any loss.
  5. Notify your employees of the procedures that will take place in the event a hurricane warning or watch is issued.
  6. Address access to your facility after the storm. This may require contacting public authorities for official identification badges to enter what may become a restricted area.
  7. Determine what furnishings or major equipment will need to be protected or moved and document how you plan to accomplish it. You’ll have to decide if you want to try to protect them in place or move them out of the area at risk. Determine what manpower and any equipment you will need to get this done. If you’re planning to protect in place, move your equipment to well-protected interior rooms or the floors above the anticipate flooding level.
  8. Identify outside furnishings and equipment such as trash cans, signs, furniture, awnings, etc., that could be blown about and become “deadly projectiles” and determine either how they will be secured or where they will be stored.
  9. Make sure that any of your facility’s rooftop equipment such as air conditioning units, antennas, exhaust fans or turbines are firmly secured or strapped down to the roof structure (e.g., the joists) to withstand high winds.
  10. Ensure you have the necessary tools to board up windows and brace doors. The first priority in protecting your facility will be to keep the wind out. Wind pressure and windblown debris can break windows and blow in doors. Sliding glass doors, large picture windows, skylights, French doors, inward opening double doors, and garage doors are particularly vulnerable. Such tools as a circular or hand saw, a drill with appropriate bits, a hammer or nail gun, hand or power-driven screwdriver, and a wrench may be needed. Nails will be sufficient on wood-framed windows and doors but screws or bolts and washers are necessary for metal-framed windows and doors.
  11. Ensure that your staff knows how to turn off the utilities (gas, water, electricity) for your buildings at the main shut off switches.

Action Items – If evacuation is recommended by local officials:

  1. Close your operations and business.
  2. Revise telephone answering system to inform callers of facility closure.
  3. Transfer essential business records and valuables (if you haven’t done so already) to a safe location out of the evacuation area.
  4. Where possible, move furnishings away from exterior windows and doors and as many items as possible off the floor.
  5. Move expensive equipment out of the area or move it to the most heavily constructed interior area of you facility. If your locale is subject to surge flooding, move your equipment to floors above the anticipated surge level. Cover equipment that can’t be moved with plastic sheeting to minimize damage from leaking roofs or broken windows.
  6. Conduct an orderly shutdown of your computer(s) and sensitive electrical equipment and disconnect and unplug nonessential equipment to protect from damage caused by power surges immediately before or after an outage.
  7. Brace exterior doors that open inward and brace any roll-up doors. To prevent them from lifting from their tracks, wedge sliding doors. (Do not block emergency exits.)
  8. If you have storm shutters, close and lock them. If not, close, lock and board up your large windows and glass doors. Lower blinds and close curtains to help hold back flying debris. Remove after the storm to provide adequate ventilation.
  9. Turn off electricity, gas, water, and other utility services.
  10. Find the safest place for vehicles.
  11. Make sure your staff has departed your organization before evacuation routes become impassable due to flooding or high winds.

Action Items – If evacuation is not recommended:

  1. If local officials do not recommend evacuation of your area, your facility may still experience high winds and heavy rain generated by a hurricane.
  2. Take appropriate protective measures outlined above as a guide to reduce the vulnerability of wind and heavy rain damage.
  3. Have your building maintenance staff on standby and materials ready to expedite speedy repairs.
  4. Be prepared for loss of utilities. This means having battery-powered lights, radio, a supply of potable water and, if possible, an emergency generator.
 
After the hurricane >>

Reentering evacuated areas

  1. If you evacuated your facility, you may have difficulty returning quickly because roads may be damaged, blocked by debris, or flooded in low lying areas.
  2. Access to storm-damaged areas may be limited by local law enforcement personnel to keep people out of areas with dangerous conditions, facilitate rescue and recovery work, and limit access to unoccupied properties.
  3. After the hurricane passes, entry to storm-damaged areas may be limited to search and rescue personnel, law enforcement personnel, firefighters, utility crews, and road clearing teams. Once officials determine it is reasonably safe, property owners and essential employees will be cleared to enter the area, but they may be required to have a permit or pass, or be included on an access list maintained by the city. Contact your local emergency management office to determine the procedures for returning to storm-damaged areas.
  4. Listen to your radio or television stations (or check the internet if possible) and return only after an “all clear” is issued by authorities. Drive only if absolutely necessary, and avoid flooded roads and washed out bridges. Driving may be especially hazardous due to debris on the streets, emergency vehicles in operation, and nonworking traffic and street signals.

Checking your facility

  1. Enter your facility with caution. Beware of insects, snakes and animals driven to higher ground by flood waters.
  2. As soon as possible after the storm, assess any damage and look for obvious structural damage to your facility and its foundations. If you see significant damage, don’t attempt to enter the affected building. Also inspect the roof, roof mounted equipment, walls, doors and windows (outside and inside).
  3. If possible, cover up openings in the building shell with plastic sheeting or tarpaulins.
  4. Identify immediate hazards, such as downed or dangling electrical lines, leaking gas, broken sewers, flammable liquids (or other hazardous materials) and broken water pipes on your property. Exercise extreme caution and stay away from downed power lines and broken sewer lines. Do not take lanterns, torches or any kind of open flame into a damaged building as there may be leaking gas or other flammable materials present. Report any damage to power, water or wastewater equipment to your utility company.
  5. Make sure the electrical outlets and appliances throughout your facility are dry and free of water before turning the power back on. If you have any doubt about the condition of wiring or appliances, have an electrician check them to make sure there are no short circuits.
  6. Clean up debris, remove water.
  7. Clean roof drains and remove debris to prevent drainage problems.
  8. Use fans to speed the drying process. Do NOT use heaters.

[/vc_column_text][/vc_column][/vc_row]