ADF connector updates: Azure Data Explorer, SFTP, REST and HTTP

by Contributed | Mar 25, 2021 | Technology

This article is contributed. See the original author and article here.

Azure Data Factory adds new authentication mechanisms in Azure Data Explorer, SFTP, REST and HTTP connectors to provide you more flexible and secure ways to access the data stores.

Azure Data Explorer connector adds managed identity authentication

When you copy data from/to Azure Data Explorer (ADX), look up data or invoke ADX command, you can now use managed identity authentication in addition to service principal authentication. Learn more about Azure Data Explorer connector.

SFTP connector adds multi-factor authentication

Data Factory enables multi-factor authentication for SFTP connector so as to fulfill enterprise’s advanced security requirement. You can now access your SFTP server using multi-factor authentication which requires both username/password and SSH key. Learn more about SFTP connector.

REST and HTTP connectors add authentication header in linked service

REST and HTTP connectors have been supporting custom request header settings at activity level. While if your request leverages the header for authentication, you would need to specify the secret in plain text and also duplicate this setting in each activity.

Now you have a new option to configure such auth header once and in secure manner in linked service, which support referencing secret in Azure Key Vault or inline encryption like we do for other credentials. On the UI, you can choose from the pre-populated well-known auth headers or input your own. Auth header can work in conjunction with supported authentication types. And you can keep configuring additional headers in activity as usual for non-auth functionalities. Learn more about REST connector and HTTP connector.

Meet the 2021 Imagine Cup Top 12 teams

by Contributed | Mar 25, 2021 | Technology

This article is contributed. See the original author and article here.

Out of the tens of thousands of students from 163 countries who registered for the 2021 Imagine Cup, 39 teams advanced to the World Finals to showcase their original tech innovations. 

Our World Finalists represent four competition categories: Earth, Education, Healthcare, and Lifestyle. Using Artificial Intelligence, IoT, Augmented Reality, gamification, and so much more, these students have developed their ideas into impactful and inspiring solutions to make a difference. During Round 1 of the World Finals, each team gave a 3-minute project pitch followed by a demo presented to expert judges, answering questions and discussing their ideas in depth.  

Twelve teams were selected to move forward based on their innovative use of Azure technology, diversity and inclusive design, and potential to make an impact in their selected category. They’re heading on to Round 2 of the World Finals – congratulations to all these incredible students for making it this far! 

Meet the World Finalists!

Intelligent Hives, Poland Earth category Project: Intelligent Hives – Bee Monitoring System   On average, 30% of bee colonies are unable to survive into the next season. Intelligent Hives brings beekeeping into the 21st century. The team’s intelligent solution helps save bees, beekeepers, money, and time.
ProTag, New Zealand Earth category Project: ProTag   ProTag is a smart ear tag for livestock that can detect the early onset of illness in real-time, lowering costs and increasing animal welfare along with farmers’ peace of mind.
Virtual Radiologist, Nepal Earth category Project: Pico Sat   Pico Sat is a miniature version of an environmental satellite, which provides users information about environmental parameters like altitude, temperature, humidity, pressure, dust, and pressure.
Hands-On Labs, United States Education category Project: Hands-On Labs – How can we enable every student to have a truly ‘hands-on’ learning experience’ online?   Hands-On Labs is a set of remote laboratories that allows students to observe and remotely control physical tools online in real-time for their courses. The team aims to provide an active learning experience to students from any background all around the world.
Nyansapo AI, United States Education category Project: Digital Literacy Assessment   The team’s mobile app enables instructors to assess the literacy level of multiple children at once and track the individual literacy progress of children over time. As a digital assessment tool, the projects makes it easy for instructors to collect, organize, and analyze assessment data to effectively place students in the best educational level.
Tandemly, United States Education category Project: Tandemly   Using an array of built-in services and Augmented Reality, Tandemly aims to enhance and transform the educational experience for individuals with learning disabilities.
Cepha, United States Healthcare category Project: Cepha   Team Cepha created an early detection platform for Parkinson’s disease utilizing smartphone sensor data.
Intelli-Sense, India Healthcare category Project: Vision – the Blind Assist   The solution seeks to aid difficulties faced by blind individuals in understanding their surroundings in situations such as walking through a road or reading a book. The solution’s main goal is to provide a sense of vision to the visually impaired.
REWEBA, Kenya Healthcare category Project: REWEBA (Remote Well Baby)   The team’s solution is an early warning system that digitally monitors growth parameters of babies and sends data to doctors remotely for timely intervention. It combines a variety of technologies to provide innovative functionalities for infant screening.
Assurance Team, United States Lifestyle category Project: Assurance – Re(Imagine) Safety   Team Assurance created a proactive and affordable system to prevent mass shootings. It works by scanning the aggregate input from security camera feeds for firearms and alerts all parties it needs to alert as soon as a firearm is detected. The system is fully extensible and is able to run on low-cost hardware.
DataMasker, China Lifestyle category Project: WellMask   WellMask is an Artificial Intelligence social mask based on semantic extraction & symbol communication. The team hopes to solve the problems of inefficient communication and high social threshold caused by masks, so that masks are no longer synonymous with isolation.
Threeotech, Thailand Lifestyle category Project: JustSigns   JustSigns is a web application for content creators to create sign language captions to improve media accessibility for hard-of-hearing viewers.

Follow these teams’ journeys on Instagram and Twitter as they advance to the next round of the World Finals. Four winning teams will be selected, taking home USD10,000 and Azure credits. These teams will also move forward to the World Championship for the chance to win USD75,000 and mentorship with Microsoft CEO, Satya Nadella. Two runner-up teams in each category will take home USD2,500 plus Azure credits. Good luck to all our competitors!  

Web Shell Threat Hunting with Azure Sentinel

by Contributed | Mar 25, 2021 | Technology

This article is contributed. See the original author and article here.

In this blog post we will provide Microsoft Azure Sentinel customers with hunting queries to investigate possible on-premises Exchange Server exploitation and identify additional attacker IOCs (Indicators of compromise) such as IP address and User Agent. These hunting techniques can also be applied to web shell techniques targeting other web applications. 

The techniques we discuss below have been adapted from the June 2020 blog post: Web shell threat hunting with Azure Sentinel and Microsoft Threat Protection. The previous blog post analysed an attack against a SharePoint server, however, many of the techniques can also be applied to Exchange servers since it also uses IIS to host its web interfaces. 

Recent vulnerabilities in on-premises Microsoft Exchange servers have led to deployment of web shells by threat actors. More information on these vulnerabilities can be found in this MSRC blog, details on threat actor HAFNIUM using these vulnerabilities can be found in this MSTIC blog. MSRC has also provided guidance for responders, a one-click tool for remediation and automatic remediation is delivered through Microsoft Defender for Endpoint. 

Our colleagues in Microsoft Defender Threat Intelligence have authored another blog that provides additional details on use of web shells in attacks taking advantage of the Exchange Server.  

The below diagram provides a high-level overview of an attacker leveraging these vulnerabilities to install a web shell on an Exchange server. 

Investigating web shell alerts 

Microsoft 365 Defender (M365D) detects web shell installation and execution activity. Security alerts and incidents generated by M365D can be written to the SecurityAlert table in Azure Sentinel by enabling the appropriate connector. An example of a web shell installation alert in the Azure Sentinel SecurityAlert table can be seen below. 

These alerts can be enriched in Azure Sentinel with new information from other log sources. When dealing with remote attacks on web application servers, one of the best enrichment sources available are the web logs that have been generated. In the case that the application server is Microsoft Exchange the W3CIISLog can be used to enrich M365D alerts with potential attacker information. Information on collecting IIS logs using the Log Analytics agent can be found here.

Identifying the Attacker IP address from Microsoft 365 Defender alerts 

The query below extracts alerts from M365D where a web script file has been observed as part of the alert. In the below example, alerts containing ASP, ASPX, ASMX and ASAX files will be extracted; these are web script files commonly used by Exchange servers. 

After extracting relevant web shell alerts the query will join the alert information with the W3CIIS log, this allows the query to identify any clients that have accessed the potential shell file, allowing the potential attacker to be identified. A version of the query below is already available as an Azure Sentinel detection and can be found here. 

let timeWindow = 3d; 
//Script file extensions to match on, can be expanded for your environment 
let scriptExtensions = dynamic([".asp", ".aspx", ".asmx", ".asax"]); 
SecurityAlert 
| where TimeGenerated > ago(timeWindow) 
| where ProviderName == "MDATP" 
//Parse and expand the alert JSON 
| extend alertData = parse_json(Entities) 
| mvexpand alertData 
| where alertData.Type == "file" 
//This can be expanded to include more file types 
| where alertData.Name has_any(scriptExtensions) 
| extend FileName = tostring(alertData.Name), Directory = tostring(alertData.Directory) 
| project TimeGenerated, FileName, Directory 
| join (  
W3CIISLog  
| where TimeGenerated > ago(timeWindow)  
| where csUriStem has_any(scriptExtensions)  
| extend splitUriStem = split(csUriStem, "/")  
| extend FileName = splitUriStem[-1] 
| summarize StartTime=min(TimeGenerated), EndTime=max(TimeGenerated) by AttackerIP=cIP, AttackerUserAgent=csUserAgent, SiteName=sSiteName, ShellLocation=csUriStem, tostring(FileName)  
) on FileName 
| project StartTime, EndTime, AttackerIP, AttackerUserAgent, SiteName, ShellLocation 

Identifying Exchange Servers & Associated Security Alerts 

Exchange servers can be challenging to identify in default log data; however using data available in W3CIISLog, Exchange servers can be identified using predictable URI strings without relying on the hostname or site name. 

The query below extracts the host name from W3CIISLog where a known Exchange URI path is observed, this provides a list of hostnames that are running Exchange. This list of host names can then be used to aggregate information from the alerts in the SecurityAlert table. 

W3CIISLog 
| where csUriStem has_any("/owa/auth/", "/ecp/healthcheck.htm", "/ews/exchange.asmx") 
| summarize by computer=tolower(Computer) 
| join kind=leftouter ( 
SecurityAlert 
| extend alertData = parse_json(Entities) 
| mvexpand alertData 
| where alertData.Type == "host" 
| extend computer = iff(isnotempty(alertData.DnsDomain), tolower(strcat(tostring(alertData.HostName), "." , tostring(alertData.DnsDomain))),tolower(tostring(alertData.HostName))) 
| summarize Alerts=dcount(SystemAlertId), AlertTimes=make_list(TimeGenerated), AlertNames=make_list(AlertName) by computer 
) on computer 
| project ExchangeServer=computer, Alerts, AlertTimes, AlertNames 

The results of the query provide insights into whether additional security alerts beyond web shell alerts have been observed on the host. Following deployment of a web shell it’s highly likely the threat actor will begin to execute further commands on the server, triggering additional alerts. In the above example three Exchange servers were observed with security alerts.  

This same technique can be used to locate other web applications within the network that use common or predictable web paths.

W3CIISLog Analysis 

W3CIISLog provides detailed logging on actions performed on Microsoft Internet Information Servers (IIS). Even when an Endpoint detection alert is not available, it is possible to explore W3CIISLogs for indicators of compromise. W3CIISLog can also provide additional insights into which hosts in the network are web application servers. 

Note: As part of the original Microsoft HAFNIUM blog post, several hunting and detection queries were created to search for artefacts specific to the use of recent vulnerabilities.

Identifying generic exploitation activity 

If the URI associated with the vulnerable file on the server is known, a query can be constructed to identify log entries that match the URI pattern. W3CIIS logging stores the URI in the column named “csUriStem”, the below query can be used to search for a specific URI in logs and provide information on which clients have accessed them. Local IP addresses have been removed. 

W3CIISLog 
| where TimeGenerated > ago(3d) 
| where not(ipv4_is_private(cIP)) 
//Insert potentially exploited URI here 
| where csUriStem =~ "/owa/auth/x.js" 
| project TimeGenerated, sSiteName, csMethod, csUriStem, sPort, cIP, csUserAgent 

For HAFNIUM attacks observed by MSTIC an indicator feed has been made available (CSV, JSON). A detection query, that will check for the presence of indicators in multiple data sources, has also been made available by the Azure Sentinel team. The detection can be found here, and IOC’s released as feeds by MSTIC can be found in this directory. 

The recent Exchange vulnerabilities do not need to be targeted at a specific file. Analysis of automated exploitation tools online shows that many randomise the filenames used; this means that no legitimate user will visit these files as they do not exist on the server. As these filenames are randomly generated, static string matching cannot be used. 

The Kusto “matches_regex” function can be used to perform regular expression matching on URI’s. The below example extracts events where the URI matches files associated with the exploitation of CVE-2021-27065 from W3CIISLog. 

W3CIISLog 
| where TimeGenerated > ago(3d) 
| where not(ipv4_is_private(cIP)) 
| where (csUriStem matches regex @"/owa/auth/[A-Za-z0-9]{1,30}.js") or (csUriStem matches regex @"/ecp/[A-Za-z0-9]{1,30}.(js|flt|css)") 
| project TimeGenerated, sSiteName, csMethod, csUriStem, sPort, cIP, csUserAgent

The previous queries can be limited when the files being exploited are commonly accessed. They would produce many candidate attacker IP addresses, making analysis challenging. 

Using the recent Exchange vulnerabilities as an example, Microsoft has seen malicious automated tools released publicly that are being used to exploit the Exchange vulnerabilities. These tools are designed to only visit specific URIs on the server that are required to perform the exploit. This activity differs from normal and legitimate Administrator or User application browsing activity and if observed should be investigated. 

It is possible to craft a query that uses basic statistical analysis to identify instances where a client has visited a disproportionately high number of exploit-related URI’s when compared to other URIs on the site., The query below calculates the total number of suspicious URIs that have been visited by each user, it then calculates the total number of URIs visited by the user. Where the number of exploit related URIs is a significant proportion of URIs visited, a result is returned. By default, the query requires over 90% of the URIs visited by the user to be suspicious. 

let timeRange = 7d;
//Calculate number of suspicious URI stems visited by user 
W3CIISLog
| where TimeGenerated > ago(timeRange)
| where not(ipv4_is_private(cIP)) 
| where (csUriStem matches regex @"/owa/auth/[A-Za-z0-9]{1,30}.js") or (csUriStem matches regex @"/ecp/[A-Za-z0-9]{1,30}.(js|flt|css)") or (csUriStem =~ "/ews/exchange.asmx") 
| extend userHash = hash_md5(strcat(cIP, csUserAgent)) 
| summarize susCount=dcount(csUriStem), make_list(csUriStem), min(TimeGenerated), max(TimeGenerated) by userHash, cIP, csUserAgent 
| join kind=leftouter  ( 
//Calculate unique URI stems visited by each user 
W3CIISLog 
| where TimeGenerated > ago(timeRange) 
| where not(ipv4_is_private(cIP))
| extend userHash = hash_md5(strcat(cIP, csUserAgent)) 
| summarize allCount=dcount(csUriStem) by userHash 
) on userHash 
//Find instances where only a common endpoint was seen 
| extend containsDefault = iff(list_csUriStem contains "/ews/exchange.asmx", 1, 0) 
//If we only see the common endpoint and nothing else dump it 
| extend result = iff(containsDefault == 1, containsDefault+susCount, 0) 
| where result != 2 
| extend susPercentage = susCount / allCount * 100 
| where susPercentage > 90 
| project StartTime=min_TimeGenerated, EndTime=max_TimeGenerated, AttackerIP=cIP, AttackerUA=csUserAgent, URIsVisited=list_csUriStem, suspiciousPercentage=susPercentage, allUriCount=allCount, suspiciousUriCount=susCount

While this query is designed to detect recent Exchange exploit activity, it can be easily adapted to other exploit chains if the pages or URIs used are known. 

Rare Client File Access 

A previously published hunting query  can be used to detect instances where resources on a server are requested by a single client – a behaviour that should be investigated in the context of web shell exploits. After the actor creates web shell on the server, it’s likely that they will be the only user to access the file to complete their intended objective.  

Investigating the Attacker 

In the previous blog post covering SharePoint exploitation, a Jupyter Notebook Guided Investigation is provided. This notebook can also be used to investigate on-prem Exchange compromises within your environment. 

The notebook extracts alerts from Microsoft 365 Defender related to web shell activity, these can then be enriched with information from W3CIIS to identify the attacker IP and User Agent. The attackers IP and User Agent can be used to hunt through multiple log sources for potential post-compromise activity. 

After the attacker details have been identified, the notebook can be used to locate files that were accessed by the attacker prior to the web shell being installed. The notebook will also locate the first instance that the attacker visited the server. 

Azure-Sentinel-Notebooks/Guided Investigation – MDE Webshell Alerts.ipynb at master · Azure/Azure-Sentinel-Notebooks (github.com) 

Instructions for getting the notebook up and running can be found in the original blog post, under the title “Building out the Investigation using Jupyter Notebooks”. 

You can stay up to date with the latest information at https://aka.ms/exchangevulns.