How to achieve cloud-native endpoint management with Microsoft Intune

How to achieve cloud-native endpoint management with Microsoft Intune

This article is contributed. See the original author and article here.

In this post, we’re focusing on what it really takes for organizations to become fully cloud-native in endpoint management—from the strategic leadership to the tactical execution.

The post How to achieve cloud-native endpoint management with Microsoft Intune appeared first on Microsoft 365 Blog.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Effective strategies for conducting Mass Password Resets during cybersecurity incidents

This article is contributed. See the original author and article here.

You’re in the middle of a cyber incident, and you know certain accounts have been compromised, but you are not certain of the full extent of the Threat Actor’s impact. What do you do? Oftentimes, Microsoft Incident Response will recommend a mass password reset. This helps you regain control of your identity plane, deny other avenues of access, and disrupt any persistence the attacker may have established in your environment. However, and especially for larger organizations, navigating mass password resets can be a complex task. In this blog post, we’ll discuss the practical challenges of performing a mass password reset, how to prepare to carry one out, and best practices in performing them.


 


Identifying the need for a mass password reset


 


A mass password reset is not always required, but it is important to identify the circumstances under which it is. Some considerations for when a mass password reset is the best course of action include:



  • Active Directory database exfiltration: When there is evidence of Active Directory Domain Services (AD DS) database exfiltration by a suspected threat actor.

  • Active Directory database staging: When there is evidence of AD DS database staging with intent to exfiltrate by a suspected threat actor.

  • Compromised privileged identities: When a threat actor has compromised credentials belonging to one or more privileged groups such as Domain Admins, Enterprise Admins, or built-in Administrators.

  • Attacker-in-the-Middle: When there is evidence of an Attacker-in-the-Middle (AiTM) attack or other threat-actor-introduced proxy services which may have gathered user credentials.

  • Cloud or third-party identity platform compromise: When there is evidence of a compromise on an authoritative Identify platform such as Microsoft Entra Connect, AD FS, RADIUS (Remote Authentication Dial In User Service) Servers, or 3rd party identity solutions.

  • Ransomware deployment: When a threat actor has been able to successfully deploy ransomware by compromising accounts belonging to privileged Active Directory (AD) groups.

  • Privileged credentials exposed in Business Email Compromise (BEC): When a BEC has exposed privileged credentials in emails.

  • Privileged credentials exposed in exfiltrated data: When data exfiltrated from productivity and collaboration tools (such as OneDrive or SharePoint) has exposed privileged credentials.

  • Privileged credentials exposed in code: When privileged credentials have been exposed in an online code or source control repository.

  • Attribution to nation state or Advanced Persistent Threat (APT): When an attack has been attributed to an APT or nation state.


 


Organizational challenges and scenarios


 


Almost all organizations have remote users: many have hybrid users, and some have entirely remote workforces. This means that every organization has unique requirements and considerations for when a mass password reset is required. In this section, we will consider some of those requirements and how organizations can best prepare and respond if the need arises. Scenarios to consider include:



  • Local users: Users primarily onsite with line of sight to a domain controller.

  • Remote users: Users who primarily use VPN (virtual private networks) or have hybrid identities.

  • Administrative controls: Whether password resets are driven by administrators or end-users.

  • Service account management: Considerations for service accounts, which often have never-expiring passwords.

  • Privileged identities: Special considerations for managing privileged cloud and on-premises accounts.


 


Users onsite with direct access to domain controllers


 


This scenario is the least complicated one: if all users are primarily onsite with line of sight to a domain controller, then a simple flag on every user account to require the user to change password at next logon can be used to enforce the password change. Users can be given a deadline and informed they are required to change their passwords by the deadline, and, if they fail to do so, their accounts will be disabled. Several PowerShell scripts are available online that allow for enumeration of users in specific organizational units (OUs) and manipulating the “User must change password at next logon” flag to facilitate a gradual password reset rollout so an organization’s helpdesk is not inundated. When the users arrive in the office and attempt to log on, a message will prompt them to change their passwords.


 


Gradual, but expedited expiration of passwords using Fine Grained Password Policies (FGPP) and the progressive reduction of password age through domain policy modifications offer alternative methods for enforcing a mass password reset for domain users. However, a significant drawback to this approach is the potential for a threat actor to remain within an authenticated session until a logon event triggers the password reset. When considering this method, it’s important to balance the urgency of credential changes with the need to provide users with a grace period. Since many organizations have a portion of their workforce operating remotely, this strategy is often employed as part of a broader series of steps designed to secure all user accounts across various scenarios.



Remote users who use VPN to access the environment


 


This scenario is more common when most users are primarily remote, or there is a mix of remote and onsite users. In this scenario, users rely on authentication mechanisms separate from their domain password; for example, certificate-based authentication. Once the users are authenticated using the VPN solution, they can be treated like the previous scenario since they will have line of sight to a domain controller.


 


An important consideration for remote users is whether you will execute an administratively managed password reset (which is where an admin resets credentials for users and relies on users to use self-service password reset (SSPR) to regain access) or allow users to change their credentials gracefully on their own.


This scenario becomes more challenging when the VPN solution relies on the domain password as one (or the primary) factor for authentication and the VPN solution does not support a password reset during the sign-in flow. In such a scenario, if the organization has been set up for SSPR before the incident occurs, it makes the password reset process much easier to handle. If an organization does not have SSPR capabilities, a mass password reset will require some manual intervention. This could take the form of users having to call in to the help desk or attend a centralized location that has been set up for this purpose, provide verification of their identity over voice, video, or in person, and then have their password manually reset.


 


Alternatively, for VPN solutions that do not support a password reset during the authentication flow, you may wish to consider migrating the authentication source of your VPN solution to Microsoft Entra ID either temporarily to allow the session to be interrupted with a password reset, or permanently to gain the benefit of additional Microsoft Entra ID features like Conditional Access policies.


 


Users primarily remote with hybrid (on-premises) identities


 


With hybrid identities, an organization’s identities (users and computers) are already synchronized to Microsoft Entra ID. In this scenario, line of sight to a domain controller is not a requirement to orchestrate a mass password reset. Microsoft Entra ID supports flagging users to reset their credentials at next sign-in, similar to on-premises Active Directory.

Admins can use Microsoft Graph to set the user attribute either to
“forceChangePasswordNextSignIn” or “forceChangePasswordNextSignInWithMfa” on the desired users to interrupt their next sign-in and allow them to change their password gracefully. If the password writeback feature is enabled in Microsoft Entra ID and the organization’s users are enabled for SSPR, then a password reset via either the MyAccount portal or SSPR portal will ensure that the newly reset password is synchronized back on-premises. If password writeback and SSPR are already enabled, this is the scenario with the fastest route to threat actor removal and least amount of manual work. There are some scenarios where an organization may not want to use SSPR, which we will discuss later in this post.


 


Considerations for service accounts


 


Service accounts with their never-expiring passwords and traditionally overprivileged nature tend to be the bane of any Active Directory administrator’s existence. This is particularly problematic when a mass password reset must be performed and little-to-no inventory exists that maps applications to service accounts. An effort should be made to inventory all service accounts and their associated services and applications. Where possible, service accounts should be migrated to Group Managed Service Accounts (gMSA). This has the dual advantage of making service accounts more manageable and removing the manual overhead associated with service accounts. This is also a great opportunity to “right size” the service accounts that tend to be traditionally overprivileged.


 


Considerations for privileged identities


 


All privileged cloud accounts should have phishing-resistant MFA enforced. Also, it is strongly advised to use Just in Time (JIT) administration methods, for example Microsoft Entra ID Privileged Identity Management (PIM). In addition, there should exist a clear separation of on-premises and cloud administration with separate identities for each realm. Identities belonging to the privileged on-premises AD DS groups should not be synchronized to Microsoft Entra ID. Conversely, all privileged cloud roles should be held by cloud native identities and must not be synchronized from AD DS. Most organizations will choose to manually reset any privileged credentials for a high level of assurance and control. It is important to verify when passwords were reset with PowerShell or Microsoft Graph; otherwise, it is very likely that some accounts may be missed.


 


Assurance and control considerations for a mass password reset


 


As we’ve detailed, there are several different scenarios that necessitate a mass password reset. This means that there are different levels of control or assurance an organization might require while performing a mass password reset. When SSPR mechanisms can be reliably used to provide assurance, organizations can use that feature to accelerate a mass password reset.


 


However, there are situations where an organization may not want to use the existing SSPR solution. For example, when an advanced threat actor has abused the organization’s SSPR system, or where there is actual evidence of AD DS database exfiltration. In such a scenario the organization would likely not choose to use that mechanism to enforce the mass password reset because the threat actor could re-establish initial access or persistence via SSPR.


 


Where an organization seeks a high degree of control and assurance for a mass password reset there will, unfortunately, be an element of manual intervention. However, with preparedness ahead of time, Microsoft Entra ID features such as a Temporary Access Pass, when combined with Conditional Access policies, can be used to automate some aspects of assurance and control. In any event where a high degree of assurance and control is desired, some level of manual intervention to verify users’ physical identities and the issuance of such temporary access passes is inevitable. In a subsequent post we will examine different Microsoft Entra ID features that can be used to accomplish this.


 


Conclusion and next steps


 


There are several variables and considerations for a mass password reset, and there is no one-size-fits-all solution. However, we can, with adequate preparedness, make this process less onerous and more manageable for organizations.


 


We recommend exploring other blogs from Microsoft Incident Response for expert guidance and tailored solutions to improve your incident response capabilities. Additionally, consider the benefits of Microsoft Entra ID for advanced identity and access management, which can strengthen your defenses against identity-related breaches.

NIST CSF 2.0 – Protect (PR) – Applications for Microsoft 365 (Part 1)

NIST CSF 2.0 – Protect (PR) – Applications for Microsoft 365 (Part 1)

This article is contributed. See the original author and article here.

shawnrosco_0-1717968678915.png


 


The National Institute of Standards and Technology (NIST) published the first version of its Cybersecurity Framework (CSF) in 2014. Ten years later NIST released the second iteration of CSF, entitled NIST CSF 2.0. Microsoft and its partners have supported organizations in implementing the original CSF guidance, going as far as building and enhancing an assessment in Microsoft Purview Compliance Manager since 2018. This blog and series will look to apply NIST CSF 2.0 to Microsoft 365 and discuss changes from the previous publication.


 


It is somewhat improper to look at any particular CSF Functions in a vacuum or singular vantage point. NIST CSWP 29 (the primary document) illustrates and describes CSF Functions as “a wheel because all of the Functions relate to one another. For example, an organization will categorize assets under IDENTIFY and take steps to secure those assets under PROTECT. Investments in planning and testing in the GOVERN and IDENTIFY Functions will support timely detection of unexpected events in the DETECT Function, as well as enabling incident response and recovery actions for cybersecurity incidents in the RESPOND and RECOVER Functions. GOVERN is in the center of the wheel because it informs how an organization will implement the other five Functions.”


 


NIST CSF 2.0 WheelNIST CSF 2.0 Wheel


 


Therefore, despite the blog title, there will be occasional references to other CSF Functions and Categories, as well as future blogs covering other Functions. This discussion will also endeavor to focus primarily on Microsoft 365 but venture into Azure topics periodically by the nature of each solution. This blog or any subsequent blogs in the series will not be an exhaustive review of all possible applications of NIST CSF 2.0, nor exhaustive of the technologies mentioned and their abilities to manage cybersecurity risks.


 


As a final caveat, Amy Adams in Talladega Nights once spoke of one of the most talented individuals behind a wheel this way, “Ricky Bobby is not a thinker. Ricky Bobby is a driver.” I want to believe I might be the latter.


 


NIST CSF 2.0 – Protect (PR)


PR as a function is intended to cover “safeguards to manage the organization’s cybersecurity risks” and contains five Categories. The prior CSF publication included six categories, but two were significantly edited and renamed. PR.MA: Maintenance for example was mostly removed with remnants found elsewhere. Let’s first dive into PR.AA. NOTE: Text in green are excerpts from CSF documentation.


 


Identity Management, Authentication, and Access Control (PR.AA): Access to physical and logical assets is limited to authorized users, services, and hardware and managed commensurate with the assessed risk of unauthorized access


 


Identity and access are not just about directories and networks. Organizations of all sizes and industries are challenged with controlling access to digital estates that are often complex and boundaryless because of accelerated technology adoption. Microsoft Entra’s family of solutions shown below employs a variety of measures to manage access to resources limited to authorized users, services, and hardware.


shawnrosco_2-1717968678930.png


 


To meet the spirit of NIST CSF 2.0 PR.AA and a multitude of organizational scenarios, access decisions will need to be based upon periodic and real-time risk assessment. Automated and agile solutions are also necessitated for IT and security teams to avoid the manual processes traditionally associated with granting and managing access rights. Lastly, organizations will need to begin implementing some of the latest phishing-resistant multifactor authentication approaches using FIDO2 security keys, passkey technology, and/or certificate-based authentication to meet the barrage of sophisticated identity threats.


 


PR.AA-01: Identities and credentials for authorized users, services, and hardware are managed by the organization


 


Microsoft Entra ID (fka Azure Active Directory) can serve firstly as the management solution of identities and credentials for “authorized users” in Microsoft 365, along with other infrastructure and platforms. In the Entra ID admin center, you can create and manage user accounts, different types of groups (e.g., security groups, distribution groups), and memberships to groups for enhanced access permissions and group-based policies.


 


Additionally, Microsoft Entra ID Governance comes with Microsoft Entra ID P2 and enables entitlement management, a feature for managing identity and access lifecycle at scale, automating workflows, assignments, reviews, and expirations. Entitlement management is able to bundle together an “access package” consisting of resources like groups, applications, and SharePoint Online site access that users need to perform their tasks or projects. These access packages can be bound to a department, role type, internal vs external user, newly onboarded employees, etc. Lastly, organizations can automate the creation and removal of identities based on employment status, integrating with HCM or HR systems for efficient governance.


 


Management of “credentials of authorized users” can also include the management of a self-service password reset (SSPR) process for users. Microsoft Entra’s SSPR allows users to change or reset their password without administrator intervention but reliant on thoughtful policy. This feature is designed to reduce help desk calls and improve productivity by enabling users to unblock themselves if they forget their password or get locked out of their account. Administrators can configure the number of methods required for password reset or unlock, and users are prompted to confirm their registered information periodically.


 


Microsoft Entra External ID also enables internal users to collaborate with external users (guests) by inviting them to your organization, managing guest accounts in the Microsoft Entra admin center or by PowerShell, and ensuring they have appropriate access while maintaining control over your resources. External ID B2B collaboration is particularly useful for inviting external business partners to access apps and resources using their own credentials. This eliminates the need for managing new credentials, as guests authenticate with their home organization or identity provider. However, it is important to document and enforce least privilege per PR.AA-05 below for these users.


 


Lastly, Microsoft Entra ID allows for central management of device identities and monitoring, with features like viewing total, stale, noncompliant, and unmanaged devices. By registering and joining devices to Entra ID, organizations can enable Seamless Sign-on to both cloud-based and on-premises resources.


 


shawnrosco_3-1717968678943.png


 


Cross-Function Tip: PR.AA-01 could be included in your organization’s plan for the Detect (DE) Function and Continuous Monitoring (DE.CM) Category. Changes and actions taken by administrators in the management process can be “adverse events” resulting from malicious lateral movement or compromised identities.


 


PR.AA-02: Identities are proofed and bound to credentials based on the context of interactions


 


Microsoft Entra ID Conditional Access policies can further prove the authenticity of an authorized user and their access activity, and dynamically adjust the bounds of that identity based upon various conditions. A Conditional Access policy can analyze details of a session or authentication attempt based upon conditions such as the network or location of the session, and the current identity risk of a user (leaked credentials, signs of a password spray attack, etc.) Microsoft Entra also uses industry standard continuous access evaluation (CAE) with token protection features to verify identities and assess based upon context.


 


shawnrosco_4-1717968678951.png


 


During the device registration process in your organization, the user’s identity is also bound to the device based upon Microsoft Entra ID policies. Enrolled Windows devices, for example, will have an encrypted session key issued by Microsoft Entra that ensures token requests are not tampered with when a device is accessed, and subsequently Microsoft 365 applications. Other Microsoft Entra ID mechanisms are in place to prove and bound identities within device interactions, as well as prevent various types of credential theft. You can read more about them here.


 


For high-risk actions, organizations can require a step up proof of identity in real-time using Microsoft Entra Verified ID


 


PR.AA-03: Users, services, and hardware are authenticated


 


Starting in 2019, Microsoft provides on-by-default multifactor authentication in all new Microsoft 365 tenants. This can be reviewed and controlled in the Microsoft Entra admin center. In addition, Microsoft started automatically rolling out “Microsoft-managed” Conditional Access policies that force an MFA action when an admin signs into the Microsoft 365 admin portal, and when any user attempts a high-risk sign-in.


 


You can view the policies and their impact using the new policy view user experience in Microsoft Entra. Within the portal, administrators can also review a policy summary, alerts, recommended actions, and an impact summary. Administrators can also create their own custom policies focused on certain users, groups, or roles if scenarios are not covered by the default policies automatically deployed. It is possible to clone a policy and then make changes to an existing templated policy as well to suit these needs.


 


Cross-Function Tip: PR.AA-03 can be included in your organization’s plan for the Detect (DE) Function and Continuous Monitoring (DE.CM) Category. Security teams can monitor authentication events using sign-in and audit logs with Microsoft Sentinel and reason over them with support from Microsoft Copilot for Security.


 


PR.AA-04: Identity assertions are protected, conveyed, and verified


 


For those less familiar with the term “identity assertion(s)”, NIST 800-63C Digital Identity Guidelines gives an explanation of this requirement in greater detail. You can define Microsoft Entra ID as an Identity Provider or IdP that can convey an assertion to a Relying Party (RP), and these “assertions are statements that contain information about a subscriber”. The RP “uses the information in the assertion to identify the subscriber and make authorization decisions about their access to resources controlled by the RP.” In other instances, Microsoft Entra ID could be considered an RP when receiving assertions from an external IdP to grant guest access to Microsoft 365.


 


Organizations can, as a part of Microsoft Entra ID P1 and P2, configure Microsoft Entra SAML token encryption with RP apps that support SAML assertions. This encryption ensures that the content of the token cannot be intercepted easily, and personal or corporate data remains secure. OpenID Connect (OIDC) alternatively is another method to enable single sign-on with an OAuth-enabled RP app by exchanging ID tokens issued by Microsoft Entra authentication servers. An organization can register an app in the Microsoft Entra admin center to configure both federated SSO scenarios (and others) for conveying, protecting and verifying identity assertions. It’s important to note that even without token encryption, Microsoft Entra ID already requires encrypted HTTPS/TLS channels for token exchanges, adding a layer of security.


 


As mentioned in PR.AA-01, eternal IdPs or external Microsoft 365 tenants can convey identity assertions to an organization through External ID. A simple invitation and redemption process lets an external party use their own credentials (via their own IdP) to access your company’s resources as the RP. Once the guest redeems their invitation or completes sign-up, they’re represented in your directory as a user object. Below is a sample flow aligned to B2B collaboration guidance, and gates 1-4 & 6 can be configured as allowable or not.


 


shawnrosco_5-1717968678957.png


 


Cross-Function Tip: PR.AA-04 can be included in your organization’s plan for the Govern (GV) Function and Cybersecurity Supply Chain Risk Management (GV.SC) Category. As an organization works with various partners and suppliers, it’s important to ensure “supply chain risk management is integrated into cybersecurity and enterprise risk management”. Compromised identities from a supplier can directly impact external access and identity assertions for example.


 


PR.AA-05: Access permissions, entitlements, and authorizations are defined in a policy, managed, enforced, and reviewed, and incorporate the principles of least privilege and separation of duties


 


Beyond identity, credential and authentication management covered in other AA subcategories, this subcategory can necessitate more robust written and technical policies. A solution to consider first is role-based access control (RBAC) enabled by Microsoft Intune and Microsoft Entra, which helps your organization manage who has access to internal resources and what they can do with those resources. By assigning roles to your users, you can limit what they can see and change. A built-In or custom role defines the set of permissions granted to users assigned to that role. Organizations can manage role assignments following least privilege principles to define which users are assigned to the role, resources they can see, and resources they can change.


 


With Microsoft Entra ID P2, Microsoft Privileged Identity Management (PIM) also allows an organization to manage, enforce, review, and incorporate least privilege and separation of duties for important administrative roles and entitlements. PIM supports a similar degree of control with groups and the associated entitlements and authorizations in Microsoft 365. From a management perspective, an approval process can be enforced to justify an elevation of a user/identity to a more privileged role and documented within PIM for later review if necessary.


 


Access to critical assets like sensitive data or information in Microsoft 365 may be authorize to users that do not have elevated administrative roles discussed in the previous paragraph. A user may be an analyst within a financial department, a human resources coordinator, or a junior engineer for a research and development project. Users in these roles may not have entitlements to export large quantities of data or grant entitlements to other users but are equally governed by defined written policies and technical policies administered in Microsoft Purview and Microsoft Entra.


An organization can develop Microsoft Purview sensitivity labels to manage and enforce access authorizations for files, emails, and meetings across Microsoft 365: SharePoint, Teams, PowerBI, OneDrive and more. Label policies can then restrict which users or groups of users have authorization to access the labeled content.  


 


PR.AA-06: Physical access to assets is managed, monitored, and enforced commensurate with risk


 


From a Microsoft 365 perspective, there is physical access to two different types of assets:



  1. Physical compute resources (such as servers) and networking components at Azure datacenters across the globe

  2. Endpoints (laptop, mobile device, etc.)


According to the shared responsibility model in the cloud, a PaaS and SaaS offering like Microsoft 365 and it’s comprised solutions alleviate the responsibility of managing monitoring or enforcing access control to physical assets hosted in the Azure data center.


 


shawnrosco_6-1717968678965.png


 


Azure datacenters deploy several safeguards to project physical assets, such as: video surveillance, perimeter fencing, 24×7 security officer postings with background checked personnel, locked server racks and server floors, integrated alarm systems, time and area-bound access granted through a protected ticketing system, and multi-factor access control including biometric scanning. As mentioned in the Category description of PR.AA, “logical access” to Microsoft 365 infrastructure, including customer data, is prohibited from within Microsoft datacenters. For more information about physical access security or to geek out on granular details like the material used to construct perimeter fences (spoiler – it’s steel and concrete), here is your muse.


 


Beyond the datacenter – laptops, mobile devices, and other endpoints can be considered a physical asset in the sense that unfettered physical access to them could create risk for an organization. The Protect (PR) Function at the highest level is defined as “safeguards to manage the organization’s cybersecurity risks”. Safeguarding endpoints as a physical asset, however, cannot be protected by fences and video surveillance realistically as you would a data center physical asset. Therefore, it is important to evoke other Categories, like Awareness and Training (PR.AT) and Data Security (PR.DS), within the PR Function to train employees to safeguard their endpoints and avoid negative impacts of a lost or stolen device. Organizations can also document this risk in your Organizational Profile and/or System Security plan.


 


Cross-Function Tip: The Identify (ID) Function contains a Category for Asset Management (ID.AM) that pertains to an organization’s strategy for inventorying and managing physical assets like endpoints. The lifecycle of a hardware component can include wiping and recommissioning/decommissioning previously used devices, as well as a process of remote wiping a lost or stolen machine using Microsoft Intune. Much of ID.AM pertaining to servers and network-related physical assets are managed at the Microsoft datacenter for Microsoft 365 tenants.


“You gotta learn to drive with the fear”


In this blog we covered all of the PR.AA Subcategories, where they apply to Microsoft 365, and how Microsoft solutions address each applicable element. This coverage is not comprehensive of all risk or all strategies; yet, the intent is to prime your organization’s approach. Subsequent blogs will explore other Categories within the Protect Function shown below.


 


NIST CSF 2.0 Protect (PR) CategoriesNIST CSF 2.0 Protect (PR) Categories


 


Organizations establishing or reviewing their risk management practices can appreciate the importance of awareness. A proper identity and access control strategy starts by being risk informed, even if the risks presently outweigh the mitigations. Let’s press into the risk to grow as organizations and security practitioners.


 


Reese Bobby offered sage wisdom to his son Ricky in the theatrical number, Talladega Nights, “You gotta learn to drive with the fear. And there ain’t nothing more frightening than driving with a live cougar in the car.” Drive with the fear. Drive with a live cougar in the car.

ADX Web updates – May 2024

ADX Web updates – May 2024

This article is contributed. See the original author and article here.

Introducing an Enhanced Connections Explorer


We are pleased to introduce a new look and feel to our Connections explorer, designed to help you manage your list of data sources more efficiently.
While maintaining the familiar functionality of the old experience, this updated interface features a modern design, improved performance and an enhanced way to manage and view your Favorites.


 


We encourage you to turn on the new Connections pane to experience its smoother and more intuitive user experience –


 


Michal_Bar_0-1717916757851.png


 


If you are managing a long list of connections in your ADX web UI, you’ll notice a performance improvement as soon as you turn the new experience on.


Moreover, the new connection pane features


1 – multiple actions to help you manage the tree connections, accessible via an intuitive menu


2 – easy access to Get data actions


3 – easy access to tables’ data profile


 


Michal_Bar_1-1717916757856.png


 


 


Please share your thoughts and feedback regarding this new enhancement KustoWebExpFeedback@service.microsoft.com!


 


Easily Favorite and Find Your Important Dashboards


We are happy to announce that you can now add dashboards to your favorites list from two convenient locations: both from the catalog and, as a newly introduced feature, directly from the dashboard itself! This enhancement, driven by your feedback, makes it easier than ever to quickly mark and access your most-used dashboards.


Michal_Bar_2-1717916757860.png


 


 


Michal_Bar_3-1717916757874.png


 


 


Azure Data Explorer Web UI team is looking forward for your feedback in KustoWebExpFeedback@service.microsoft.com


You’re also welcome to add more ideas and vote for them here – https://aka.ms/adx.ideas


Read more:


PostgreSQL for your AI app’s backend | Azure Database for PostgreSQL Flexible Server

PostgreSQL for your AI app’s backend | Azure Database for PostgreSQL Flexible Server

This article is contributed. See the original author and article here.

Use PostgreSQL as a managed service on Azure. As you build generative AI apps, explore advantages of Azure Database for PostgreSQL Flexible Server such as integration with Azure AI services, as well as extensibility and compatibility, integrated enterprise capabilities to protect data, and controls for managing business continuity.


 


Main.png


 


Charles Feddersen, product team lead for PostgreSQL on Azure, joins host Jeremy Chapman to share how Flexible Server is a complete PostgreSQL platform for enterprise and developers.


 


 


Generate vector embeddings for data and images.


 


1.png


 


Enhance search accuracy and semantic matches. Watch how to use the Azure AI extension with Azure Database for PostgreSQL here.


 


 


Leverage the Azure AI extension.


 


2.png


 


Calculate sentiment and show a summarization of reviews using PostgreSQL. See it here.


 


 


Simplify disaster recovery for enterprise apps.


 


3.png


 


Achieve multi-zone high availability, zero data loss, and planned failover with GeoDR.


 


 


Watch our video here:


 


 







QUICK LINKS:


00:00 — Azure Database for PostgreSQL Flexible Server
00:51 — Microsoft and PostgreSQL 
01:40 — Open-source PostgreSQL
03:18 — Vector embeddings for data
04:32 — How it works with an app
06:59 — Azure AI Vision
08:14 — Azure AI extension using PostgreSQL
09:37 — Text generation using Azure AI extension
10:30 — High availability and disaster recovery|
12:45 — Wrap up


 


 


Link References


Get started with the Azure Database for PostgreSQL flexible server at https://aka.ms/postgresql


Stay current with all the updates at https://aka.ms/AzurePostgresBlog


 


 


Unfamiliar with Microsoft Mechanics?


As Microsoft’s official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.



 


 


Keep getting this insider knowledge, join us on social:











Video Transcript:


– Postgres is one of the most popular open-source databases in use today, and with its built-in vector index, plays a vital role in powering natural language generative AI experiences by searching across billions of data points to find similarity matches to support the generation of more accurate responses. But did you know that you can also use Postgres as a managed service on Azure? Today, in fact, as you build generative AI apps, we’re going to explore Azure Database for Postgres flexible server and the unique advantages such as integration with Azure AI services, as well as extensibility and compatibility, integrated enterprise capabilities to protect your data, controls for managing business continuity and more. And to walk us through all this, I’m joined, once again, by Charles Feddersen who leads the product team for Postgres on Azure. Welcome back to the show.


 


– Thanks for having me back, Jeremy. It’s great to be here.


 


– And it’s great to have you back on. You know, before we get into this, it’s probably worth explaining how Microsoft’s role is as part of the Postgres community. We’re not just putting an instance of Postgres on Azure, right?


 


– Yeah, what a lot of people don’t realize actually is Microsoft is a really significant contributor to Postgres, both major contributions in open-source Postgres and the surrounding ecosystem of features. We’ve contributed to many of the features that you’re probably using every day in Postgres, which include optimizations that speed up queries over highly petitioned tables. Or perhaps the single largest contribution we’re making to Postgres is to enable asynchronous and direct I/O for more efficient read and write operations in the database. We’ve learned a lot from running really demanding Postgres workloads in Azure, and this has inspired many of the performance optimizations that we’ve contributed upstream to open-source Postgres, so that everybody benefits.


 


– So given the pace of innovation then for the open-source community with Postgres, how do we make sure that, on Azure, we’ve got all the features and that they’re compatible with Azure Database for Postgres?


 


– Well, the first thing I really want to emphasize is that it’s pure open-source Postgres, and that’s by design. And this means you can run normal tools like pgAdmin, as you can see here. And there’s a really high level of compatibility with Postgres throughout the stack. And we ship new major versions of Postgres on Azure within weeks of the community release, which lets you test those latest features really quickly. Flexible service supports over 60 of the most common extensions, including PostGIS for geospatial workloads and Postgres FDW, which allows you to access data in external Postgres service. It also supports a great community-built extension called pgvector that enables Postgres to store index and query embeddings. And last year, we added the Azure AI extension, which provides direct integration between Postgres and the Azure OpenAI Service to generate vector embeddings from your data. And it also enables you to hook into capabilities like sentiment analysis, summarization, language detection and more. In fact, Azure AI support for Postgres is a major advantage of running Postgres on Azure. And this is in addition to several enterprise capabilities, such as built-in support for Microsoft Entra’s identity and access management, as well as broader security controls, like networking over private endpoints to better protect your data in transit, along with Key Vault encryption, using your own keys, including managed hardware security modules, or HSM, and more.


 


– Right, and this means basically that your Postgres implementation is natively integrated with your security policies for enterprise workloads, but you also mentioned that AI is a major benefit here in terms of Postgres on flexible server in Azure. So can you show us or walk through an example?


 


– Sure. Let me walk you through one using a travel website where the Azure AI extension has been used to generate vector embeddings for data for the travel site. And this also works for images where we can use the Azure AI Vision service to convert images to text and vectorize that information, all of which is stored in index in Postgres flexible server. And if you’re new to vectors, they’re a coordinate-like way to refer to chunks of data in your database and used for search for semantic matches. So when users submit natural language searches, those two are converted into vector embeddings. And unlike traditional keyword searches, similarity lookups find the closest semantic meaning between the vector embeddings from the user’s prompt and the embeddings stored in the database. Now additionally, the travel website uses Azure OpenAI’s GPT large language model itself to generate natural language responses using the data presented from Postgres as its context. So let’s try this out with a real app. Here’s our travel website and I’m going to book a much needed vacation. So I’ll search for San Diego and I’ve got over 120 accommodation options that I need to scroll through or filter. But now, I’m also traveling with my dog Mabel as well. So I need to find places where she can also stay. I’m going to add, allow small dogs to my search and this is going to use semantic search with embeddings to find suitable accommodations. And now, we’re down to about 90 results. So let’s look at the code to see how this works. Now, to perform the semantic similarity searches, we first need to generate text embeddings stored in a vector type in Postgres. I’ll create a new generator column of type vector and name it lodging_embedding. And this is going to store the text embeddings in our lodgings table that are based on the text descriptions column. Every time a new record is inserted, the Azure AI extension will call the OpenAI embedding model ada-002, pass the description text and return the embedding to stored. So I’ll run that query and now I’ll add an index to this new column to improve query efficiency. This is a special vector index called hnsw. It’s not your regular B-tree. And so I’ll run that and now we can do a test query against the embeddings. So I’ll switch to the vector similarity tab. And this query does a couple of interesting things. If you look at the order by clause, you can see that we’re ordering by the result of the comparison between the lodging_embedding column and the embedding we dynamically created from the search term to find the best result for allow small dogs. Now, we’re also using the PostGIS extension to add geospatial capabilities to find relevant lodging within 30 miles of a point of interest in San Diego. So I’ll run this query and you can see the top six results within 30 miles of a point of interest, ranked in order of the best semantic results for my small dog.


 


– So I get it, instead of creating another table or database, what you’re showing here is actually that Postgres provides a native type for embedding, so that you can actually incorporate your semantic search into your existing relational SQL workload.


 


– Exactly, and that’s the power of it. You don’t need a different database to handle embeddings. If you’ve got any existing Postgres apps, adding embeddings and semantic search and flexible server is as easy as adding a column and running a SQL function to call the Azure OpenAI service. So let’s go back to our hotel booking example. We also want to book a room with a beach view. I’ll add that to the search and how this works as I’m going to show you next is really cool. So I’ll head back over to a notebook and I’ve got one of the images from a property listing. Let’s take a look at the notebook cell. I can use the Azure AI Vision service to extract the embeddings from this image. And if I run this, you could see the embedding has been created and I could go ahead and store that in Postgres as well. And if we check our app again, you can see that we’re doing a text search for beach view, which is actually returning property images with a beach visible from the rooms. And the results are further refined with the suitability for my small dog. And as we can see on the left, it’s in the right distance range, within 30 miles of San Diego, which we’ve specified using geospatial in Postgres. And the amazing thing is we do it all with OpenText search, which is infinitely flexible, and not predefined filters. So I don’t need to hunt around for that often-hidden pets allowed filter.


 


– And the neat thing here is, as you mentioned, all of this is happening at the database layer, because we’ve actually converted all the text and all the images into vector embeddings, as part of data ingest and that’s all using Azure AI services.


 


– That’s right. That’s exactly right. And next, I’ll show you how you can make the search experience even richer by bringing Azure AI to summarize reviews and measure sentiment on a property. One of the most time-consuming parts of finding a great place to stay is reading the reviews. Here, we can use the Azure AI extension to calculate the sentiment and show a summarization of the reviews using Postgres. This is the output of the Coastal View College, with a 98% favorable sentiment and summary of reviews. So let’s take a look at the code. In this query, you can see we’re calling the azure_cognitive.analyze_sentiment function and passing the review_text that we want to score. I’ll run that and here you can see a positive sentiment of 98% returns. Now I’ll switch to the summary example. It’s a similar query pattern, except this time, we’re using the summarize_abstractive function to summarize the reviews into a small amount of easily-consumable text. So I’ll run this query, and here, you can see that summarized text,


 


– Right, and what you’ve shown here is more than just using embeddings, but also how the database can leverage other Azure capabilities to improve your app.


 


– That’s right. I’ve shown SQL queries that are returning results directly from the AI services, but alternatively, you could return those and store them in Postgres to reuse later. It’s really up to you, as a developer, about how you want to architect your app. Flexible server with the Azure AI extension just makes it easy to do it all using SQL. Now let’s move on to text generation, which is another area where we can use the Azure AI extension. I’m back in the website and I’ve selected the Coastal View Cottage for my stay. On the right, I can ask a freeform question about the property, but I’ve got a suggested prompt to look for hidden fees. These always seem to get me. So here, we’re using the Davinci model in the Azure OpenAI service to generate a response and it’s found a hidden fee buried in the fine print. So moving back to VS Code, I’ll run another query with the hidden fees prompt and I’ll capture those results. Now that I have the relevant context from the database, I’ll pass that to the Azure OpenAI Service Completion API and the prebuilt Davinci model to compose a response based on the results I took from the database. And this is how everything works.


 


– And this is a really great example of harnessing all of the AI capabilities. But something else that’s really important for an enterprise app is high availability and also disaster recovery.


 


– It is, and flexible server has all of those covered as well. This includes multi-zone high availability with zero data loss, zero redundant backups across regions, and recently we announced the general availability of planned failover, GeoDR. Here’s how you can configure that. I’m going to start in the portal on the Overview blade, and you can see I’ve got the Postgres flexible server called geodr running in the East US 2 region. I’ll scroll down on the left nav panel and head over to Replication where I’ve got two options: here to either create an endpoint, or create a read replica. Let’s create the read replica first. I’ll enter the replica server name and I’ll go create that in Australia Southeast, because that’s pretty much as far from East US 2 as you can get. I’ll click Review and create, and that’s now submitted. So once the replica is created on the other side of the planet, I need to create a virtual endpoint, which gives me a single endpoint for my application, so that when I do fail over, I don’t need to make any application changes to update connection strings. This time, I’ll create an endpoint. I’ll head over to the right panel and give it a name geodrvip, and you can see that the name has been appended to each of the writer and reader endpoint names below. And the reader server is the replica I just created. I’ll hit Create. And now, you can see I’ve got my virtual endpoint. So let’s test the failover using promotion. I’ll click the small Promote icon next to my replica server name. Now I’ve got some options. I can either promote this to the primary server, which means I reverse the roles of my two servers, that the replica becomes the writer, and the current writer becomes the replica. Or alternatively, I can promote this server to Standalone. I can also select if this as Planned, which means all data is synchronized to the replica prior to failover, or Forced, which executes immediately and doesn’t wait for the asynchronous replication to finish. I’ll leave everything as is and I’ll click Promote. And now, once this is finished, my geodr server that was the primary is now the replica under the reader endpoint and geodrausse is now the primary.


 


– Okay, so now you’ve got all your enterprise-grade data protections in place. You’ve got native vector search support and also GenAI capabilities for your apps, all powered by Postgres flexible server on Azure on the backend. So what’s next?


 


– So I’ve shown you how Flexible Server is a complete Postgres platform for enterprise and developers, and it’s only going to get better. We’ve got really big plans for the future, so stay tuned.


 


– So for everyone who’s watching right now, what do you recommend for them to get started?


 


– So to get started with the Azure Database for Postgres flexible server, go to aka.ms/postgresql, and to stay current with all the updates that we’re constantly shipping, check out our blog at aka.ms/AzurePostgresBlog.


 


– Thanks so much for joining us today, Charles. Always great to have you on to share all the updates to Postgres. Looking forward to having you back on the show. Of course, keep checking back to Microsoft Mechanics. We’ll see you next time and thanks for watching.