Azure Managed Lustre with Automatic Synchronisation to Azure BLOB Storage

by Contributed | Nov 30, 2023 | Technology

This article is contributed. See the original author and article here.

Introduction

This blog post walks through how to setup an Azure Managed Lustre Filesystem (AMLFS) that will automatically synchronise to an Azure BLOB Storage container. The synchronisation is achieved using the Lustre HSM (Hierarchical Storage Management) interface combined with the Robinhood policy engine and a tool that reads the Lustre changelog and synchronises metadata with the archived storage. The lfsazsync repository on GitHub contains a Bicep template to deploy and setup a virtual machine for this purpose.

Disclaimer: The lfsazsync deployment is not a supported Microsoft product you are responsible for the deployment and operation of the solution. There are updates that need applying to AMLFS that will require a Support Request to be raised through the Azure Portal. These updates could effect the stabaility of AMLFS and customer requiring the same level of SLA should speak to their Microsoft representative.

Initial Deployment

The following is required before running the lfsazsync Bicep template:

• Virtual Network


• Azure BLOB Storage Account and container (HNS is not supported)


• AMLFS deployed without HSM enabled

The lfsazsync repository contains a test/infra.bicep example to create the required resources:

To deploy, first create a resource group, e.g.

TODO: set the variables below

resource_group=

location=

az group create –name $resource_group –location $location

Then deploy into this resource group:

az deployment group create –resource-group $resource_group –template-file test/infra.bicep

Note: The bicep file has parameters for names, ip ranges etc. that should be set if you do not want the default values.

Updating the AMLFS settings

Once deployment is complete, navigate to the Azure Portal, locate the AMLFS resource and click on “New Support Request”. The following shows the suggested request to get AMLFS updated:

The lctl commands needed are listed here.

Deploying Azure BLOB Storage Synchronisation

The lfsazsync deployment sets up a single virtual machine for all tasks. The HSM copytools could be run on multiple virtual machines to increase transfer peformance. The bandwidth for archiving and retrieval is constrained to approximately half the network bandwidth available to the virtual machine. It is important to note that the same network will be utilized for both accessing the Lustre filesystem and accessing Azure Storage. This should be considered when deciding the virtual machine size. The virtual machine sizes and expected network performance is available here.

The Bicep template has the following parameters:

The SAS key can be generated using the following Azure CLI command:

# TODO: set the account name and container name below

account_name=

container_name=

start_date=$(date -u +”%Y-%m-%dT%H:%M:%SZ”)

expiry_date=$(date -u +”%Y-%m-%dT%H:%M:%SZ” –date “next month”)
az storage container generate-sas

   –account-name $account_name

   –name $container_name

   –permissions rwld

   –start $start_date

   –expiry $expiry_date

   -o tsv

The following Azure CLI command can be used to get the subnet ID:

# TODO: set the variable below

resource_group=

vnet_name=

subnet_name=

az network vnet subnet show –resource-group $resource_group –vnet-name $vnet_name –name $subnet_name –query id –output tsv

The following Azure CLI command can be used to deploy the Bicep template (as an alterative to setting environment variables, the parameters could be set in a parameters.json file):

# TODO: set the variables below

resource_group=

subnet_id=

vmsku=”Standard_D32ds_v4″

admin_user=

ssh_key=

lustre_mgs=

storage_account_name=

storage_container_name=

storage_sas_key=

ssh_port=

github_release=”v1.0.1″

os=”almalinux87″

az deployment group create

    –resource-group $resource_group

    –template-file lfsazsync.bicep

    –parameters

        subnet_id=”$subnet_id”

        vmsku=$vmsku

        admin_user=”$admin_user”

        ssh_key=”$ssh_key”

        lustre_mgs=$lustre_mgs

        storage_account_name=$storage_account_name

        storage_container_name=$storage_container_name

        storage_sas_key=”$storage_sas_key”

        ssh_port=$ssh_port

        github_release=$github_release

        os=$os

After this call completes the virtual machine will be deployed although it will take more time to install and import the metadata from Azure BLOB storage into the Lustre filesystem. The progress can be monitored by looking at the /var/log/cloud-init-output.log file on the virtual machine.

Monitoring

The install will set up three systemd services for lhsmd, robinhood and lustremetasync. The log files are located here:

• ‘lhsmd’: /var/log/lhsmd.log


• ‘robinhood’: /var/log/robinhood*.log


• ‘lustremetasync’: /var/log/lustremetasync.log

Default archive settings

The synchronisation parameters can be controlled through the Robinhood config file, /opt/robinhood/etc/robinhood.d/lustre.conf. Below are some of the default settings and their locations in the config file:

To update the config file, edit the file and then restart the robinhood service, systemctl restart robinhood.

The lustremetasync service is processing the Lustre ChangeLog continuously. Therefore, actions will happen immediately unless there is a lot of IO all at once where it may take a few minutes to catch up. The following operations will be handled:

• 
  

  Create/delete directories

  
  

  Directories are created in BLOB storage as an empty object with the name of the directory. There is metadata on this file to indicate that it is a directory. The same object is deleted when removed on the filesystem.

  
  


• 
  

  Create/delete symbolic links

  
  

  Symbolic links are create in BLOB storage as an empty object with the name of the symbolic link. There is metadata on this file to indicate that it is a symbolic link and this contains the path that it is linking to. The same object is deleted when removed on the filesystem.

  
  


• 
  

  Moving files or directories

  
  

  Moving files or directories requires everything being moved to be restored to the Lustre filesystem. The files are then marked as dirty in their new location and the existing files are deleted from BLOB storage. Robinhood will handle archiving the files again in their new location.

  
  


• 
  

  Updating metadata (e.g. ownership and permissions)

  
  

  The metadata will only be updated for archived files that isn’t modified. Modified files will have the metadata set when Robinhood updated the archived file.

  
  

References

• Azure Managed Lustre File System Documentation


• GitHub repositories
  
  
  
  • lfsazsync
  
  
  • Robinhood
  
  
  • lemur
  
  
  
  

Defender EASM – Performing a Successful Proof of Concept (PoC)

by Contributed | Nov 29, 2023 | Technology

This article is contributed. See the original author and article here.

Welcome to an introduction of the concepts and simple approach required for executing a successful Proof of Concept (PoC) for Microsoft Defender External Attack Surface Management (Defender EASM). This article will serve as a high-level guide to help you execute a simple framework for evaluating Defender EASM, and other items to consider when embarking on the journey to understand the Internet exposed digital assets that comprise your external attack surface, so you can view risks through the same lens as a malicious threat actor.

Planning for the PoC 

To ensure success, the first step is planning. This entails understanding the value of Defender EASM, identifying stakeholders who need to be involved, and scheduling planning sessions to determine use cases & requirements and scope before beginning. 

For example, one of the core benefits of the Defender EASM solution is that it provides high value visibility to Security and IT (Information Technology) teams that enables them to: 

• Identify previously unknown assets 


• Prioritize risk 


• Eliminate threats 


• Extends vulnerability and exposure control beyond the firewall 

Next, you should identify all relevant stakeholders, or personas, and schedule in 1-2 short planning sessions to document the tasks and expected outcomes, or requirements. These sessions will establish the definition of success for the PoC.  

Who are the common stakeholders that should participate in the initial planning sessions? The answer to that question will be unique to each organization, but some common personas include the following: 

• Vulnerability Management Teams 


• IT personnel responsible for Configuration Management, Patching, Asset Inventory Databases 


• Governance, Risk, & Compliance (GRC) Teams 

• (Optional) GRC aligned Legal, Brand Protection, & Privacy Teams 


• Internal Offensive Penetration Testing and Red Teams 


• Security Operations Teams 


• Incident Response Teams 


• Cyber Threat Intelligence, Hunting, and Research Teams 

Use Cases & Requirements 

Based on the scope, you can begin collaborating with the correct people to establish use cases & requirements to meet the business goals for the PoC. The requirements should clearly define the subcomponents of the overarching business goals within the charter of your External Attack Surface Management Program. Examples of business goals and high-level supporting requirements might include: 

• Discover Uknown Assets 

• Find Shadow IT 

• Discover Abandoned Assets 

• Resulting from Mergers, Acquistions, or Divestitures 


• Insufficient Asset Lifecycle Management in Dev/Test/QA Environments  

• Identification of Vulnerabilities 

• Lack of Patching or Configuration Management 

• Assignment of Ownership to Assets 

• Line of Business or Subsidiary 


• Based on Geographic Location 


• On-Prem vs Cloud 

• Reporting, Automation, and Defender EASM Data Integrations 

• Data Connector integration with Log Analytics or Kusto 

• Use of a reporting or visualization tool, such as PowerBI 


• Logic Apps to automate management of elements of your attack surface 

Prerequisites to Exit the Planning Phase 

• Completion of the Planning Phase! 


• Configure an Azure Active Directory or personal Microsoft account. Login or create an account here. 


• Set up a Free 30-day Defender EASM Trial 

– Visit the following link for information related to setting up your Defender EASM attack surface today for free. 

• Deploy & Access the Defender EASM Platform 

– Login to Defender EASM 

– Follow the deployment Quick Start Guide 

Measuring Success?

Determining how success will establish the criteria for a successful or failed PoC. Success and Acceptance Criteria should be established for each requirement identified. Weights may be applied to requirements, but measuring success can be as simple as writing out criteria as below:

Requirement: Custom Reporting

Success Criteria: As a vulnerability manager, I want to view a daily report that shows the assets with CVSSv2 and CVSSv3 scores of 10.

Acceptance Criteria:

• Data must be exported to Kusto


• Data must contain assets & CVSS (Common Vulnerability Scoring System) scores


• Dashboards must be created with PowerBI and accessible to user


• Dashboard data must be updated daily

Validation: Run a test to validate that acceptance criteria has been met.

Pass / Fail: Pass

Executing the PoC 

Implementation and Technical Validation 

We will now look at five different use cases & requirements, define the success and acceptance criteria for each, and validate that the requirements are met by observing the outcome of each in Defender EASM. 

Use Case 1: Discover Unknown Assets, Finding Shadow IT 

Success Criteria: As a member of the Contoso GRC team, I want to identify Domain assets in our attack surface that have not been registered with the official company email address we use for domain registrations.

Acceptance Criteria: 

• Defender EASM allows for searches of Domain WHOIS data that returns the “Registrant Email” field in the result set.  

Validation: 

1. Click the “Inventory” link on the left of the main Defender EASM page. 

Figure: Launch the inventory query screen

2. Execute a search in Defender EASM that excludes Domains registered with our official company email address of ‘domainadmin@constoso.com’ and returns all other Domains that have been registered with an email address that contains the email domain ‘contoso.com’.

Figure: Query for incorrectly registered Domain assets

3. Click on one of the domains in the result set to view asset details. For example, “woodgrovebank.com” domain.


4. When the asset details open and confirm that the domain ‘woodgrovebank.com’ is in the upper left corner.


5. Click on the “Whois” tab.


6. Note that this Domain asset has been registered with an email address that does not match the corporate standard (i.e., “employeeName@contoso.com”) and should be investigated for the existence of Shadow IT.

Figure: WHOIS asset details

Resources:

• Understand asset details: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/understanding-asset-details


• Domain asset filters: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/domain-asset-filters


• Understanding WHOIS:  https://en.wikipedia.org/wiki/WHOIS

Use Case 2: Abandoned Assets, Acquisitions

Success Criteria: As a member of the Contoso Vulnerability Management team, who just acquired Woodgrove Bank, I want to ensure acquired web sites using the domain “woodgrovebank.com” are redirected to web sites using the domain “contoso.com”.  I need to obtain results of web sites that are not redirecting as expected, as those may be abandoned web sites.

Acceptance Criteria:

• Defender EASM allows for search of specific initial and final HTTP (Hypertext Transfer Protocol) response codes for Page assets


• Defender EASM allows for search of initial and final Uniform Resource Locator (URL) for Page assets

Validation:

1. Run a search in Defender EASM that looks for Page assets that have:
   
   
   
   1. Initial response codes that cause HTTP redirects (i.e., “301”, “302”)
   
   
   2. Initial URLs that contain “woodgrovebank.com”
   
   
   3. Final HTTP response codes of “200”
   
   
   4. Final URL, post HTTP redirect, that do not contain “contso.com”
   
   
   
   

Figure: Query for incorrect page redirection

2. Click one of the Page assets in the result set to see the asset details.

Figure: Page asset overview

1. Validate:
   
   
   
   1.  Initial URL contains “woodgrovebank.com”
   
   
   2. Initial response code is either “301” or “301”
   
   
   3. Final URL does not contain “contoso.com”
   
   
   4. Final response code is “200”
   
   
   
   

Resources:

• Asset details summary view: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/understanding-asset-details


• Defender EASM inventory filters overview: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/inventory-filters


• Page asset filters: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/page-asset-filters


• HTTP Response Codes: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

Use Case 3: Identification of Vulnerabilities, Lack of Patching or Configuration Management

Success Criteria: As a member of the Contoso Vulnerability Management team, I need the ability to retrieve a list of assets with high priority vulnerabilities and remediation guidance in my attack surface.

Acceptance Criteria:

• Defender EASM provides a dashboard of prioritized risks in my external attack surface


• Defender EASM provides remediation guidance for each prioritized vulnerability


• Defender EASM provides an exportable list of assets impacted by vulnerability

Validation:

1. From the main Defender EASM page, click “Attack Surface Summary” to view the “Attack Surface Summary” dashboard


2. Click the link that indicates the number of assets impacted by a specific vulnerability to view a list of impacted assets

Figure: Attack Surface Insights Dashboard

3. Validate that Defender EASM provides additional information about vulnerabilities and remediation guidance.


4. Click the link in the upper right corner titled “Download CSV report” and validate the contents within

Figure: Vulnerability remediation details

Resources:

• Understanding dashboards: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/understanding-dashboards


• Understanding CVEs: https://nvd.nist.gov/vuln

Use Case 4: Assignment of Ownership to Assets, Line of Business or Subsidiary

Success Criteria: As a member of the Contoso GRC team, I need the ability to assign ownership of assets to specific business units through, along with a mechanism to quickly visualize this relationship.

Acceptance Criteria:

• Defender EASM provides an approach to assigning ownership via labels


• Defender EASM allows users to apply labels to assets that meet specific indicators that indicate affiliation with a specific business unit


• Defender EASM provides the ability to apply labels in bulk

Validation:

1. Click the “Inventory” link on the left of the main Defender EASM page to launch the search screen


2. Run a search that returns all Page assets that are on the IP Block “10.10.10.0/24”. The Page assets on this network all belong to the Financial Services line of business, so it is the only indicator of ownership needed in this example.

Figure: Query to determine Page asset ownership by IP Block

3. Select all assets in the result set by clicking the arrow to the right of the checkbox as shown in the following image and choose the option for all assets.

Figure: Selecting assets for bulk modification

5. Click the link to modify assets, followed by the link to “Create a new label” on the blade that appears.


6. A new screen will appear that allows the creation of a label. Enter a descriptive “Label name”, an optional “Display name”, select a desired color, and click “Add” to finish creating a label.

Figure: Link to modify assets and create a label

Figure: Create label detail

7. After creating the label, you will be directed back to the screen to modify assets. Validate that the label was created successfully.


8. Click into the label text box to see a list of labels available to choose from and select the one that was just created.


9. Click “Update”

Figure: Label selected assets

10. Click the bell icon to view task notifications to validate the status of labels update.

Figure: View status of label update task

11. When the task is complete, run the search again to validate that labels have been applied to the assets owned by the Financial Services organization.

Figure: Query to validate labels have been applied to assets

Resources:

• Asset modification overview: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/labeling-inventory-assets


• Defender EASM inventory filters overview: https://learn.microsoft.com/en-us/azure/external-attack-surface-management/inventory-filters

Finishing the PoC

Summarize Your Findings

Identify how the Defender EASM solution has provided increased visibility to your organization’s attack surface in the PoC.

1. Have you discovered unknown assets related to Shadow IT?


2. Were you able to find potentially abandoned assets related to an acquisition?


3. Has your organization been able to better prioritize vulnerabilities to focus on the most severe risks?


4. Do you know have a better view of asset ownership in your organization?

Feedback?

We would love to hear any ideas you may have to improve our Defender EASM platform or where and how you might use Defender EASM data elsewhere in the Microsoft Security ecosystem or other security 3^rd party applications. Please contact us via email at mdesam-pm@microsoft.com to share any feedback you have regarding Defender EASM.

Interested in Learning About New Defender EASM Features?

 Please join our Microsoft Security Connection Program if you are not a member and follow our Private & Public Preview events. You will not have access to this exclusive Teams channel until you complete the steps to become a Microsoft Security Connection Program member. Users that would like to influence the direction/strategy of our security products are encouraged to participate in our Private Preview events. Members who participate in these events will earn credit for respective Microsoft product badges delivered by Credly.

Conclusion

You now understand how to execute a simple Defender EASM PoC, to include deploying your first Defender EASM resource, identifying common personas, how to set requirements, and measure success. Do not forget! – you can enjoy a free 30-day trial by clicking on the link below.

You can discover your attack surface discovery journey today for free.

Personal Desktop Autoscale on Azure Virtual Desktop generally available

by Contributed | Nov 28, 2023 | Technology

This article is contributed. See the original author and article here.

We are excited to announce that Personal Desktop Autoscale on Azure Virtual Desktop is generally available as of November 15, 2023! With this feature, organizations with personal host pools can optimize costs by shutting down or hibernating idle session hosts, while ensuring that session hosts can be started when needed.

Personal Desktop Autoscale

Personal Desktop Autoscale is Azure Virtual Desktop’s native scaling solution that automatically starts session host virtual machines according to schedule or using Start VM on Connect and then deallocates or hibernates (in preview) session host virtual machines based on the user session state (log off/disconnect).

The following capabilities are now generally available with Personal Desktop Autoscale:

• Scaling plan configuration data can be stored in all regions where Azure Virtual Desktop host pool objects are, including Australia East, Canada Central, Canada East, Central US, East US, East US 2, Japan East, North Central US, North Europe, South Central US, UK South, UK West, West Central US, West Europe, West US, West US 2, and West US 3. It needs to be stored in the same region as the host pool objects it will be assigned to, however, we support deploying session host virtual machines in all Azure regions.


• You can use the Azure portal, REST API, PowerShell to enable and manage Personal Desktop Autoscale.

The following capabilities are new in public preview with Personal Desktop Autoscale:

• Hibernation is available as a scaling action. With the Hibernate-Resume feature in public preview, you will have a better experience as session state persists when the virtual machine hibernates. As a result, when the session host virtual machine starts, the user will be able to quickly resume where they left off. More details of the Hibernate-Resume feature can be found here.

Getting started

To enable Personal Desktop Autoscale, you need to:

• Create a personal scaling plan.


• Define whether to enable or disable Start VM on Connect.


• Choose what action to perform after a user session has been disconnected or logged off for a configurable period of time.


• Assign a personal scaling plan to one or more personal host pools.

A screenshot of a scaling plan in Azure Virtual Desktop called “fullweek_schedule”. The ramp-down is shown as repeating every day of the week at 6:00 PM Beijing time, starting VM on Connect. Disconnect settings are set to hibernate at 30 minutes. Log off settings are set to shut down after 30 minutes.

If you want to use Personal Desktop Autoscale with the Hibernate-Resume option, you will need to self-register your subscription and enable Hibernate-Resume when creating VMs for your personal host pool. We recommend you create a new host pool of session hosts and virtual machines that are all enabled with Hibernate-Resume for simplicity. Hibernation can also work with Start VM on Connect for cost optimization.

You can set up diagnostics to monitor potential issues and fix them before they interfere with your Personal Desktop Autoscale scaling plan.

Helpful resources

We encourage you to learn more about setting up autoscale and review frequently asked questions for more details on how to use autoscale for Azure Virtual Desktop. You may also find these resources helpful:

• https://aka.ms/hibernate-resume


• Reduce Compute Costs by Pausing VMs (now in public preview)


• Microsoft Mechanics: Automated scaling, imaging & monitoring


• Azure Academy: Enabling and managing Personal Desktop Autoscale