Activity Logs Insights (Preview)

by Contributed | Apr 24, 2022 | Technology

This article is contributed. See the original author and article here.

Activity Logs Insights is a centralised place to see all of the activities done in a resource or resource group, information such as who administrators deleted, updated or created resources, and whether the activities failed or succeeded. This article explains how to enable and use Activity log insights.

• Note: This current version of Activity Log Insights is a preview, and offers basic Activity logs insights. The next version, planned for the upcoming months, will offer additional information and insights

To enable Activity Logs Insights, simply configure the Activity log to export to a Log Analytics workspace.

To view Activity logs insights on a resource group or a subscription level:

1. 
   

   In the Azure portal, select Monitor > Workbooks.

   
   


2. 
   

   Select Activity Logs Insights in the Insights section.

   
   


3.  At the top of the Activity Logs Insights page, select:
   
   
   
   1. One or more subscriptions from the Subscriptions dropdown.
   
   
   2. Resources and resource groups from the CurrentResource dropdown.
   
   
   3. A time range for which to view data from the TimeRange dropdown.
   
   
   
   

o To view Activity Logs Insights on a resource level:

1. 
   

   In the Azure portal, go to your resource, select Workbooks.

   
   


2. 
   

   Select Activity Logs Insights in the Activity Logs Insights section.

   
   

    

   
   

   At the top of the Activity Logs Insights page, select:

   
   
   
   
   1. A time range for which to view data from the TimeRange dropdown.
   
   
   
   

• To learn more about Activity Logs Insights, see this article.

Build AlphaFold2 cluster on Azure CycleCloud

by Contributed | Apr 23, 2022 | Technology

This article is contributed. See the original author and article here.

Since released from July last year, AlphaFold2 protein folding algorithm is often used by more researchers and companies to drive more innovations for molecular analysis, drug discovery & etc. To build an AlphaFold2 computing cluster rapidly on the cloud will be the necessary step to leverage agility of cloud computing without CAPEX ahead.

Azure HPC stack has complete portfolio suitable for running AlphaFold2 in large scale, including GPU, storage and orchestrator service. This blog brings detailed steps of building AlphaFold2 HPC cluster on Azure to fasten your process.

Architecture

 Build Steps

1. Prerequisites




1. Check GPU quota and Azure NetApp Files(ANF) quota. SKU of NCsv3_T4 will be used in this building and NV_A10_v5 SKU (in preview) will also be suitable in the next.


2. Create a storage account with unique name (eg. saAlphaFold2 ) for CycleCloud using.


3. Prepare a SSH key pair in Azure portal.


4. Determine your working region with consideration of ANF service availability (eg. Southeast Asia).


5. Create a resource group in selected region (eg. rgAlphaFold).


6. Set Azure Cloud Shell ready for use.




2. Build CycleCloud environment following ARM template method. Set the VNet name as “vnetprotein“. Use the “saAlphaFold2” as the related storage account. After all the resources are built, you can find the CycleCloud UI portal address in console “Home->Virtual Machines->cyclecloud->Overview->DNS name”. Go through the first login process using your username and password.


3. Config ANF storage. Follow the steps to set up an ANF volume. Consider the dataset size of AlphaFold2, suggest to set the capacity pool and volume size as 4TB at least. Set the volume name as “volprotein” and create a dedicate subnet with CIDR “10.0.2.0/24” in Visual Network “vnetprotein“. In “Protocol” settings, set file path also as “volprotein” and select “NFSv4.1”. After volume is ready, remember the “Mount path” info like “10.0.2.4:/volprotein“.


4. Prepare the VM image.
   
   
   
   1. Boot a VM using the “CentOS-based 7.9 HPC – x64 Gen2” marketplace image and change the OS disk size as 128GB.
   
   
   2. Connect the VM by SSH and install AlphaFold2 components using below commands.

      sudo  yum install epel-release python3 -y
      sudo yum install aria2 -y
      sudo yum-config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
      sudo yum repolist -v
      sudo yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.4.3-3.1.el7.x86_64.rpm
      sudo yum install docker-ce -y
      sudo systemctl --now enable docker
      distribution=$(. /etc/os-release;echo $ID$VERSION_ID) 
         && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
      sudo yum clean expire-cache
      sudo yum install -y nvidia-docker2
      sudo systemctl restart docker
      sudo usermod -aGdocker $USER
      newgrp docker
      docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
      sudo su
      cd /opt
      git clone https://github.com/deepmind/alphafold.git
      cd alphafold/
      sudo docker build -f docker/Dockerfile -t alphafold .
      sudo pip3 install -r docker/requirements.txt​

      Check the “docker images” to confirm the “alphafold:latest” is ready in the list.

   
   
   3. Build the custom image. Keep in the same SSH window and execute as below and go ahead with a ‘y’ confirmation. 

      sudo waagent -deprovision+user​

      Back to Cloud Shell. Execute these commands to produce the custom image. 

      export myVM=vmImgAlpha
      export myImage=imgAlphaFold2
      export myResourceGroup=Rampup-study
      az vm deallocate --resource-group $myResourceGroup --name $myVM
      az vm generalize --resource-group $myResourceGroup --name $myVM
      az image create --resource-group $myResourceGroup --name $myImage --source $myVM --hyper-v-generation V2​

      After accomplished, find the image’s “Resource ID” in console “Home->Images->Properties” page and remember it for further usage, which the form is as “/subscriptions/xxxx-xxxx-x…/resourceGroups/…/providers/Microsoft.Compute/images/imgAlphaFold2“.

   
   
   
   


5. Create HPC cluster for Alphafold2. 
   
   
   
   1. Create a new cluster in CycleCloud and select “Slurm” as the scheduler type. Set parameter as below with other as is. Save the configuration then.
      
      
      
      • “Require setting” page – HPC VM Type: Standard_NC8as_T4_v3, Max HPC Cores: 24, Subnet ID: vnetprotein-compute.
      
      
      • “Network Attached Storage” page – Add NFS Mount: clicked, NFS IP: 10.0.2.4, NFS Mount point: /volprotein, NFS Export Path: /volprotein.
      
      
      • “Advanced Settings” page – Scheduler & HPC OS both with “Custom image” option clicked and stuff with custom image resource ID string in step 4.
      
      
      
      
   
   
   2. Start the cluster and wait several minutes to wait cluster in ready.
   
   
   3. Login scheduler. Below steps aim to prepare dataset. Total size of the Alphafold2 dataset is ~2.2TB. Suggest to execute each download sentence in download_all_data.sh if you want to save some time, such as download_pdb70.sh, download_uniref90.sh & etc.. Dataset preparation may need several hours as expected.

      mkdir /volprotein/AlphaFold2
      mkdir /volprotein/AlphaFold2/input
      mkdir /volprotein/AlphaFold2/result
      sudo chmod +w /volprotein/AlphaFold2
      /opt/alphafold/scripts/download_all_data.sh /volprotein/AlphaFold2/​

   
   
   
   


6. Run samples
   
   
   
   1. A sample Slurm job script is as below. Save it as run.sh.

      #!/bin/bash
      #SBATCH -o job%j.out
      #SBATCH --job-name=AlphaFold
      #SBATCH --nodes=1
      #SBATCH --cpus-per-task=4
      #SBATCH --gres=gpu:1
      INPUT_FILE=$1
      WORKDIR=/opt/alphafold
      INPUTDIR=/volprotein/AlphaFold2/input
      OUTPUTDIR=/volprotein/AlphaFold2/result
      DATABASEDIR=/volprotein/AlphaFold2/
      
      sudo python3 $WORKDIR/docker/run_docker.py --fasta_paths=$INPUTDIR/$INPUT_FILE --output_dir=$OUTPUTDIR --max_template_date=2020-05-14 --data_dir=$DATABASEDIR --db_preset=reduced_dbs​

   
   
   2. Now we can submit the AlphaFold2 computing jobs! Submit this job with a test sample (*.fa or *.fasta) in /volprotein/AlphaFold2/input. At the first running, cluster need several minutes waiting compute nodes get ready. Parallel jobs can be submitted and will be running on different compute node according Slurm’s allocation. Then we can use “squeue” to check the Slurm queue status. Meanwhile, there are resource monitoring graphic in CycleCloud UI to grasp the performance status of this AlphaFold2 cluster. After certain job is done, check the info in .out file and the pdb result file in /volprotein/AlphaFold2/result.

      sbatch run.sh input.fa
      sbatch run.sh P05067.fasta​

      

   
   
   3. Tear down. When no need to use this cluster, directly delete the resource group “rgAlphaFold” will tear down the related resources in it.
   
   
   
   

Reference links

deepmind/alphafold: Open source code for AlphaFold. (github.com)

Azure CycleCloud Documentation – Azure CycleCloud | Microsoft Docs

Azure NetApp Files documentation | Microsoft Docs

New Reply-all Storm Protection Report, Settings UI, and Alert Policy

by Contributed | Apr 22, 2022 | Technology

This article is contributed. See the original author and article here.

The Reply-all Storm Protection feature in Exchange Online helps protect your organization from unwanted and disruptive reply-all storms. Last year we updated the feature to give admins the ability to customize key settings for reply-all storm detection and block duration, and in that announcement we noted that several additional updates were planned for a future date. We’re pleased to announce that we’re rolling out three updates:

1. A Reply-all Storm Protection Report to track reply-all storms and the messages blocked by the feature.


2. The ability to customize feature settings within the EAC (in addition to Remote PowerShell).


3. A mail flow system alert policy to notify admins when a reply-all storm hits your organization.

The report includes charts for detected reply-all storms and associated blocked messages, and a pop-out storm details panel available when you click on a storm name at the bottom of each chart. additional details about each storm, it also includes a CSV-exportable table with key details of each storm such as subject, original message, total messages, and message ID. The report is available in the new EAC under the Reports > Mail flow section. Here’s a sample report:

Roll-out of the report starts this week and should finish by the end of May for the WW environments (including GCC), with availability in the GCC-High environment expected by the end of July. More information about the report can be found in Reply-all storm protection report in the new EAC in Exchange Online.

It’s been possible to customize key settings for the Reply-all storm protection feature using Remote PowerShell for over a year. Today, we’re happy to announce that you can now also customize these settings in the new EAC under the Settings > Mail flow panel, available now, as shown below:

To change these settings, you must have permission to change Transport configuration information (e.g., Set-TransportConfig) as part of the Organization Transport Settings role group (and included as part of the Exchange Admin and Global Admin roles).

Lastly, we’re currently working on a mail flow system alert policy that will notify admins when a reply-all storm has been detected and at least one reply-all has been blocked. Like all mail flow alerts it will be customizable so you can configure who gets notified and other common alert parameters. When the Reply-all Storm Protection alert policy is release (by the end of July) it will appear in the Mail flow > Alert policies section of the new EAC.

We hope you find these new updates for the Reply-all Storm Protection feature useful, and we look forward to any feedback or suggestions for future updates you might have.

Exchange Transport Team