Accelerated Networking on HB, HC and HBv2

by | Jan 18, 2021 | Technology | 0 comments

This article is contributed. See the original author and article here.

Azure Accelerated Networking is now enabled on modern Azure HPC VM Sizes – HB, HC and HBv2. Accelerated Networking enables Single Root IO Virtualization (SR-IOV) for a VM’s SmartNIC resulting in throughput of up to 30 gigabits/sec, and lower and more consistent latencies.


 


Note that this enhanced Ethernet capability does not replace the dedicated InfiniBand network included with these modern HPC VM types. Instead, Accelerated Networking simply enhances the functions that the dedicated Ethernet interface has always performed such as loading OS images, connectivity to Azure Storage resources, or communicating with other resources in a customer’s VNET.


 


When enabling Accelerated Networking on supported Azure HPC VMs it is important to understand the changes you will see within the VM and what those changes may mean for your HPC workload:


 


A new network interface:


As the front-end ethernet NIC is now SR-IOV enabled, it will show up as a new network interface. For example, following is a screenshot from Azure HBv2 VM instance.


 


ibv_devinfo.PNG


Figure 1: “mlx5_0” is the ethernet interface, and “mlx5_1” is the InfiniBand interface.


 


 


ibstat.PNG


Figure 2: InfiniBand interface status (mlx5_1)


 


Changes to MPI launcher scripts:


Most MPIs do not need any changes to MPI launcher scripts. However, certain MPI libraries, especially those using UCX without a Libfabric layer, may need the InfiniBand device to be specified explicitly because of the additional network interface introduced with Accelerated Networking.


 


For example, OpenMPI using UCX (pml=ucx) fails by default as it tries to use the “mlx5_0” interface (i.e. the Accelerated Networking NIC) when what it requires is really the “mlx5_1” (i.e. the InfiniBand NIC).


 


This can be addressed by explicitly specifying the correct interface (UCX_NET_DEVICES=mlx5_1:1).


 


Below are the environment parameters for specifying the interface names for various MPI libraries:


 


MVAPICH2: MV2_IBA_HCA=mlx5_1 


 


OpenMPI & HPC-X: UCX_NET_DEVICES=mlx5_1:1


 


IntelMPI (mlx provider): UCX_NET_DEVICES=mlx5_1:1


 

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Submit a Comment