GitOps and secret management with AKS, Flux CD, SOPS, and Azure Key Vault

by Contributed | Apr 28, 2021 | Technology

This article is contributed. See the original author and article here.

Create a Kubernetes cluster

export RESOURCE_GROUP_NAME=gitops-demo-rg
export LOCATION=westeurope
export CLUSTER_NAME=GitOpsDemoCluster

az group create -n $RESOURCE_GROUP_NAME -l $LOCATION

az aks create -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME --enable-managed-identity

az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME

kubectl get nodes

NAME                                STATUS   ROLES   AGE     VERSION
aks-nodepool1-30631669-vmss000000   Ready    agent   5m27s   v1.19.9
aks-nodepool1-30631669-vmss000001   Ready    agent   5m30s   v1.19.9
aks-nodepool1-30631669-vmss000002   Ready    agent   5m29s   v1.19.9

Set up Flux CD

curl -s https://toolkit.fluxcd.io/install.sh | sudo bash

flux check --pre

► checking prerequisites
✔ kubectl 1.19.7 >=1.18.0-0
✔ Kubernetes 1.19.9 >=1.16.0-0
✔ prerequisites checks passed

git clone git@github.com:adrianmo/gitops-demo.git
cd gitops-demo

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<name-of-your-repo>
export CLUSTER_NAME=GitOpsDemoCluster

flux bootstrap github 
--owner=$GITHUB_USER 
--repository=$GITHUB_REPO 
--branch=main 
--path=./clusters/$CLUSTER_NAME

► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/adrianmo/gitops-demo.git"
✔ cloned repository
[...]
✔ kustomize-controller: deployment ready
✔ helm-controller: deployment ready
✔ all components are healthy

flux check

► checking prerequisites
[...]
► checking controllers
[...]
✔ all checks passed

git pull origin main

Set up the application

flux create kustomization demoapp 
  --namespace=flux-system 
  --source=flux-system 
  --path="./manifests" 
  --prune=true 
  --validation=client 
  --interval=5m 
  --export > ./clusters/$CLUSTER_NAME/demoapp-kustomization.yaml

mkdir manifests

cat > ./manifests/namespace.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: demoapp
EOF

cat > ./manifests/deployment.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demoapp
  namespace: demoapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demoapp
  template:
    metadata:
      labels:
        app: demoapp
    spec:
      containers:
        - name: demoapp
          image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
          ports:
            - containerPort: 80
              protocol: TCP
EOF

cat > ./manifests/service.yaml <<EOF
---
apiVersion: v1
kind: Service
metadata:
  name: demoapp
  namespace: demoapp
spec:
  type: ClusterIP
  selector:
    app: demoapp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
EOF

kubectl get kustomization -A
NAMESPACE     NAME          READY   STATUS                                                            AGE
flux-system   demoapp       True    Applied revision: main/a5f6b27feca2e1009afb474adc84c95c972018ad   10m
flux-system   flux-system   True    Applied revision: main/a5f6b27feca2e1009afb474adc84c95c972018ad   58m

kubectl -n demoapp get pod,deploy,svc
NAME                           READY   STATUS    RESTARTS   AGE
pod/demoapp-6b757bbc47-bx7rx   1/1     Running   0          1m
pod/demoapp-6b757bbc47-rh4db   1/1     Running   0          1m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demoapp   2/2     2            2           1m

NAME              TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)        AGE
service/demoapp   LoadBalancer   10.0.252.90   51.144.185.175   80:31122/TCP   1m

Secret management

RESOURCE_GROUP_ID=$(az group show -n $RESOURCE_GROUP_NAME -o tsv --query id)
AKS_RESOURCE_GROUP_NAME=$(az aks show -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME -o tsv --query nodeResourceGroup)
AKS_RESOURCE_GROUP_ID=$(az group show -n $AKS_RESOURCE_GROUP_NAME -o tsv --query id)
KUBELET_CLIENT_ID=$(az aks show -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME -o tsv --query identityProfile.kubeletidentity.clientId)

az role assignment create --role "Virtual Machine Contributor" --assignee $KUBELET_CLIENT_ID --scope $AKS_RESOURCE_GROUP_ID
az role assignment create --role "Managed Identity Operator" --assignee $KUBELET_CLIENT_ID --scope $RESOURCE_GROUP_ID

cat > ./manifests/aad-pod-identity.yaml <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: aad-pod-identity
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: aad-pod-identity
  namespace: aad-pod-identity
spec:
  url: https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
  interval: 10m
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: aad-pod-identity
  namespace: aad-pod-identity
spec:
  interval: 5m
  chart:
    spec:
      chart: aad-pod-identity
      version: 4.0.0
      sourceRef:
        kind: HelmRepository
        name: aad-pod-identity
        namespace: aad-pod-identity
      interval: 1m
  values:
    nmi:
      allowNetworkPluginKubenet: true
EOF

kubectl get helmrelease -A​

NAMESPACE          NAME               READY   STATUS                             AGE
aad-pod-identity   aad-pod-identity   True    Release reconciliation succeeded   14m

kubectl -n aad-pod-identity get pods

NAME                                   READY   STATUS    RESTARTS   AGE
aad-pod-identity-mic-5c9b5845c-4ktft   1/1     Running   0          14m
aad-pod-identity-mic-5c9b5845c-bskj8   1/1     Running   0          14m
aad-pod-identity-nmi-6h6rl             1/1     Running   0          14m
aad-pod-identity-nmi-76vk5             1/1     Running   0          14m
aad-pod-identity-nmi-tr9bk             1/1     Running   0          14m

az identity create -n SopsDecryptorIdentity -g $RESOURCE_GROUP_NAME -l $LOCATION

CLIENT_ID=$(az identity show -n SopsDecryptorIdentity -g $RESOURCE_GROUP_NAME -o tsv --query "clientId")
OBJECT_ID=$(az identity show -n SopsDecryptorIdentity -g $RESOURCE_GROUP_NAME -o tsv --query "principalId")
RESOURCE_ID=$(az identity show -n SopsDecryptorIdentity -g $RESOURCE_GROUP_NAME -o tsv --query "id")

export KEY_VAULT_NAME=GitOpsDemoKeyVault​

az keyvault create --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP_NAME --location $LOCATION

az keyvault key create --name sops-key --vault-name $KEY_VAULT_NAME --protection software --ops encrypt decrypt

az keyvault set-policy --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP_NAME --object-id $OBJECT_ID --key-permissions encrypt decrypt​

az keyvault key show --name sops-key --vault-name $KEY_VAULT_NAME --query key.kid

https://gitopsdemokeyvault.vault.azure.net/keys/sops-key/b7bc85c1a4ef4180be9d1de46725304c​

cat > ./clusters/$CLUSTER_NAME/sops-identity.yaml <<EOF
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: sops-akv-decryptor
  namespace: flux-system
spec:
  clientID: $CLIENT_ID
  resourceID: $RESOURCE_ID
  type: 0 # user-managed identity
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: sops-akv-decryptor-binding
  namespace: flux-system
spec:
  azureIdentity: sops-akv-decryptor
  selector: sops-akv-decryptor  # kustomize-controller label will match this name
EOF​

cat > ./clusters/$CLUSTER_NAME/flux-system-kustomization.yaml <<EOF
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - flux-system
patchesStrategicMerge:
  - sops-kustomize-patch.yaml
EOF

cat > ./clusters/$CLUSTER_NAME/sops-kustomize-patch.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kustomize-controller
  namespace: flux-system
spec:
  template:
    metadata:
      labels:
        aadpodidbinding: sops-akv-decryptor  # match the AzureIdentityBinding selector
    spec:
      containers:
      - name: manager
        env:
        - name: AZURE_AUTH_METHOD
          value: msi
EOF

We also have to tell the kustomize-controller that our app Kustomization needs to use SOPS as the decryption provider and therefore be able to decrypt certain fields of our manifests.

Update the kustomization YAML file in “clusters/$CLUSTER_NAME/demoapp-kustomization.yaml” and add the `spec.decryption` block as shown below.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: demoapp
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./manifests
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client
  decryption:
    provider: sops​

Apply the changes and check that the identity binding and patching have been applied successfully.

At this point, the kustomize-controller should be able to decrypt files encrypted with SOPS via our Key Vault key.

 

Encrypt secrets

Install SOPS locally following the instructions from their repository. Then create a “.sops.yaml” file to contain the SOPS configuration. We will tell SOPS to encrypt only the “data” and “stringData” blocks of our YAML files. Therefore, only the values of our Kubernetes secrets will be encrypted. If that rule is omitted, SOPS will encrypt all keys in our YAML files, which is not necessary.

Also, we need to tell SOPS to use our Azure Key Vault key to encrypt the files. Before running SOPS make sure you are logged in with a user that has encrypt access to the Key Vault key being used, otherwise the encryption will fail.

cat > .sops.yaml <<EOF
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    azure_keyvault: https://gitopsdemokeyvault.vault.azure.net/keys/sops-key/b7bc85c1a4ef4180be9d1de46725304c
EOF​

Create a temporary secret file which we won’t commit to the repository.

cat > ./secret.yaml <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: demoapp-credentials
  namespace: demoapp
type: Opaque
stringData:
  username: admin
  password: t0p-S3cr3t
EOF

SOPS will use the logged in user in AZ CLI, therefore, make sure the logged in user has “encrypt” and “decrypt” Key Permissions as shown below.

 

You can also add those permissions with the following commands.

SIGNED_IN_USER_OBJECT_ID=$(az ad signed-in-user show -o tsv --query objectId)
az keyvault set-policy --name $KEY_VAULT_NAME --resource-group $RESOURCE_GROUP_NAME --object-id $SIGNED_IN_USER_OBJECT_ID --key-permissions encrypt decrypt

And encrypt the secret.

sops --encrypt secret.yaml > ./manifests/secret.enc.yaml

The encrypted secret at “./manifests/secret.enc.yaml” will look something like this.

apiVersion: v1
kind: Secret
metadata:
    name: demoapp-credentials
    namespace: demoapp
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:21q4bo0=,iv:LOLxXQurjQR6cu9heQlZDdmhNgYO6VCBybbQHV6rO0w=,tag:58ep32CDrlCFuuDnD65VEQ==,type:str]
    password: ENC[AES256_GCM,data:oTZDkadQKL45dA==,iv:5VVbXC55xTVwH/n3t5gtKNtlkB3q7t8lW7Jw1czNSL0=,tag:WuqdubjTu6mQN5x1b3zDyw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv:
        - vault_url: https://gitopsdemokeyvault.vault.azure.net
          name: sops-key
          version: b7bc85c1a4ef4180be9d1de46725304c
          created_at: "2021-04-23T14:22:15Z"
          enc: KuFxRbcge198GU7hwHs078JNd_1EFtvcFqQ6bOLJDYMnWaW0kSbeD4DCxY0jX9MA17Rv3UMKHGfImgEbNfXGGIh7UucLPygpiuUyn9I73ClSQQ4trc4bD2yVkonCMwz5-0MiPVC3muhQpn3KjhThSucOgjhBnqQy_ymwTeUP9PWi1pSp1jc3S2BxQIuKy09-oEakQogU4BRy55219befizYN7EFe8mstSIkvpksqGxKccH6dQum2k-OqsBUH2jkxiVgi5CEU35COy0pNWVJpZGuOaDMkGGqo7lrT4XKEGxtFKvEDxr6bTfjjQafuuxW9-4a9ZtaBkHCKopk55R9dcQ
    hc_vault: []
    age: []
    lastmodified: "2021-04-23T14:22:18Z"
    mac: ENC[AES256_GCM,data:aw5mfREh5xdeiwbchkiiBS96tGuLJnEqme6VdDrPWKV9R0A4ATIM/1+HcbdAzGBXb9TmhO71hZMl3IvmX9DrNA/tvpPwFvLCkDfNhoWXJoXRRv6aRR7AJPlfcXkVMxxYaRDqz+ugAJkZG+5dhYeh1QAmiswjZOXaINEOw3Jf5dI=,iv:p/M2OhPdh2Naxu37Jt7EwiLf9Eb9OgExsmXX3hSUOJQ=,tag:fVqJ2jy++6GxHBPGXZHmHw==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.7.1

We can now delete the plain secret and push the encrypted secret to the repo.

rm secret.yaml
git add manifests/secret.enc.yaml
git commit -m "Add encrypted secret"
git push

Flux will read the encrypted secret, decrypt it using the identity and the key, and apply it.

kubectl describe secret -n demoapp demoapp-credentials

Output.

Name:         demoapp-credentials
Namespace:    demoapp
Labels:       kustomize.toolkit.fluxcd.io/checksum=c7c24c5836c9f935d9ab866ab9e31192bd98268e
              kustomize.toolkit.fluxcd.io/name=demoapp
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>

Type:  Opaque

Data
====
password:  10 bytes
username:  5 bytes

From this point on, Flux will keep our app up to date with the latest Kubernetes definitions in your repository, including secrets. You can find the code generated by this guide in the github.com/adrianmo/aks-flux GitHub repository.

 

Next steps

Automatic image updates

 

Flux has a couple of optional components called Image Automation Controllers, which can monitor a container registry and detect when new image tags are uploaded. Then, they can automatically update your deployments to roll out an update to the new container image. Check the official documentation to know more.

 

Keep Flux up to date

 

There are multiples ways to keep the Flux components up to date when new versions are released.

Flux’s system components are defined in the “./clusters/${CLUSTER_NAME}/flux-system/gotk-components.yaml” manifest file, therefore, if we regenerate that file with a newer version of Flux (i.e. running “flux install –export ./clusters/${CLUSTER_NAME}/flux-system/gotk-components.yaml”), the manifests will be updated with the new container images and configurations and the Flux system running in the cluster will apply those changes and update itself.

The above procedure can be turned into a scheduled CI workflow and create a Pull Request when changes are made to the manifests, giving us the possibility to review and approve the changes before they are applied. An example implementation of this workflow can be found here.

Notifications and monitoring

 

Since there are many operations that happen automatically, it can be become quite challenging to understand what is going on in our cluster. Therefore, it is crucial to know the state of our system at any time, but especially when things go south.

With Flux we can configure Notifications to forward events to collaboration and messaging apps like Microsoft Teams or Slack, and also to Git repositories in the form of commit statuses.

Flux also comes with a Monitoring stack composed of Prometheus, for metric collection, and Grafana dashboards, for displaying the control plane resource usage and reconciliation stats.

Mastering Configuration in Defender for Office 365 – Part One

by Contributed | Apr 28, 2021 | Technology

This article is contributed. See the original author and article here.

This blog is part one of a three-part series detailing the journey we’re on to simplify configuration of threat protection capabilities in Office 365 to enable best-in class protection for our customers.

Effective security is a never-ending battle to achieve balance between security and productivity. If we apply too many security controls to an environment, we limit the ability of its users to function efficiently. And if we err on the side of restraint, we do not hinder users in any way, but we leave the door open to threats. Email security is complex and ever-changing. With over 90 percent of threats surfacing through email, it’s critical that organizations are empowered to configure security tools in a way that works for their environment.

Configuration is key

We’re committed to offering Office 365 customers the best email protection by continually focusing on improving the effectiveness of our solutions both within Exchange Online Protection (EOP) as well as Defender for Office 365. EOP has a rich legacy of policy granularity and customizations that help customers meet their unique needs. As we’ve built and innovated on Microsoft Defender for Office 365, we have applied those same principles to the new advanced protection capabilities we offered as part of Defender for Office 365, while still respecting many of the EOP settings.

This deeply customizable protection stack within Office 365 has allowed customers over the years to implement policies and rules that fulfill an endless list of requirements. The drawback here, however, is that as customizations are added, they require regular review, upkeep, modifications, and even removal over time. In the absence of that continued focus, there is a high risk of creating an overall reduced state of protection.  And while that might sound counter-intuitive, we see this very often. Here are some examples of how these configurations can inadvertently get out of hand:

• An organization in Europe had configured 198 domains to be allowed to bypass our filters


• A firm in India had over 900 URLs stipulated to bypass by our detonation service per week


• An enterprise in Asia had over 50,000 known phishing URLs configured to bypass our filters

In each of these cases, the result was an increase in phishing campaigns making their way to end users. And these are just a few examples of what we see as a widespread problem – custom policies and configurations put in place with perhaps the best of intentions but without considering the immediate or long-term security impact of creating them or keeping them in place permanently.

Across Office 365, we estimate that 20% of phishing mails are delivered to user mailboxes as a result of poorly configured (often legacy) policies that haven’t been revisited for a long time. It was clear that we needed to help customers through this. It wasn’t sufficient that we educate customers of the problem, we had to actively help with getting customers to a more secure state. That started a series of efforts for the past many months that have resulted in capabilities, tools and changes in the product that we’ll walk you through in this blog series. But before we get into it, it might help to get a better appreciation for how the problem arises in the first place.

How did we get here?

The natural question to ask is, how did we arrive at a place where customer configuration could be a problem?

Historical settings can age

In some ways, Exchange Online represents the final frontier. The promise of the cloud is a world where upgrades to Exchange no longer occur every few years. Over the lifespan of Exchange, many customers have migrated with existing mail flow configurations and transport rules from Exchange 2010, to Exchange 2013, and ultimately ending up with Exchange Online in Office 365. Many of our customers reading this may have relied on Exchange versions long before Exchange 2010!

And these configurations and rules may have been implemented at a time where the worst thing that could happen as a result of an overly permissive policy was a spam email getting through. All of that has changed over the past few years.

New attack scenarios

Just as technology has evolved, so have attackers. A lot has changed since we first launched our advanced email security solution in 2015. Since then, email borne attacks have been increasing exponentially both in volumes and complexity. We’ve seen phishing evolve to become only the entry point for much more sophisticated attacks, like business email compromise. We’ve seen attackers pivot away from malware in favor of attacks that help them establish persistence through account compromise and external forwarding. We know that attackers are savvy cybercriminals that will continue to evolve their techniques to take advantage of email users.  And one common path they look to exploit are these aging and overly permissive controls or poorly protected pockets within the organization.

New security controls

As the threat landscape evolves, so do our protections. Microsoft Defender for Office 365 employs a multi-layered protection stack that is always being updated to meet the needs of our customers. As we introduce new capabilities and make improvements to existing ones, it’s important that our customers are able to take advantage of these capabilities. That sometimes requires frequent evaluation of settings to ensure the latest protections are turned on. Failing that discipline, it’s possible that the latest protections are not being applied to all users in the organization.

Naturally, these three challenges signify the importance of secure posture. It’s more important than ever that configuring protection against threats is easy to achieve and maintain.

Figure 1: The new and updated layers of the Defender for Office 365 protection stack

So how can we solve this problem?

Over the past many months, we’ve been on an aggressive journey to eliminate misconfigurations across Office 365 – to give customers the right tools to achieve secure posture simply and maintain these configurations over time. There are two broad categories of focus:

Eliminating overly permissive configurations

First, it’s critical that these (often) legacy settings or other inadvertent rules and policies don’t come in the way of us being able to keep users protected.

Preventing inadvertent gaps in protection coverage

Second, we want to make sure that organizations can easily protect all their users with the very best of protections that we offer as and when we make them available. This is critical in a fast-changing threat landscape where we’re constantly innovating to ensure users are protected.

As we’ve approached tackling both classes of problems, we’ve applied the following principles:

1. Give customers the awareness and tools to get secure


2. Actively help customers ‘get to secure’ through changes in the product


3. Help customers with the right tools/guardrails to stay secure.

Through this blog series we’ll show how we’re applying all three principles to help customers.

What we have accomplished so far

We’ve been hard at work over the last year to achieve these goals of raising awareness on configuration gaps and preventing these gaps from inhibiting effective threat protection. I want to share with you some of the enhancements we’ve released.

Preset Security Policies

In order to help customers understand the impact of misconfigurations, we needed to do something fundamental – we had to establish what the ideal configuration looked like. Last year we released preset security policies for Exchange Online Protection and Defender for Office 365. These policies provide a simplified method to apply all of the recommended spam, malware, and phishing policies to users across your organization. Since different organizations have different security needs, we released these presets in multiple variations, and allow customers to apply our standard or our strict presets to their users as they see fit.

We’ve seen tremendous adoption of preset security policies since they launched in 2020, with over 18,000 tenants enabling a preset policy in their environment. Preset security policies not only give customers a choice, but they also help them stay up to speed with changing recommendations as the threat landscape evolves. To learn more about preset security policies, check out our documentation.

Figure 2: Preset policies can be applied to users, groups, or domains.

Configuration Analyzer

Once we’d established the ideal configuration based on our own recommendations, we needed to give customers the ability to identify the instances where their configurations deviate from our recommended settings, and a way to adopt these recommendations easily.

In 2019, we launched ORCA, the Office 365 Recommended Configuration Analyzer. ORCA gives customers a programmatic way to compare their current configuration settings against recommendations via PowerShell. As a result of the overwhelming success of ORCA, last year we built Configuration Analyzer right into the product. Customers can now view policy discrepancies right from within the admin portal, and can even choose to view recommended adjustments to reach our standard or our strict recommendations.

We’ve seen incredible adoption of the configuration analyzer as well, with 290,000 policy changes made across more than 26,000 tenants since we launched the capability last year! With a few clicks, policies can be updated to meet the recommended settings, and as a result, it’s never been easier to keep email security configurations up to date. Learn more about configuration analyzer here.

Figure 3: Configuration Analyzer shows policies that do not meet our recommended settings.

Overrides Reports and Alerts

You’ll hear us refer to overrides frequently throughout this series. We define overrides as tenant level or user level configurations that instruct Office 365 to deliver mail even when the system has determined that the message is suspicious or contains malicious content. Examples of overrides could be an Exchange transport rule that bypasses filtering for a specific range of IP addresses, or a user level policy like an allowed sender or domain at the mailbox level.

The thing to understand about overrides is that they represent scenarios where policies are properly configured, but other settings have neutralized their effect. It’s important that we allow organizations to customize their Office 365 environment to meet their needs, but that doesn’t mean we feel comfortable allowing malicious content like malware or phish to land in the inbox of users.

We’ve added a view to the Threat protection status report that allows you to view overrides across your environment. By filtering the report to view data by Message Override, you can view overrides over time by type of override, like Exchange transport rule or user safe sender, and you can dig deeper in the details table to identify the causes of these overrides.

Figure 4: The Threat protection status report shows overrides by type and date

What comes next?

We’ve shared in this blog the steps we’ve taken to shed light on configuration gaps, and to help customers understand the impact configurations have on their environment. In the next blog, we will share details about the capabilities we are building to eliminate the legacy override problem, and what you can do to minimize the impact these overrides have on security posture.

Do you have questions or feedback about Microsoft Defender for Office 365? Engage with the community and Microsoft experts in the Defender for Office 365 forum.

How to apply a hierarchy of edge devices at scale to ISA-95 IIoT environments

by Contributed | Apr 28, 2021 | Technology

This article is contributed. See the original author and article here.

Using a hierarchy of IoT Edge devices in manufacturing scenarios lets customers keep their existing infrastructure but at the same time customers can benefit from being able to fully address, manage and control the IoT edge devices on the factory floor, even if they are completely disconnected from the Internet.

This helps to address critical challenges in the implementation of Industrial IoT such as the balance between security and business value, as well as how to effectively implement new technology within the context of a legacy infrastructure.

One of the established concepts in operational technology is the ISA-95 standard and the Purdue model of control. These define the models, interactions and relationships between systems in well understood layers. These layers are in turn used to define the network infrastructure, with strict interfaces between layers controlled by firewalls.

Azure IoT Edge in a nested hierarchy is ideally placed to address many of these requirements and help you to build secure and manageable edge solutions within established structures.

Let us move beyond the Tutorials to create a hierarchy of IoT Edge devices and answer the following questions:

• Which steps and configurations should you carry out to make your IIoT edge solution production ready?


• How to secure communications between edge devices and the IoT Hub


• What are best practices to scale your deployments across a fleet of devices and keep your configurations clear and manageable.


• What are the extra steps needed to deploy a hierarchy of  IoT Edge devices with 3 or more layers?


• How would you manage and control lower-level disconnected devices?


• How can I control what happens when connectivity between layers or the cloud is lost?

To make this real we have set up a sample configuration of 3 physical IoT Edge devices in a nested configuration defined as:

“Level 3 – Lower Layer Device”, “Level 4 – Lower Layer Gateway” and “Level 5 – Top Layer Gateway”.

To give you a glimpse of the configuration we have built and how it maps to the ISA-95 levels, check out this diagram:

You can find a step-by-step tutorial how to enable and use all of these best practices on this GitHub. Make sure to check it out.

Getting your IIoT edge solution production ready

Most of the best practice for implementing edge solutions for industrial IoT applies, whether you are using a nested configuration or not. Introductory reading here is the excellent checklist for getting your IoT edge instance ready for production.

On top of that, we would advise you to use a set of best practices that the Azure IoT Services have to offer, namely:

• Securing communications
  
  
  
  • Use Root CA X.509 certificates to authenticate and secure communications between parent and child devices, as well as to the IoT Hub.
  
  
  • Use existing company public key infrastructure if possible, for managing your certificates with dedicated intermediate certificates for signing the edge certificates.
  
  
  • Consider a separate intermediate certificate outside the enterprise chain of trust to sign the edge CA certificates.
  
  
  
  


• Deployment management
  
  
  
  • Tag your devices in the Device Twins to identify a target group of devices via queries
  
  
  • Use automatic deployments and jobs to apply the same configuration to a set of devices
  
  
  • Use layered deployments to extend the capabilities of some devices without having to change configurations for all devices in the fleet
  
  
  
  


• Connectivity resilience
  
  
  
  • Store and forward messages if the devices are separated from the network on the device using the Time To Live Setting in Device Twin
    
    
  
  
  
  

Certificates for a hierarchy of edge devices

To establish communication and authentication between the devices and the cloud, Industrial IoT companies often prefer using certificate-based authentication instead of using symmetric keys. We also recommend using X.509 based authentication with a chain extending to a trusted root CA for added security in productive environments.

On top of using a certificate for the authentication to the cloud, you would be required to use certificates to establish trust between the parent and child devices for transparent gateway operations.

Even in a nested scenario, every IoT Edge device needs to have its own unique device identity in the IoT Hub. Crucially, this means that you would usually generate two sets of certificates for each device:

• An optional device client certificate and key for each device to identify and authenticate the device to the IoT Hub. Alternatively, this could be a symmetric key and the user is free to mix authentication modes within the nested hierarchy.


• A required signing CA certificate, so that the edge can sign its own workload client certificates. These are used to secure connections between the different components on an IoT Edge device or between an IoT Edge device and any leaf devices in a nested hierarchy. This concept is further explained in our documentation.

If you are unsure about the terminology, check out this blogpost: The layman’s guide to X.509 certificate jargon

On the layer 4 device, the respective part of the files would look like the following. Look closely at the naming differences to understand the different certificate sets:

If you need more details how to apply the certificates to the devices and to IoT Hub, check out the GitHub.

Using layered deployments to roll out configurations at scale

Instead of setting the modules and deployment manifests manually for each IoT Edge device, it is recommended to use automatic deployments for a fleet of devices and especially in production scenarios. In a nested edge scenario, we have seen particular value in the use of layered deployments to fine tune workloads.

Deployments are rolled out to device populations by matching user defined target conditions, for example device twin properties or tags, in a strict priority order.

Layered Deployments can be used to expand the deployment to include additional modules or override configurations on top of a base deployment manifest.

Tags are very useful to group a set of devices by using the Device Twin or Module Twin of IoT Edge Devices.

Our devices have the following tags set to true in their device twins.

In detail we have the following configuration:

• One base deployment manifest for the top layer Gateway (layer 5) for devices where topnestededge=true. This is a basic deployment manifest with the system modules and the API proxy module needed for nested operations. Since this has an internet connection we get obtain images directly from the Microsoft Container Registry (MCR).


• One layered deployment manifest for the layer 5 edge devices to deploy the Simulated Temperature also from the MCR. This targets device with both tags.tempsensor=true AND tags.topnestededge=true


• One base deployment manifest for the lower layer devices (layer 3 + 4) with tag nestededge. This only contains the system modules with an image that points to $upstream. This tells the edge runtime to pull images from the parent edge device.


• One layered deployment manifest for the lower layer 4 Edge Device to deploy the IoTEdgeAPIProxy from $upstream. This applies to all devices with tag nestedgateway.


• One layered deployment manifest for the lower layer 3 + 4 Edge Devices to deploy the Simulated Temperature Sensor from $upstream.

In IoT Hub, all deployments together look like the following: 

Some other examples of layered deployments are also available in the GitHub.

Use the Module Twin or Direct Methods to send commands and desired configurations to lower-layer devices

As we have created IoT Hub Device Identities for all devices, we are able to control these devices from the cloud directly.

If we would like to update the Module Twin of our Level 3 device, for example, we would change the desired property for the respective module to enable it.

On top of that, we can also send direct methods directly from the cloud to a lower layer device. The message will be received by the top layer device and be routed through the edgeHubs of each device in the hierarchy until it reaches its designated device.

While sending direct methods to a lower layer device is a good way to directly control even disconnected devices, it requires the lower layer device to be connected to and discoverable by a parent device.

But what happens if a lower-layer device would go offline due to a networking failure of its parent?

What happens to the up – and downstream data flow on the lower levels?

How do the devices recover if the connection between the layers is separated?

In a nested hierarchy, each device has a dependency on its parent devices for connectivity. If the network connection is severed on a device in a given layer, then all devices that are residing in lower layers will also be unable to continue sending messages.

In our example, if the network connection of the layer 4 device is disconnected, both the layer 4 device and layer 3 device recognize this, and the normal store and forward functionality of the Edge Hub starts buffering. Layer 5 remains unaware and continues sending its messages to the IoT Hub. Once the network connections are re-established, each device automatically sends their queued messages in the original order. The time of how long a message should be stored is defined as the TTL (“Time To Live”) in the Module Identity twin for edgeHub of each device. The standard TTL setting is 7200 seconds or two hours.

"$edgeHub":{
	"properties.desired": {
		"schemaVersion": "1.0",
		"routes": {},
		"storeAndForwardConfiguration": {
			"timeToLiveSecs": 7200
		}
	}
}

If fine tuning is required the both the TTL and the priority of messages can also be defined in the routing rules for the edgeHub. The actual number of messages able to be stored on the device of course depends on the disk size of the device.

Device twin updates, cloud-to-device messages and direct method calls that are targeting a device that is temporarily offline, will also be queued in the IoT Hub whilst the edge is offline. They will be sent on, as long as the device reappears within the defined TTL period for C2D or the timeout for a direct method. This needs to considered in the solution design to avoid potential message loss or undesired operations occurring after they are still needed. The most robust way of communicating configuration, operational parameters and status remains the device or module twins.

See Operate devices offline – Azure IoT Edge | Microsoft Docs if you would like to learn more about offline behavior of IoT Edge devices.

Conclusion

We have walked through the practical application of nesting IoT Edge devices in industrial IoT and shown some best practice in managing the complexity in managing many heterogenous workloads on the edge. For more technical detail please visit the linked GitHub repository.

Want to try out and deploy a simulated purdue network model using IoT Edge?  Look at this Github!

Stay informed. For the latest updates on new releases, tools, and resources, stay tuned to this blog and follow us @MSIoTDevs on Twitter.