Save ingestion costs by splitting logs into multiple tables and opting for the basic tier!

This article is contributed. See the original author and article here.

In this blog post I am going to talk about splitting logs to multiple tables and opting for basic tier to save cost in Microsoft Sentinel. Before we delve into the details, let’s try to understand what problem we are going to solve with this approach.

Azure Monitor offers several log plans which our customers can opt for depending on their use cases. These log plans include:

• Analytics Logs – This plan is designed for frequent, concurrent access and supports interactive usage by multiple users. This plan drives the features in Azure Monitor Insights and powers Microsoft Sentinel. It is designed to manage critical and frequently accessed logs optimized for dashboards, alerts, and business advanced queries.


• Basic Logs – Improved to support even richer troubleshooting and incident response with fast queries while saving costs. Now available with a longer retention period and the addition of KQL operators to aggregate and lookup.


• Auxiliary Logs – Our new, inexpensive log plan that enables ingestion and management of verbose logs needed for auditing and compliance scenarios. These may be queried with KQL on an infrequent basis and used to generate summaries.

Following diagram provides detailed information about the log plans and their use cases:

 

I would also recommend going through our public documentation for detailed insights about feature-wise comparison for the log plans which should help you in taking right decisions for choosing the correct log plans.

**Note** Auxiliary logs are out of scope for this blog post, I will write a separate blog on the Auxiliary logs later.

So far, we know about different log plans available and their use cases.

 The next question is which tables support Analytics and Basic log plan?

• Analytics Logs: All tables support the Analytics plan.


• Basic Logs: All DCR-based custom tables and some Azure tables support the Basic log plan.

You can switch between the Analytics and Basic plans; the change takes effect on existing data in the table immediately.

When you change a table’s plan from Analytics to Basic, Azure monitor treats any data that’s older than 30 days as long-term retention data based on the total retention period set for the table. In other words, the total retention period of the table remains unchanged, unless you explicitly modify the long-term retention period.

Check our public documentation for more information on setting the table plan.

I will focus on splitting Syslog table and setting up the DCR-based table to Basic tier in this blog.

Typically Firewall logs contribute to high volume of log ingestion to a SIEM solution.

In order to manage cost in Microsoft Sentinel its highly recommended to thoroughly review the logs and identify which logs can be moved to Basic log plan.

At a high level, the following steps should be enough to achieve this task:

• Ingest Firewall logs to Microsoft Sentinel with the help of Linux Log Forwarder via Azure Monitor Agent.


• Assuming the log is getting ingested in Syslog table, create a custom table with same schema as Syslog table.


• Update the DCR template to split the logs.


• Set the table plan to Basic for the identified DCR-based custom table.


• Set the required retention period of the table.

At this point, I anticipate you already have log forwarder set up and able to ingest Firewall logs to Microsoft Sentinel’s workspace.

Let’s focus on creating a custom table now

This part used to be cumbersome but not anymore, thanks to my colleague Marko Lauren who has done a fantastic job in creating this PowerShell Script which can create a custom table easily. All you need to do is to enter the pre-existing table name and the script will create a new DCR-Based custom table with same schema.

Let’s see it in action:

• Download the script locally.


• Open the script in PowerShell ISE and update workspace ID & resource ID details as shown below.

 

• Save it locally and upload to Azure PowerShell.


• Load the file and enter the table name from which you wish to copy the schema.


• Provide the new table name as per your wish, ensure the name has suffix “_CL” as shown below:

 

This should create a new DCR-based custom table which you can check in Log Analytics Workspace > Table blade as shown below:

 

**Note** We highly recommend you should review the PowerShell script thoroughly and do proper testing before executing it in production. We don’t take any responsibility for the script.

 

The next step is to update the Data Collection Rule template to split the logs

Since we already created custom table, we should create a transformation logic to split the logs and send less relevant log to the custom table which we are going to set to Basic log tier.

For demo purposes, I’m going to split logs based on SeverityLevel. I will drop “info” logs from Syslog table and stream it to Syslog_CL table.

Let’s see how it works:

• Browse to Data Collection Rule blade.


• Open the DCR for Syslog table, click on Export template > Deploy > Edit Template as shown below:

 

 

• In the dataFlows section, I’ve created 2 streams for splitting the logs. Details about the streams as follows:
  
  
  
  • 1^st Stream: It’s going to drop the Syslog messages where SeverityLevel is “info” and send the logs to Syslog table.
  
  
  • 2^nd Stream: It’s going to capture all Syslog messages where SeverityLevel is “info” and send the logs to Syslog_CL table.
  
  
  
  

 

• Save and deploy.

 

Let’s validate if it really works

Go to the Log Analytics Workspace > Logs and check if the tables contains the data which we have defined it for.

In my case as we can see, Syslog table contains all logs except those where SeverityLevel is “info”

 

Additionally, our custom table: Syslog_CL contains those Syslog data where SeverityLevel is “info”

 

 

Now the next part is to set the Syslog_CL table to Basic log plan

Since Syslog_CL is a DCR-based custom table, we can set it to Basic log plan. Steps are straightforward:

• Go to the Log Analytics Workspace > Tables


• Search for the table: Syslog_CL


• Click on the ellipsis on the right side and click on Manage table as shown below:

 

• Select the table plan to Basic and set desired retention period

 

• Save the settings.

 

Now you can enjoy some cost benefits, hope this helps.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.