eDiscovery launches a modern, intuitive user experience

by Contributed | Aug 12, 2024 | Technology

This article is contributed. See the original author and article here.

This month, we have launched a redesigned Microsoft Purview eDiscovery product experience in public preview. This improved user experience revolutionizes your data search, review and export tasks within eDiscovery. Our new user-friendly and feature-rich eDiscovery experience is not just about finding and preserving data, it’s about doing it with unprecedented efficiency and ease. The modern user experience of eDiscovery addresses some long-standing customer requests, such as enhanced search capabilities with MessageID,  Sensitive Information Types (SITs) and sensitivity labels. It also introduces innovative features like draft query with Copilot and search using audit log. These changes, driven by customer feedback and our commitment to innovation, offer tangible value by saving time and reducing costs in the eDiscovery process. 

The new eDiscovery experience is exclusively available in the Microsoft Purview portal. The new Microsoft Purview portal is a unified platform that streamlines data governance, data security, and data compliance across your entire data estate. It offers a more intuitive experience, allowing users to easily navigate and manage their compliance needs.  

Unified experience 

One of the benefits of the new improved eDiscovery offers a unified, consistent, and intuitive experience across different licensing tiers. Whether your license includes eDiscovery standard or premium, you can use the same workflow to create cases, conduct searches, apply holds, and export data. This simplifies the training and education process for organizations that upgrade their license and want to access premium eDiscovery features. Unlike the previous experience, where Content Search, eDiscovery (Standard), and eDiscovery (Premium) had different workflows and behaviors, the new experience lets you access eDiscovery capabilities seamlessly regardless of your license level. E5 license holders have the option to use premium features such as exporting cloud attachments and Teams conversation threading at the appropriate steps in the workflow. Moreover, users still have access to all existing Content Searches and both Standard and Premium eDiscovery cases on the unified eDiscovery case list page in the Microsoft Purview portal.  

The new experience also strengthens the security controls for Content Search by placing them in an eDiscovery case. This allows eDiscovery administrators to control who can access and use existing Content Searches and generated exports. Administrators can add or remove users from the Content Search case as needed. This way, they can prevent unauthorized access to sensitive search data and stop Content Search when it is no longer required. Moreover, this helps maintain the integrity and confidentiality of the investigation process. The new security controls ensure that only authorized personnel can access sensitive data, reducing the risk of data breaches and complying with legal and regulatory standards.

Enhanced data source management 

Efficient litigation and investigation workflows hinge on the ability to precisely select data sources and locations in the eDiscovery process. This enables legal teams to swiftly preserve relevant information and minimize the risk of missing critical evidence. The improved data source picking capability allows for a more targeted and effective search, which is essential in responding to legal matters or internal investigations. It enables users to apply holds and conduct searches with greater accuracy, ensuring that all pertinent information is captured without unnecessary data proliferation. This improvement not only enhances the quality of the review, but also reduces the overall costs associated with data storage and management. 

The new eDiscovery experience makes data source location mapping and management better as well. You can now perform a user or group search with different identifiers and see their data hierarchy tree, including their mailbox and OneDrive. For example, eDiscovery users can use any of following identifiers: Name, user principal name (UPN), SMTP address, or OneDrive URL. The data source picker streamlines the eDiscovery workflow by displaying all potential matches and their locations, along with related sources such as frequent collaborators, group memberships, and direct reports. This allows for the addition of these sources to search or hold scope without relying on external teams for information on collaboration patterns, Teams/Group memberships, or organizational hierarchies. 

Figure 1: New data source view with ability to associate person’s mailbox and OneDrive, exploring to a person’s frequent collaborator and ability to query data source updates.

The “sync” capability in the new data source management flow is a significant addition that ensures eDiscovery users are always informed about the latest changes in data locations. With this feature, users can now query whether a specific data source has newly provisioned data locations or if any have been removed. For example, if a private channel is created for a Teams group, this feature alerts eDiscovery users to the new site’s existence, allowing them to quickly and easily include it in their search scope, ensuring no new data slips through the cracks. This real-time update capability empowers users to make informed decisions about including or excluding additional data locations in their investigations. This capability ensures that their eDiscovery process remains accurate and up-to-date with the latest data landscape changes. It is a proactive approach to data management that enhances the efficiency and effectiveness of eDiscovery operations, providing users with the agility to adapt to changes swiftly. 

Improved integration with Microsoft Information Protection 

The new eDiscovery experience now supports querying by Sensitive Information Types (SITs) and sensitivity labels. Labeling, classifying, and encrypting your organization’s data is a best practice that serves multiple essential purposes. It helps to ensure that sensitive information is handled appropriately, reducing the risk of unauthorized access and data breaches. By classifying data, organizations can apply the right level of protection to different types of information, which is crucial for compliance with various regulations and standards. Moreover, encryption adds a layer of security that keeps data safe even if it falls into the wrong hands. It ensures that only authorized users can access and read the information, protecting it from external threats and internal leaks.  

The new eDiscovery search functionality supports searches for emails and documents classified by SITs or specific sensitivity labels, facilitating the collection and review of data aligned with its classification for thorough investigations. This capability compresses the volume of evidence required for review, significantly reducing both the time and cost of the process. The support of efficient document location and management by targeting specific sensitivity labels unlocks the ability for organizations to validate and understand how sensitivity labels are utilized. This is exemplified by the ability to conduct collections across locations or the entire tenant for a particular label, using the review set to assess label application. Additionally, combining this with SIT searches helps verify correct data classification. For example, it ensures that all credit card data is appropriately labeled as highly confidential by reviewing items containing credit card data that are not marked as such, thereby streamlining compliance and adherence to security policies. 

Figure 2: Better integration with Microsoft Information Protection means the ability to search labeled and protected data by SIT and sensitivity label.Figure 3: Better integration with Microsoft Information Protection means the ability to search labeled and protected data by SIT and sensitivity label.

Enhanced investigation capabilities 

The new eDiscovery experience introduces a powerful capability to expedite security investigations, particularly in scenarios involving a potentially compromised account. By leveraging the ability to search by audit log, investigators can swiftly assess the account’s activities, pinpointing impacted files. As part of the investigative feature, eDiscovery search can also make use of evidence file as search input. It enables a rapid analysis of file content patterns or signatures. This feature is crucial for identifying similar or related content, providing a streamlined approach to discover if sensitive files have been copied or moved, thereby enhancing the efficiency and effectiveness of the security response. 

The enhanced search capability by identifier in the new eDiscovery UX is a game-changer for customers, offering a direct route to the exact message or file needed. With the ability to search using a messageID for mailbox items or a path for SharePoint items, users can quickly locate and retrieve the specific item they require. This precision not only streamlines evidence collection but also accelerates the process of purging leaked data for spillage cleanup. It’s a significant time-saver that simplifies the workflow, allowing customers to focus on what matters most – securing and managing their digital environment efficiently, while targeting relevant data. 

Building on the data spillage scenario, our search and purge tool for mailbox items, including Teams messages, also received a significant 10x enhancement. Where previously administrators could only purge 10 items per mailbox location, they can now purge up to 100 items per mailbox location. This enhancement is a benefit for administrators tasked with responding to data spills or needing to remediate data within Teams or Exchange, allowing for a more comprehensive and efficient purge process. With all these investigative capability updates, now the security operations team is ready to embrace the expanded functionality and take their eDiscovery operations to the next level. 

Microsoft Security Copilot capabilities 

The recently released Microsoft Security Copilot’s capabilities in eDiscovery are transformative, particularly in generating KeyQL from natural language and providing contextual summarization and answering abilities in review sets. These features significantly lower the learning curve for KeyQL, enabling users to construct complex queries with ease. Instead of mastering the intricacies of KeyQL, users can simply describe what they are looking for using natural language, and Copilot translates that into a precise KeyQL statement. This not only saves time but also makes the power of eDiscovery accessible to a broader range of users, regardless of their technical expertise. 

Figure 4: Draft query faster with Copilot’s N2KeyQL capability.

Moreover, Copilot’s summarization skills streamline the review process by distilling key insights from extensive datasets. Users can quickly grasp the essence of large volumes of data, which accelerates the review process and aids in identifying the most pertinent information. This is particularly beneficial in legal and compliance contexts, where time is often of the essence, and the ability to rapidly process and understand information can have significant implications. 

Figure 5: Copilot summarization skill in Review Set helps reviewer review content by assessing summary of the item – even when the conversation is in not in English.

Additional export options 

The new eDiscovery experience introduces a highly anticipated suite of export setting enhancements. The contextual conversation setting is now distinct from the conversation transcript setting, offering greater flexibility in how Teams conversations are exported. The ability to export into a single PST allows for the consolidation of files/items from multiple locations, simplifying the post-export workflow. Export can now give friendly names to each item, eliminating the need for users to decipher item GUIDs, and making identification straightforward. Truncation in export addresses the challenges of zip file path character limits. Additionally, the expanded versioning options empower users to include all versions or select the latest 10 or 100, providing tailored control over the data. These improvements not only meet user expectations but also significantly benefit customers by streamlining the eDiscovery process and enhancing overall efficiency. 

Additional enhancements 

As part of the new experience, we are introducing the review set query report, which generates a hit-by-term report based on a KQL query. This query report allows users to quickly see the count and volume of items hit on a particular keyword or a list of compound queries, and can be optionally downloaded.  By providing a detailed breakdown of where and how often each term appears, it streamlines the review by focusing on the most relevant documents, reducing the volume of data that needs to be manually reviewed, and offers a better understanding of which terms may be too broad or too narrow. 

As part of the improved user experience, all long-running processes now show a transparent and informative progress bar. This progress bar provides users with real-time visibility into the status of their searches and exports, allowing eDiscovery practitioners to better plan their workflow and manage their time effectively. This feature is particularly beneficial in the context of legal investigations, where timing is often critical, and users need to anticipate when they can proceed to the next steps. This level of process transparency allows users to stay informed and make decisions accordingly. 

Figure 6: Transparent progress bar for all long-running processes detailing scope of the process and estimated time to complete.

In addition to progress transparency, all processes in the new eDiscovery experience will include a full report detailing the information related to completed processes. The defensibility of eDiscovery cases and investigations is paramount. The full reporting capabilities for processes such as exports, searches, and holds provide critical transparency. For example, it allows for a comprehensive audit of what was searched or exported, the specific timing, and the settings used. For customers, this means a significant increase in trust and defensibility of the eDiscovery process. This enhancement not only bolsters the integrity of the eDiscovery process but also reinforces the commitment to delivering customer-centric solutions that meet the rigorous demands of legal compliance and data management. 

Hold policy detail view also received an upgrade as part of this new eDiscovery release. Customers now can access the hold policy view with detailed information on all locations and their respective hold status. This detailed view is instrumental in providing a transparent audit of what location is on hold, ensuring that all relevant data is preserved, and that no inadvertent destruction of evidence occurs during the process. Customers can download and analyze the full detailed hold location report, ensuring that all necessary content is accounted for and that legal obligations are met.  

As we conclude this exploration of the modernized Microsoft Purview eDiscovery (preview) experience, it’s clear that the transformative enhancements are set to redefine the landscape of legal compliance and security investigations. The new experience, with its intuitive design and comprehensive set of new capabilities, streamlines the eDiscovery process, making it more efficient and accessible than ever before. The new eDiscovery experience is currently in public preview and is expected to be Generally Available by the end of 2024.  

Thank you for joining us on this journey through the latest advancements in eDiscovery. We are excited to see how these changes will empower legal and compliance teams to achieve new levels of efficiency and effectiveness in their important work. To learn more about the changes in eDiscovery, visit our product documentation. As always, we are eager to hear your feedback and continue innovating to improve your experience. We welcome your thoughts via the Microsoft Purview portal’s feedback button.  

We hope these enhancements improve your day-to-day experience and ultimately streamline the eDiscovery process, making it more efficient and accessible than ever before. The new eDiscovery experience is currently in public preview and is expected to be Generally Available by the end of 2024.  

Learn more

We are excited to see how these changes will empower legal and compliance teams to achieve new levels of efficiency and effectiveness in their important work. Check out our interactive guide at https://aka.ms/eDiscoverynewUX to better understand the changes in eDiscovery. As always, we are eager to hear your feedback and continue innovating to improve your experience. We welcome your thoughts via the Microsoft Purview portal’s feedback button.  

To learn more about eDiscovery, visit our Microsoft documentation at http://aka.ms/eDiscoveryPremium, or our “Become an eDiscovery Ninja” page at https://aka.ms/ediscoveryninja. If you have yet to try Microsoft Purview solutions, we are happy to share that there is an easy way for eligible customers to begin a free trial within the Microsoft Purview compliance portal. By enabling the trial in the compliance portal, you can quickly start using all capabilities of Microsoft Purview, including Insider Risk Management, Records Management, Audit, eDiscovery, Communication Compliance, Information Protection, Data Lifecycle Management, Data Loss Prevention, and Compliance Manager.

Utilizing Azure Storage and Runbooks for scheduled automated backups of Azure SQL Databases

by Contributed | Aug 11, 2024 | Technology

This article is contributed. See the original author and article here.

In this article, we are going to provide detailed steps to create a scheduled Azure SQL Database backup to storage account using automation. This is a useful technique for maintaining regular backups of your database and storing them in a secure and accessible location. You will get an actual backup of Azure SQL Database stored in a storage account in .bacpac format, which you can restore or migrate as needed. The automation process involves creating an automation account that triggers a PowerShell script through a runbook to run the backup command and save the output to a blob container.

Prerequisites

1. Azure Storage account: The storage account is needed to host the database backups. You are required to set up a container within this account to store the backups.

2. Azure SQL Database and Server: This is the database you will back up, along with its hosting server. Ensure that you grant Azure services permission to access the server where this database is hosted by selecting “Allow Azure services and resources to access this server” within the networking section, as illustrated in the following documentation: Network Access Controls – Azure SQL Database & Azure Synapse Analytics | Microsoft Learn

3. Azure Automation account and PowerShell Workflow runbook: These components are utilized to set up automatic backups and their scheduling. make sure to using a PowerShell version 7.2 or above when creating the runbook. To learn more about setting up a PowerShell Workflow runbook in Azure Automation, please check the following documentation: Tutorial – Create a PowerShell Workflow runbook in Azure Automation | Microsoft Learn

Setup

Once all the necessary prerequisites are in place, you should navigate to the runbook and click on “Edit.” Then choose the “Edit in portal” option as illustrated below:

 The editor interface for the runbook will launch, displaying the runbook you created earlier. Enter the following code into the editor area:

# Connect to Azure with system-assigned managed identity
Connect-AzAccount -Identity
# set and store context
$AzureContext = Set-AzContext –SubscriptionId "*****"
# Resource group name
$resourceGroup = "*****"
# Storage account name that will have the backups
$storageAccountName = "*****"
# Storage account access key that will have the backups
$storageKey = "*****”
# Container name that will have the backups
$containerName = "*****"
# storage blob uri with the datetime
$storageUri = "https:// *****.blob.core.windows.net/*****/db-$(Get-Date -UFormat "%Y-%m-%d_%H-%m-%S").bacpac"
#Storage access key type
$storageKeyType = "StorageAccessKey"
#SQL server name
$server_name ="*****"
#Database name to be exported
$SQL_db = "*****"
#SQL Auth Username
$SQL_username = "*****"
#SQL Auth Password
$SQL_secure_secret = ConvertTo-SecureString -String "*****" -AsPlainText -Force
# Run the Export job with the required parameters
New-AzSqlDatabaseExport -ResourceGroupName $resourceGroup -ServerName $server_name -DatabaseName $SQL_db -StorageKeyType $storageKeyType -StorageKey $storageKey -StorageUri $storageUri -AdministratorLogin $SQL_username -AdministratorLoginPassword $SQL_secure_secret

Ensure you fill in all the ***** with the appropriate values for the step you’ve designed, then press the “Save” button located at the top left corner. The parameters given relate to details on subscription and resource group, as well as information on the SQL database, server, and storage account. You’ll find explanations for each parameter in the code snippet provided above. 

Select the test button on the toolbar, which will display the test pane allowing you to test the script prior to scheduling. Hit the start button and monitor the requests; if the run is successful, you should observe an outcome like this:

You can monitor the progress of backups and access the export history through the SQL server in the “Import/Export history” section, as illustrated below:

After the backup finishes, it will appear in the storage account with the datetime suffix provided by the script:

After verifying the script operates correctly, you may go ahead with publishing the runbook and link it to a schedule based on your requirements according to the following documentation:

Manage schedules in Azure Automation | Microsoft Learn

You can initiate the automation by pressing the start button, schedule it to run automatically, or set a webhook to trigger the process. Remember that you can only run one export job at a time. If you attempt to run two jobs simultaneously, one will not succeed and you’ll receive an error message stating that another export job is already in progress.

Recommendations

Establishing a routine backup schedule for a large database over an undefined timeframe can lead to substantial storage consumption and potentially significant costs. It’s important to regularly monitor your storage account and remove unnecessary backups, or consider relocating them to more cost-effective storage tiers, such as cold or archive.

It might be beneficial to consider implementing a storage lifecycle management policy to manage data within the storage account and decrease the costs associated with storing database backups. Lifecycle management can help you in creating an automated schedule to delete blobs or transition blobs to a different tier that is less expensive like cold or archive tiers based on creation date. For additional details on storage lifecycle management and instructions for configuration, please consult the provided documentation:

Configure a lifecycle management policy – Azure Blob Storage | Microsoft Learn

If you have soft delete enabled on the storage account, make sure that the retention period is set appropriately to avoid incurring additional charges for retaining soft-deleted data over an extended period.