Email Stuck in Transport Queues

by | Jan 1, 2022 | Technology | 0 comments

This article is contributed. See the original author and article here.

UPDATE (1/1/22 @10:45pm PST):  We’ve created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:


Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS “Microsoft” Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can’t convert “2201010001” to long.

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

Please note the following regarding the solution:



  • Implementation of the solution requires customer actions, and it will take some time to make the necessary changes, download the updated files, and clear the transport queues.

  • Customer actions can be automated with the scan engine reset script from https://aka.ms/ResetScanEngineVersion or they can be performed manually. Every action taken by the script is listed in the manual steps below.

  • Whether you perform the steps automatically or manually, they must be performed on every Exchange 2016 and Exchange 2019 server in your organization. If you use the automated script, you can run it on multiple servers in parallel.

  • We recommend starting with the servers in your environment that have the largest queues. You can use Get-Queue to view queue size and determine which queues are the largest.

  • If you previously disabled or bypassed antimalware scanning as a mitigation for this issue, we recommend that you re-enable it using the Enable-AntimalwareScanning.ps1 script after performing the steps below.

  • The version info for the updated file represents only a sequence change. Once the solution is implemented, your server will continue to receive new AV updates as expected.


Using the Automated Solution


Run the script on each Exchange mailbox server in your organization. Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:


[PS] C:Program FilesMicrosoftExchange ServerV15Scripts>.Reset-ScanEngineVersion.ps1
EXCH1 Stopping services…
EXCH1 Removing Microsoft engine folder…
EXCH1 Emptying metadata folder…
EXCH1 Starting services…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Filtering Management Service (FMS)’ to start…
WARNING: Waiting for service ‘Microsoft Exchange Transport (MSExchangeTransport)’ to start…
EXCH1 Starting engine update…
Running as EXCH1-DOMAdministrator.
——–
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
——–
[PS] C:Program FilesMicrosoftExchange ServerV15Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked           : 01/01/2022 08:58:22 PM -08:00
LastUpdated           : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001
UpdateStatus          : UpdateAttemptSuccessful

Using the Manual Solution


In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization. Edge Transport servers are unaffected by this issue.


Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%MicrosoftExchange ServerV15FIP-FSDataEnginesamd64Microsoft.
4. Remove all files from the following folder: %ProgramFiles%MicrosoftExchange ServerV15FIP-FSDataEnginesmetadata.


Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%MicrosoftExchange ServerV15Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.


Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.


After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.


 


Original post below:


We are aware of and working on an issue causing messages to be stuck in transport queues on Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it not a failure of the AV engine itself. This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.


We are actively working on resolving this issue and expect to release details on how to resolve this issue later today. In the meantime, if your organization performs malware scanning of messages outside of your on-premises Exchange servers (for example, by routing mail through Exchange Online, or by using a third-party message hygiene solution), you can bypass or disable malware scanning on your Exchange servers and clear your transport queues. You should use one of these workarounds only if you have an existing malware scanner for email other than the engine in Exchange Server. See the following articles for details on how to disable or bypass malware scanning:



Our engineers were working around the clock on a fix that would eliminate the need for customer action, but we determined that any change that did not involve customer action would require several days to develop and deploy. We are working on another update which is in final test validation. The update requires customer action, but it will provide the quickest time to resolution.


We expect to have this update to you shortly along with the actions required by you. We are sorry for any inconvenience that this issue has caused.


— The Exchange Team

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Submit a Comment