This article is contributed. See the original author and article here.
By now you are hopefully aware of the TLS 1.0/1.1 deprecation efforts that are underway across the industry and Microsoft 365 in particular. Head out to our documentation for more details and references if you need a refresher! Also check out this blog entry to see how you can use reporting in Exchange Online to get an overview about the TLS versions used by mails submitted to your tenant. This topic may be super-relevant to you, because as confirmed by the message center post MC229914, TLS 1.0 and TLS 1.1 deprecation started enforcing for Exchange Online mail flow endpoints beginning January 11th, 2021. The rollout will continue over the following weeks and months. This essentially means, soon this deprecation process will be over, and we will no longer accept TLS 1.0 and TLS 1.1 email connections from external sources. Also note that Exchange Online will never use TLS 1.0 or 1.1 to send outbound email.
We wanted to talk about what this means for SMTP traffic destined to Exchange Online in particular. What happens if a server on your side can only use TLS 1.0 with SMTP? Will sending fail, and if yes, how do you notice TLS 1.0 or TLS 1.1 is the root cause of your email problems? There are different variables that impact this and we will try to mention most frequent scenarios.
Before diving into further details, keep in mind that generally speaking, the TLS implementation in Exchange on-premises or Exchange Online is done opportunistically. This means:
- For receiving mail into Exchange: If the sending server does not support TLS, or if the TLS negotiation fails, Exchange Online will still accept messages unencrypted and without TLS (provided the sending server’s configuration allows that).
- For sending mail from Exchange: For outbound email, if the receiving server does not support TLS (does not advertise the STARTTLS Verb), Exchange on-premises and Exchange Online will send email without TLS (provided TLS is not forced on the send connector or outbound connector).
Another point to keep in mind is that Exchange will always attempt to initially negotiate the highest possible version of TLS which is enabled on the other server. Once this version is selected during the TLS handshake – Exchange does not attempt a lower version of TLS/SSL that might also be enabled on the server. In case there is a failure during communication, Exchange will instead re-attempt the delivery without TLS. Our previously published 3 part blog posts (Exchange Server TLS guidance part 1, Part 2 and Part 3) extensively covered how various components like Schannel, WinHTTP, .Net, etc. work together to decide the version of TLS Exchange server should use during TLS handshakes.
Other than TLS versions, another factor that we tend to overlook are the Cipher Suites supported by Office 365. While the servers or devices may use TLS 1.2, not supporting one of the ciphers suites adopted by Office 365 from the published list could also cause mail flow issues.
Let us look at the details of each scenario!
3rd party SMTP server sending to Exchange Online
The experience here will mostly depend on the sending server’s implementation. In most cases, there should be no impact. Once the TLS 1.0 attempt fails, the sender should fall back to not using TLS at all and send in an unencrypted manner. If the sender is relying solely on TLS 1.0 or TLS 1.1 and cannot send unencrypted, it is again up to the sending server’s implementation on what happens – the mail might remain queued while the sender keeps retrying. Ultimately the sending server should generate an error or an NDR after the message expiration timeout.
Exchange server (external to the organization) sending to Exchange Online
This applies to the case where your Exchange servers in contoso.com would be sending to a different organization, let’s say fabrikam.com, which is hosted in Exchange Online. For most organizations, mail flow will not break. This is because send connectors in Exchange are by default created with the setting “RequireTLS: false”, meaning they will attempt a TLS connection if the remote party supports it, but if TLS negotiation fails, they will simply fall back to not using TLS and will send anyway. The SMTP Send protocol logs will contain entries that resemble the following:
You will see that initially the mail could not be sent to Office 365 and it failed with error: TLS negotiation failed with error SocketError
#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2021-01-11T16:43:14.811Z,Connector2Fabrikam,08D8B64FC6449F2A,0,,10.1.0.16:25,*,SendRoutingHeaders,Set Session Permissions
2021-01-11T16:43:14.811Z,Connector2Fabrikam,08D8B64FC6449F2A,1,,10.1.0.16:25,*,,attempting to connect
2021-01-11T16:43:14.817Z,Connector2Fabrikam,08D8B64FC6449F2A,2,10.0.0.16:6933,10.1.0.16:25,+,,
2021-01-11T16:43:14.969Z,Connector2Fabrikam,08D8B64FC6449F2A,3,10.0.0.16:6933,10.1.0.16:25,<,”220 BN3USG02FT012.mail.protection.office365.us Microsoft ESMTP MAIL Service ready at Mon, 11 Jan 2021 17:43:14 +0100″,
2021-01-11T16:43:14.969Z,Connector2Fabrikam,08D8B64FC6449F2A,4,10.0.0.16:6933,10.1.0.16:25,>,EHLO exc16.contoso.com,
2021-01-11T16:43:15.012Z,Connector2Fabrikam,08D8B64FC6449F2A,5,10.0.0.16:6933,10.1.0.16:25,<,250 BN3USG02FT012.mail.protection.office365.us Hello [10.0.0.16] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS X-ANONYMOUSTLS AUTH NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
2021-01-11T16:43:15.013Z,Connector2Fabrikam,08D8B64FC6449F2A,6,10.0.0.16:6933,10.1.0.16:25,>,STARTTLS,
2021-01-11T16:43:15.016Z,Connector2Fabrikam,08D8B64FC6449F2A,7,10.0.0.16:6933,10.1.0.16:25,<,220 2.0.0 SMTP server ready,
2021-01-11T16:43:15.016Z,Connector2Fabrikam,08D8B64FC6449F2A,8,10.0.0.16:6933,10.1.0.16:25,*,” CN=mail.contoso.com CN=R3, O=Let’s Encrypt, C=US 03C6CCE6D57C1D2DA908BF69EBD10963AE74 AF15A9798388DD9C0C03FEBC897025CD76963178 2020-12-05T09:46:36.000Z 2021-03-05T09:46:36.000Z mail.contoso.com;autodiscover.contoso.com;”,Sending certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
2021-01-11T16:43:15.043Z,Connector2Fabrikam,08D8B64FC6449F2A,9,10.0.0.16:6933,10.1.0.16:25,*,,TLS negotiation failed with error SocketError
2021-01-11T16:43:15.043Z,Connector2Fabrikam,08D8B64FC6449F2A,10,10.0.0.16:6933,10.1.0.16:25,-,,Remote
A network capture will resemble the following, which clearly explains the reason behind the failure. As you see in the following screenshot, the sending server, after the exchange of STARTTLS verb, tried to negotiate transport layer security using TLS version 1.1. The Exchange Online server instantly disconnected the session with a “FINISH” flag (FIN):
However, immediately after that, the sending server should fall back to not using TLS and will send the email anyway and it will be accepted by Exchange Online:
2021-01-11T16:43:15.047Z,Connector2Fabrikam,08D8B64FC6449F2B,0,,10.1.0.16:25,*,SendRoutingHeaders,Set Session Permissions
2021-01-11T16:43:15.047Z,Connector2Fabrikam,08D8B64FC6449F2B,1,,10.1.0.16:25,*,,attempting to connect
2021-01-11T16:43:15.050Z,Connector2Fabrikam,08D8B64FC6449F2B,2,10.0.0.16:6934,10.1.0.16:25,+,,
2021-01-11T16:43:15.053Z,Connector2Fabrikam,08D8B64FC6449F2B,3,10.0.0.16:6934,10.1.0.16:25,<,”220 BN3USG02FT012.mail.protection.office365.us Microsoft ESMTP MAIL Service ready at Mon, 11 Jan 2021 17:43:14 +0100″,
2021-01-11T16:43:15.053Z,Connector2Fabrikam,08D8B64FC6449F2B,4,10.0.0.16:6934,10.1.0.16:25,>,EHLO exc16.contoso.com,
2021-01-11T16:43:15.055Z,Connector2Fabrikam,08D8B64FC6449F2B,5,10.0.0.16:6934,10.1.0.16:25,<,250 BN3USG02FT012.mail.protection.office365.us Hello [10.0.0.16] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS X-ANONYMOUSTLS AUTH NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
2021-01-11T16:43:15.058Z,Connector2Fabrikam,08D8B64FC6449F2B,6,10.0.0.16:6934,10.1.0.16:25,*,,sending message with RecordId 40900973559810 and InternetMessageId <5149fa60b89741cfaf6e05d5767776a9@contoso.com>
2021-01-11T16:43:15.059Z,Connector2Fabrikam,08D8B64FC6449F2B,7,10.0.0.16:6934,10.1.0.16:25,>,MAIL FROM:<user@contoso.com> SIZE=9031,
2021-01-11T16:43:15.059Z,Connector2Fabrikam,08D8B64FC6449F2B,8,10.0.0.16:6934,10.1.0.16:25,>,RCPT TO:<user@fabrikam.com>,
2021-01-11T16:43:15.118Z,Connector2Fabrikam,08D8B64FC6449F2B,9,10.0.0.16:6934,10.1.0.16:25,<,250 2.1.0 Sender OK,
2021-01-11T16:43:15.120Z,Connector2Fabrikam,08D8B64FC6449F2B,10,10.0.0.16:6934,10.1.0.16:25,<,250 2.1.5 Recipient OK,
2021-01-11T16:43:15.121Z,Connector2Fabrikam,08D8B64FC6449F2B,11,10.0.0.16:6934,10.1.0.16:25,>,BDAT 2932 LAST,
2021-01-11T16:43:18.300Z,Connector2Fabrikam,08D8B64FC6449F2B,12,10.0.0.16:6934,10.1.0.16:25,<,”250 2.6.0 <5149fa60b89741cfaf6e05d5767776a9@contoso.com> [InternalId=171798691842, Hostname=BN3USG02FT012.mail.protection.office365.us] 4228 bytes in 2.816, 1.466 KB/sec Queued mail for delivery”,
2021-01-11T16:43:18.314Z,Connector2Fabrikam,08D8B64FC6449F2B,13,10.0.0.16:6934,10.1.0.16:25,>,QUIT,
2021-01-11T16:43:18.316Z,Connector2Fabrikam,08D8B64FC6449F2B,14,10.0.0.16:6934,10.1.0.16:25,<,221 2.0.0 Service closing transmission channel,
2021-01-11T16:43:18.316Z,Connector2Fabrikam,08D8B64FC6449F2B,15,10.0.0.16:6934,10.1.0.16:25,-,,Local
Note: to see where the SMTP Send protocol logs are stored on your on-premises server, run “Get-TransportServer <servername> | fl SendProtocolLog*”. Logs will be generated once you enable logging with a cmdlet like “Set-SendConnector <connectorname> -ProtocolLoggingLevel Verbose”.
If you explicitly configured your send connector with the setting “RequireTLS: True”, the fallback to non-TLS will not happen. In this case, the behavior will be similar to what is described in the next section.
On-premises Exchange server in a hybrid configuration sending to Exchange Online (internal to the organization)
In this scenario, mails are sent from your on-premises recipients to your Exchange Online recipients. When your Exchange servers are configured for hybrid, by default, the “Outbound to Office 365…” connector has “RequireTLS: True”. This means that on-premises servers won’t fall back to sending unencrypted. If the TLS 1.0/1.1 attempt fails, Exchange will keep retrying the connection using TLS several times at various intervals (the exact retry intervals and counts are described here.) The send protocol log entries will be similar to those shown above, with the difference that the “TLS negotiation failed with error SocketError” entries will just keep repeating, since there is no fallback. Unless you modified the default retry configuration, the on-premises Exchange server will keep retrying for 2 days. Throughout this time, the affected mails will stay in the queue. The queue details will look similar to this:
[PS] C:>Get-Queue <queue ID> | fl
(…)
Status : Retry
LastError : [{LED=451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to SocketError};{MSG=};{FQDN=<servername>};{IP=<serverIP>};{LRT=1/11/2021 6:02:39 PM}]
(…)
By default, the sender will receive a delay DSN (the subject starts with “Delivery delayed”, localized) after 4 hours. Unless you do some manual intervention sooner, the sending Exchange server will normally give up after 2 days and generate an NDR. The NDR message would look like this:
Delivery has failed to these recipients or groups:
user@contoso.com
Several attempts to deliver your message were unsuccessful and we stopped trying. It could be a temporary situation. Try to send your message again later.
Diagnostic information for administrators:
Generating server: <servername>
Receiving server: <servername>
user@contoso.com
1/7/2021 7:24:14 PM – Server at <servername> returned ‘550 5.4.300 Message expired -> 451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to SocketError’
1/7/2021 7:23:14 PM – Server at mail.contoso.com (10.0.0.16) returned ‘451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to SocketError’
To avoid such problems, be sure to configure your on-premises Exchange servers to support TLS 1.2, as described in our three-part blog series starting here.
Exchange Online sending to Exchange server (external to the organization)
This experience will depend on how the receiving server has implemented inbound mail flow. Assuming the receiving server supports TLS (advertises STARTTLS Verb), Exchange Online will only use TLS 1.2 to send outbound email. If the receiving server does not support TLS 1.2, Exchange Online being opportunistic will try to send email without TLS. If the receiving mail server does not have TLS enforced for inbound email flow, the email will be sent without TLS. You will know if your server is enforcing TLS by querying for the RequireTLS property of the Receive Connector, e.g. ‘Get-ReceiveConnector “Default Frontend <ServerName>” | fl RequireTLS’. If TLS is enforced at the receiving end, Exchange Online will continue retrying and the email will remain queued, and eventually we will generate NDR message after 24 hours (which is default message expiration timeout for Exchange Online).
On-premises non-Exchange server, application or device relaying external emails through your Exchange Online tenant following this article
If you have an on-premises non-Exchange server, application or device that relays email through your Office 365 tenant either by SMTP AUTH client submission or by using a certificate based inbound connector, make sure these servers or devices or applications support TLS 1.2. If they do not support TLS 1.2, the TLS negotiation will fail, and a subsequent non-TLS retry might be attempted. SMTP AUTH client submission does not work without TLS. And in case relay is configured through a certificate based inbound connector, the common name (CN) or subject alternative name (SAN) verification will fail during non-TLS communication. This will cause an “550 5.7.0. Relay Access Denied” error in both scenarios. Email delivery to mailboxes hosted in your Office 365 tenant will continue to work albeit it will be treated as “anonymous” submission.
Hopefully, this clarifies what you need to look for in case mail flow starts to break with the disablement of TLS 1.0/1.1! We also want to take a moment to thank Mike Brown, Nino Bilic and Sean Stevenson for their contributions and review.
and
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.
Recent Comments