Lesson Learned #269: Unable to connect – Is unavailable or does not exist – Connection Time out

Lesson Learned #269: Unable to connect – Is unavailable or does not exist – Connection Time out

This article is contributed. See the original author and article here.

Today, I worked on a service request that your customer is facing the following error message: During handling of the above exception, another exception occurred: Traceback (most recent call last): File “src/pymssql/_pymssql.pyx”, line 653, in pymssql._pymssql.connect pymssql._pymssql.OperationalError: (20009, b’DB-Lib error message 20009, severity 9:nUnable to connect: Adaptive Server is unavailable or does not exist (servername.database.windows.net)nNet-Lib error during Connection timed out (110)nDB-Lib error message 20009, severity 9:nUnable to connect: Adaptive Server is unavailable or does not exist (servername.database.windows.net)nNet-Lib error during Connection timed out (110)n’)


 


It is a python application using pymssql library running in Ubuntu 18.04. Our customer reported that previous connections were fine  and this issue suddenly happened. 


 


After checking the port 1433 and redirection ports in Network Security Groups we didn’t see any issue.


 


To check if the ports are available from this machine we ran the command telnet servername.database.windows.net 1433 and we saw  that is not possible to connect.


 


The IP reported is 10.10.1.25. This IP looks like a private link but checking the private link the IP has dynamically changed to 10.10.1.26. In this situation, we checked the DNS server and Local DNS for Private Link and everything is fine, so the next action was to review if we have any configuration in the hosts file of Linux. We found that they have this configuration in their file. 


 


Changing the value of /etc/host file from 10.10.1.25 to 10.10.1.26 everything was started to work correctly and we suggested to discuss with their IT Security team to check why this situation happened or change the private link to static. 


 


Jose_Manuel_Jurado_0-1673182507754.png


 


Enjoy!


 

Azure App Service & ASP.NET Core 3.1 500.30 – One odd cause

This article is contributed. See the original author and article here.

Summary


The problem in this case was, somehow, being caused by the customer’s App Service having the .NET Core 3.1 runtime installed via Site Extension, instead of using the built-in runtime that comes with App Services.


The issue resolved when the Site Extension was removed, and the App Service was stopped and re-started. 


 


Deeper Dive into the Data


This issue showed different symptoms depending on whether the ASP.NET Core app was running in-process or out-of-process. 


 


In-Process


In-process, the symptom was a 500.30 In-Process Start Failure with error code 8007023e. This exception code means “unhandled exception.” Viewing the eventlog.xml in the App Service via Kudu came up with this couplet of events every time:



   
       
        1018
        1
        0
        Keywords
       
        -1368025656
        Application
        [redacted]
       
   
   
       
Application ‘/LM/W3SVC/1365716517/ROOT’ with physical root ‘C:homesitewwwroot’ hit unexpected managed exception, exception code = ‘0xc0000005’. Please check the stderr logs for more information.
        Process Id: 4236.
        File Version: 13.1.22230.29. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 21d42143378ad6cc4bcbaebfda5f3acddf13aa47
   


    …
   
       
Application ‘/LM/W3SVC/1365716517/ROOT’ with physical root ‘C:homesitewwwroot’ failed to load coreclr. Exception message: CLR worker thread exited prematurely
        Process Id: 4236.
        File Version: 13.1.22230.29. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 21d42143378ad6cc4bcbaebfda5f3acddf13aa47
   

 


It seems CoreCLR was trying to load and failed with a native access violation exception (c0000005). Very odd. We did not get a dump of this but I wish we had.


 


Out-of-Process


When switching the app to run out-of-process, we encountered a different error. This is from the eventlog.xml:




   
        Application ‘/LM/W3SVC/1365716517/ROOT’ with physical root ‘C:homesitewwwroot’ failed to start process with commandline ‘”dotnet” .[redacted].dll’ with multiple retries. Failed to bind to port ‘31490’. First 30KB characters of captured stdout and stderr logs from multiple retries:
        Process Id: 7032.
        File Version: 13.1.22287.31. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: fbe05294ac5c88be848b4d57d60cb2657874da9b
   

Nothing really useful there.


 


We enabled AspNetCoreModule’s Enhanced Diagnostic Logging and saw that it was timing out while waiting for the app to report itself as started:


[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x8027025a at D:a_work1ssrcServersIISAspNetCoreModuleV2OutOfProcessRequestHandlerserverprocess.cpp:727 

8027025a= E_APPLICATION_ACTIVATION_TIMED_OUT: The app didn’t start in the required time.


 


We also enabled the stdout log via the web.config and found the app had started just fine:


dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
      Hosting starting

dbug: Microsoft.AspNetCore.Server.Kestrel[0]
      No listening endpoints were configured. Binding to http://localhost:5000 by default.
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000

info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Test2
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:homesitewwwroot
dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
      Hosting started


Turns out the app for some unknown reason was starting on the default localhost:5000. When hosting ASP.NET Core behind IIS and everything is working fine, AspNetCoreModule sets the ASPNETCORE_PORT environment variable to a dynamic port. Kestrel during startup is supposed to poll the value of that variable and use that port to listen on at 127.0.0.1. So in this case something was breaking down either on the environment variable side in ANCM or on the Kestrel side, or in between with the environment itself. Unfortunately we don’t have more data to drill deeper into that because we were tinkering with the App Service based on an observation I made, and the problem appears to have been resolved.

 

Another Observation & Resolution


While perusing the ANCM Enhanced Diagnostic Logging I mentioned earlier, I came across this:

[aspnetcorev2.dll] Initializing logs for ‘C:homeSiteExtensionsAspNetCoreRuntime.3.1.x86ancmaspnetcorev2.dll‘. Process Id: 7632.. File Version: 13.1.22287.31. Description: IIS ASP.NET Core Module V2. Commit: fbe05294ac5c88be848b4d57d60cb2657874da9b.

That struck me as odd because App Services itself provides all the .NET Core runtimes (including no-longer-supported ones like 3.1). So why was ANCM loading from a Site Extension?

In a new, test App Service with a basic ASP.NET Core 3.1 app deployed to it, this is what that log looks like:

[aspnetcorev2.dll] Initializing logs for ‘C:Program Files (x86)IISAsp.Net Core ModuleV2aspnetcorev2.dll‘. Process Id: 7496.. File Version: 13.1.19331.0. Description: IIS ASP.NET Core Module V2. Commit: 62eee6e6d21c95668a9e9529dce6562cc6c9f3bf.

That is where ANCM is normally located.

As a test on one of my own App Services, I installed the latest-available Site Extension for the .NET Core 3.1 runtime. I still had no issues, and I confirmed the ANCM log showed the location of ANCM had changed to the Site Extension one, same as the customer’s. 

 

I, personally, am not familiar with App Service Site Extensions and why the .NET Core runtime is available to use there when it’s already built-in; however, it’s just another copy of the runtime in a different location that theoretically shouldn’t have issues. I will say, in this case the customer had an older version of the runtime installed via the Site Extension, while the latest available was 3.1.32 (the latest build).

 

On the call with the customer, as a test we removed the Site Extension completely, restarted the site, and confirmed ANCM was using the built-in version that comes with App Services. This immediately resolved the issues for both in-process and out-of-process setups.

 

Unfortunately, we likely won’t be able to get more data on this problem and what was happening. I am thinking perhaps the fact that the Site Extension was out-of-date/an older version possibly had something to do with it. Thus, the takeaway here is if you have an app experiencing odd startup issues and if you have a Site Extension installed that contains the runtime for the app you are trying to run, try removing that Site Extension (or maybe update it if it needs to be updated?) and see if your issues go away. Make sure to stop and start the App Service as well, to make sure everything is fully picked-up.

New Year, new Microsoft 365 Core advanced deployment guides for Exchange, SharePoint, and Teams!

New Year, new Microsoft 365 Core advanced deployment guides for Exchange, SharePoint, and Teams!

This article is contributed. See the original author and article here.

New Years’ Resolutions:


  



  1. Work out 3-5 days a week. 

  2. Spend more time with family. 

  3. Deploy Microsoft 365 services to make my company more efficient and secure. 


  


We can’t help you with the first two, but we can help you deploy Microsoft 365 services to improve your company’s efficiency and security with the new Core onboarding advanced deployment guides.  


 


  


Learn how:


 


 


Did you know that thousands of customers use the Core onboarding advanced deployment guides each month? The Microsoft 365 Admin Center features Core advanced deployment guides that simplify moving from On-Premises IT to the Cloud. 


IT Pros can use our core onboarding guides to speed deployment and configure Azure Active Directory, DNS, networking, identity management, and more with advanced deployment guides from Microsoft. 


 


 


Core onboarding involves service provisioning and tenant and identity integration. The Core onboarding advanced deployment guides for onboarding services include:  



 


Here’s how IT Pros can access the Core advanced deployment guides by clicking the purple hammer link in the below tile. 


An image demonstrating how to access the Core onboarding advanced deployment guides in the Training, guides, & assistance section of the Microsoft 365 admin center.An image demonstrating how to access the Core onboarding advanced deployment guides in the Training, guides, & assistance section of the Microsoft 365 admin center.


 


 


Once there, Core Advanced deployment guides are found in the Identity and authentication section.


 


An image demonstrating how to access the Identity and authentication section.An image demonstrating how to access the Identity and authentication section.


 


 


Completing the deployment steps outlined in the Core advanced deployment guides creates a strong foundation upon which additional Microsoft 365 Cloud Services can be deployed successfully and with confidence.


 


Helpful resources


 


Learn about all deployment guides and setup wizards:



Access advanced deployment guides without signing in:



Getting started with advanced deployment guides within the Microsoft 365 admin center:



  • Go to https://admin.microsoft.com/ and log in

  • Once logged in, go to the Home page and look for the Training, guides, & assistance card 

  • Select Advanced deployment guides & assistance