Lesson Learned #459:HikariCP – Unusual system clock change detected, soft-evicting connections pool

by | Dec 16, 2023 | Technology | 0 comments

This article is contributed. See the original author and article here.

This week, we’ve been tackling a support case where our client encountered the following error message: ‘[HikariPool] 2023-11-29 04:13:20,491 WARN (HikariPool.java:602) %h – AppNamePool – Unusual system clock change detected, soft-evicting connections from pool.‘ I’d like to share with you the troubleshooting steps we undertook to diagnose this error message.


 


The first step is to understand when this message occurs in the HikariCP code. To do this, I checked various versions of HikariCP and found that in version 2.4.0 this error message appears in the HikariPool.java within the private class HouseKeeper in the version, for example, HikariCP/src/main/java/com/zaxxer/hikari/pool/HikariPool.java at HikariCP-2.4.0 · brettwooldridge/HikariCP (github.com),


 


Also, we identified a possible bug (Hikari might not recover from state “Unusual system clock change detected” · Issue #354 · brettwooldridge/HikariCP (github.com)) in this code but, basically, this error means:


 



  • “Unusual system clock change detected, soft-evicting connections from pool.” is a warning from HikariCP. It indicates that HikariCP has detected an unusual change in the system clock and is performing a “soft eviction” of connections from the pool. This message relates to connection management and handling potential inconsistencies caused by system time changes.

  • A “soft eviction” means that HikariCP immediately closes connections if they are not being used by the application or simply marks them for eviction later when the connection is returned to the pool. That is, connections in use are not interrupted but are flagged to be closed once they return to the pool. The rationale behind this is that a significant change in the system clock, either backward or forward, can impact HikariCP’s calculations related to idle time, lifetime, or various connection timeouts, potentially leading to unexpected or unstable pool behavior.



  • This behavior in HikariCP is triggered when the system clock moves backward or jumps forward beyond an acceptable threshold. For instance, if the clock moves backward or jumps forward by more than a minute, HikariCP will proceed to evict connections. This mechanism serves as a precaution to handle situations where the system time is unreliable.

  • It’s important to note that these unusual changes in the system clock can be caused by various factors, including NTP adjustments, power-saving modes on laptops, or even the JVM temporarily hanging. Developers need to be aware of these possibilities and how they might affect the operation of their applications, especially in production environments where a high number of database connections is critical. 


 


In summary, HikariCP’s message is an indication that it is actively managing connections in response to unusual changes in the system time, thereby ensuring the stability and consistency of the connections in the pool. Checking new versions of HikariCP we’ve found that this code has been modified and changed the error message. brettwooldridge/HikariCP: 光 HikariCP・A solid, high-performance, JDBC connection pool at last. (github.com)


 


 

            // Detect retrograde time, allowing +128ms as per NTP spec.
            if (plusMillis(now, 128) < plusMillis(previous, housekeepingPeriodMs)) {
               logger.warn("{} - Retrograde clock change detected (housekeeper delta={}), soft-evicting connections from pool.",
                           poolName, elapsedDisplayString(previous, now));
               previous = now;
               softEvictConnections();
               return;

 


 


So, in this such situation, it is important to maintain the system clock update to advoid this type of issue that is not related with Azure SQL Database, used the latest version of HikariCP and in case of any additional questions or issue open a service request with this team. 


 


Enjoy!


 

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Submit a Comment