changeset 5346:566e8d32bd3a

Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
author Alain Mazy <am@osimis.io>
date Thu, 29 Jun 2023 09:43:20 +0200
parents 65d55cc86a41
children 4f686f6150c7
files NEWS OrthancServer/Resources/ImplementationNotes/memory_consumption.txt OrthancServer/Sources/ServerContext.cpp OrthancServer/Sources/ServerContext.h
diffstat 4 files changed, 42 insertions(+), 10 deletions(-) [+]
line wrap: on
line diff
--- a/NEWS	Wed Jun 28 11:54:41 2023 +0200
+++ b/NEWS	Thu Jun 29 09:43:20 2023 +0200
@@ -5,7 +5,10 @@
 -------
 
 * Orthanc now anonymizes according to Basic Profile of PS 3.15-2023b Table E.1-1
-* Added metrics "orthanc_storage_read_bytes" and "orthanc_storage_written_bytes"
+* Added metrics:
+  - "orthanc_storage_read_bytes" 
+  - "orthanc_storage_written_bytes"
+  - "orthanc_memory_trimming_duration_ms"
 
 REST API
 --------
@@ -34,6 +37,8 @@
 * Metrics can be stored either as floating-point numbers, or as integers
 * Upgraded dependencies for static builds:
   - boost 1.82.0
+* Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle
+  CPU load (https://discourse.orthanc-server.org/t/onchange-callbacks-and-cpu-loads/3534).
 
 
 Version 1.12.0 (2023-04-14)
--- a/OrthancServer/Resources/ImplementationNotes/memory_consumption.txt	Wed Jun 28 11:54:41 2023 +0200
+++ b/OrthancServer/Resources/ImplementationNotes/memory_consumption.txt	Thu Jun 29 09:43:20 2023 +0200
@@ -1,6 +1,7 @@
 In Orthanc 1.11.3, we have introduced a Housekeeper thread that 
 tries to give back unused memory back to the system.  This is implemented 
-by calling malloc_trim every 100ms.
+by calling malloc_trim every 30s (note: on 1.11.3 and 1.12.0, the interval
+was 100ms which caused high idle CPU load).
 
 
 Here is how we validated the effect of this new feature:
@@ -133,3 +134,16 @@
   specifies the minimum size that is released.  So, even without
   malloc_trim, Orthanc is able to give back memory to the system.
 - free() never gives back block allocated by mmap() to the system, only malloc_trim() does !
+
+UPDATE on June 2023:
+-------------------
+
+Given this discussion: https://discourse.orthanc-server.org/t/onchange-callbacks-and-cpu-loads/3534,
+changed the interval from 100ms to 30s.
+We also added a metrics to monitor the duration: orthanc_memory_trimming_duration_ms
+
+Good reference article:
+https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory/
+
+
+
--- a/OrthancServer/Sources/ServerContext.cpp	Wed Jun 28 11:54:41 2023 +0200
+++ b/OrthancServer/Sources/ServerContext.cpp	Thu Jun 29 09:43:20 2023 +0200
@@ -110,16 +110,28 @@
 
 #if HAVE_MALLOC_TRIM == 1
   void ServerContext::MemoryTrimmingThread(ServerContext* that,
-                                           unsigned int sleepDelay)
+                                           unsigned int intervalInSeconds)
   {
+    boost::posix_time::ptime lastExecution = boost::posix_time::second_clock::universal_time();
+
     // This thread is started only if malloc_trim is defined
     while (!that->done_)
     {
-      boost::this_thread::sleep(boost::posix_time::milliseconds(sleepDelay));
-      
-      // If possible, gives memory back to the system 
-      // (see OrthancServer/Resources/ImplementationNotes/memory_consumption.txt)
-      malloc_trim(128*1024);
+      boost::posix_time::ptime now = boost::posix_time::second_clock::universal_time();
+      boost::posix_time::time_duration elapsed = now - lastExecution;
+
+      if (elapsed.total_seconds() > intervalInSeconds)
+      {
+        // If possible, gives memory back to the system 
+        // (see OrthancServer/Resources/ImplementationNotes/memory_consumption.txt)
+        {
+          MetricsRegistry::Timer timer(that->GetMetricsRegistry(), "orthanc_memory_trimming_duration_ms");
+          malloc_trim(128*1024);
+        }
+        lastExecution = boost::posix_time::second_clock::universal_time();
+      }
+
+      boost::this_thread::sleep(boost::posix_time::milliseconds(100));
     }
   }
 #endif
@@ -439,7 +451,8 @@
       changeThread_ = boost::thread(ChangeThread, this, (unitTesting ? 20 : 100));
       
 #if HAVE_MALLOC_TRIM == 1
-      memoryTrimmingThread_ = boost::thread(MemoryTrimmingThread, this, 100);
+      LOG(INFO) << "Starting memory trimming thread at 30 seconds interval";
+      memoryTrimmingThread_ = boost::thread(MemoryTrimmingThread, this, 30);
 #else
       LOG(INFO) << "Your platform does not support malloc_trim(), not starting the memory trimming thread";
 #endif
--- a/OrthancServer/Sources/ServerContext.h	Wed Jun 28 11:54:41 2023 +0200
+++ b/OrthancServer/Sources/ServerContext.h	Thu Jun 29 09:43:20 2023 +0200
@@ -189,7 +189,7 @@
 
 #if HAVE_MALLOC_TRIM == 1
     static void MemoryTrimmingThread(ServerContext* that,
-                                     unsigned int sleepDelay);
+                                     unsigned int intervalInSeconds);
 #endif
 
     void SaveJobsEngine();