changeset 5159:f5907aecbaed

conditional usage of malloc_trim
author Alain Mazy <am@osimis.io>
date Thu, 02 Feb 2023 11:28:35 +0100
parents 02cfd23a556a
children 9f16979b6124
files NEWS OrthancServer/CMakeLists.txt OrthancServer/Resources/ImplementationNotes/memory_consumption.txt OrthancServer/Resources/RunCppCheck.sh OrthancServer/Sources/ServerContext.cpp
diffstat 5 files changed, 40 insertions(+), 7 deletions(-) [+]
line wrap: on
line diff
--- a/NEWS	Thu Feb 02 10:33:15 2023 +0100
+++ b/NEWS	Thu Feb 02 11:28:35 2023 +0100
@@ -12,11 +12,10 @@
 * New configuration "KeepAliveTimeout" with a default value of 1 second.
 * ResourceModification jobs (/modify + /anonymize) can now use multiple threads to speed up processing
   - New configuration "JobsEngineThreadsCount.ResourceModification" to configure the number of threads.
-* Introduced a new Housekeeper thread in Orthanc (different from the Housekeeper sample plugin).  This thread
+* For systems using glibc > 2.8 (most of Linux systems except LSB binaries):
+  Introduced a new Housekeeper thread in Orthanc (different from the Housekeeper sample plugin).  This thread
   regularly try to give back memory that Orthanc no longer uses to the system.  This reduces the overall memory
-  consumption.  However, only the memory at the end of a memory arena is given back to the system.  Fragmented memory
-  is not given back and therefore, the memory consumption may still stay high.  More information in 
-  OrthancServer/Resources/ImplementationNotes/memory_consumption.txt.
+  consumption.  More information in  OrthancServer/Resources/ImplementationNotes/memory_consumption.txt.
 
 REST API
 --------
--- a/OrthancServer/CMakeLists.txt	Thu Feb 02 10:33:15 2023 +0100
+++ b/OrthancServer/CMakeLists.txt	Thu Feb 02 11:28:35 2023 +0100
@@ -281,6 +281,7 @@
 #####################################################################
 
 check_symbol_exists(mallopt "malloc.h" HAVE_MALLOPT)
+check_symbol_exists(malloc_trim "malloc.h" HAVE_MALLOC_TRIM)
 
 if (HAVE_MALLOPT)
   add_definitions(-DHAVE_MALLOPT=1)
@@ -288,6 +289,11 @@
   add_definitions(-DHAVE_MALLOPT=0)
 endif()
 
+if (HAVE_MALLOC_TRIM)
+  add_definitions(-DHAVE_MALLOC_TRIM=1)
+else()
+  add_definitions(-DHAVE_MALLOC_TRIM=0)
+endif()
 
 if (STATIC_BUILD)
   add_definitions(-DORTHANC_STATIC=1)
--- a/OrthancServer/Resources/ImplementationNotes/memory_consumption.txt	Thu Feb 02 10:33:15 2023 +0100
+++ b/OrthancServer/Resources/ImplementationNotes/memory_consumption.txt	Thu Feb 02 11:28:35 2023 +0100
@@ -111,3 +111,25 @@
     Malloc_trim returns 1 if it actually released any memory, else 0.
     On systems that do not support "negative sbrks", it will always
     return 0.
+
+
+glibc internals
+---------------
+
+Lots of useful info here: https://man7.org/linux/man-pages/man3/mallopt.3.html
+
+summary:
+- malloc uses sbrk() or mmap() to allocate memory.  mmap() is used to allocate 
+  large memory chunks, larger than M_MMAP_THRESHOLD.
+- about mmap(): On the other hand, there are some disadvantages to
+              the use of mmap(2): deallocated space is not placed on the
+              free list for reuse by later allocations; memory may be
+              wasted because mmap(2) allocations must be page-aligned;
+              and the kernel must perform the expensive task of zeroing
+              out memory allocated via mmap(2).  Balancing these factors
+              leads to a default setting of 128*1024 for the
+              M_MMAP_THRESHOLD parameter.
+- free() employs sbrk() to release memory back to the system and M_TRIM_THRESHOLD
+  specifies the minimum size that is released.  So, even without
+  malloc_trim, Orthanc is able to give back memory to the system.
+- free() never gives back block allocated by mmap() to the system, only malloc_trim() does !
--- a/OrthancServer/Resources/RunCppCheck.sh	Thu Feb 02 10:33:15 2023 +0100
+++ b/OrthancServer/Resources/RunCppCheck.sh	Thu Feb 02 11:28:35 2023 +0100
@@ -49,6 +49,7 @@
             -DCIVETWEB_HAS_WEBDAV_WRITING=1 \
             -DDCMTK_VERSION_NUMBER=365 \
             -DHAVE_MALLOPT=1 \
+            -DHAVE_MALLOC_TRIM=1 \
             -DMONGOOSE_USE_CALLBACKS=1 \
             -DJSONCPP_VERSION_MAJOR=1 \
             -DJSONCPP_VERSION_MINOR=0 \
--- a/OrthancServer/Sources/ServerContext.cpp	Thu Feb 02 10:33:15 2023 +0100
+++ b/OrthancServer/Sources/ServerContext.cpp	Thu Feb 02 11:28:35 2023 +0100
@@ -107,14 +107,16 @@
   void ServerContext::HousekeeperThread(ServerContext* that,
                                         unsigned int sleepDelay)
   {
+    // note, right now, this thread is started only if malloc_trim is defined
     while (!that->done_)
     {
       boost::this_thread::sleep(boost::posix_time::milliseconds(sleepDelay));
       
       // If possible, gives memory back to the system 
       // (see OrthancServer/Resources/ImplementationNotes/memory_consumption.txt)
-
-      malloc_trim(256*1024);
+#if HAVE_MALLOC_TRIM == 1
+      malloc_trim(128*1024);
+#endif
     }
   }
   
@@ -430,8 +432,11 @@
 
       listeners_.push_back(ServerListener(luaListener_, "Lua"));
       changeThread_ = boost::thread(ChangeThread, this, (unitTesting ? 20 : 100));
+#if HAVE_MALLOC_TRIM == 1
       housekeeperThread_ = boost::thread(HousekeeperThread, this, 100);
-
+#else
+      LOG(INFO) << "Your platform does not support malloc_trim(), not starting the housekeeper thread";
+#endif
       dynamic_cast<DcmtkTranscoder&>(*dcmtkTranscoder_).SetLossyQuality(lossyQuality);
     }
     catch (OrthancException&)