Mercurial > hg > orthanc
annotate OrthancServer/Resources/ImplementationNotes/memory_consumption.txt @ 5719:f488fc5c972a find-refactoring-clean tip
removed StatelessDatabaseOperations::ExpandedResource
| author | Sebastien Jodogne <s.jodogne@gmail.com> |
|---|---|
| date | Sat, 13 Jul 2024 01:01:50 +0200 |
| parents | 566e8d32bd3a |
| children |
| rev | line source |
|---|---|
| 5153 | 1 In Orthanc 1.11.3, we have introduced a Housekeeper thread that |
| 2 tries to give back unused memory back to the system. This is implemented | |
|
5346
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
3 by calling malloc_trim every 30s (note: on 1.11.3 and 1.12.0, the interval |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
4 was 100ms which caused high idle CPU load). |
| 5153 | 5 |
| 5156 | 6 |
| 5153 | 7 Here is how we validated the effect of this new feature: |
| 8 ------------------------------------------------------- | |
| 9 | |
| 10 We compared the behaviour of 2 osimis/orthanc Docker images from the mainline | |
| 11 on Feb 1st 2023. One image without the call to malloc_trim and the other with | |
| 12 this call. | |
| 13 | |
| 5156 | 14 |
| 5153 | 15 1st test: unconstrained Docker containers |
| 16 ......................................... | |
| 17 | |
| 18 5 large studies are uploaded to each instance of Orthanc (around 1GB in total). | |
| 19 A script triggers anonymization of these studies as quick as possible. | |
| 20 We compare the memory used by the containers after 2 minutes of execution | |
| 21 (using `docker stats`): | |
| 22 - without malloc_trim: 1500 MB | |
| 23 - with malloc_trim: 410 MB | |
| 24 | |
| 5156 | 25 |
| 5153 | 26 2nd test: memory constrained Docker containers |
| 27 .............................................. | |
| 28 | |
| 29 Each Orthanc container is limited to 400MB (through the docker-compose configuration | |
| 30 `mem_limit: 400m`) | |
| 31 5 large studies are uploaded to each instance of Orthanc (around 1GB in total). | |
| 32 Each study is anonymized manually, one by one and then, we repeat the operation. | |
| 5157 | 33 We compare the memory used by the containers after each anonymization |
| 5153 | 34 (using `docker stats`): |
| 35 | |
| 36 # study without malloc_trim with_malloc_trim | |
| 37 0 ~ 50 MB ~ 50 MB | |
| 38 1 ~ 140 MB ~ 140 MB | |
| 39 2 ~ 390 MB ~ 340 MB | |
| 40 3 ~ 398 MB ~ 345 MB | |
| 41 4 out-of-memory crash ~ 345 MB | |
| 42 5..20 ~ 380 MB (stable) | |
| 43 | |
| 5156 | 44 |
| 45 3rd test: memory constrained Docker containers | |
| 5155 | 46 .............................................. |
| 47 | |
| 48 In this last test, we lowered the memory allocation to 300MB and have been able to | |
| 49 run the first test script for at least 7 minutes (we did not try longer !). The | |
| 50 consumed memory is most of the time around 99% but it seems that the memory constrain | |
| 51 is handled correctly. Note that, in this configuration, 128 MB are used by the Dicom | |
| 52 Cache. | |
| 53 | |
| 54 The same test without malloc_trim could never run for more than 35 seconds. | |
| 55 | |
| 56 | |
| 5156 | 57 4th test: performance impact of malloc_trim and available memory |
| 58 ................................................................ | |
| 59 | |
| 60 In this test, we have measured the time required to anonymize a 2000 instances study | |
| 61 with various configurations. It appears that malloc_trim or the total amount | |
| 5158 | 62 of memory available in the system has no significant impact on performance. |
| 5156 | 63 |
| 64 - No malloc trim, 300 MB in the system: ~ 38s | |
| 65 - No malloc trim, 1500 MB in the system: ~ 38s | |
| 66 - With malloc trim, 300 MB in the system: ~ 38s | |
| 67 - With malloc trim, 1500 MB in the system: ~ 38s | |
| 68 | |
| 69 | |
| 70 Conclusion: | |
| 71 ---------- | |
| 5155 | 72 |
| 5157 | 73 The use of malloc_trim reduces the overall memory consumption of Orthanc |
| 74 and avoids some of the out-of-memory situations. | |
| 75 | |
| 76 However, it does not guarantee that Orthanc will never reach a | |
| 5153 | 77 out-of-memory error, especially on very constrained systems. |
| 5157 | 78 |
| 5153 | 79 Depending on the allocation pattern, the Orthanc memory can get |
| 5157 | 80 very fragmented and increase regularly since malloc_trim only releases memory |
| 5153 | 81 at the end of each of malloc arena. However, note that, even long before the |
| 82 introduction of malloc_trim, we have observed Orthanc instances running for years | |
| 83 without ever reaching out-of-memory errors and Orthanc is usually considered as | |
| 84 very stable. | |
| 85 | |
| 5157 | 86 Moreover, before each release, Orthanc integration tests are run against Valgrind |
| 87 and no memory leaks have been identified. | |
| 5153 | 88 |
| 89 | |
| 90 malloc_trim documentation | |
| 91 ------------------------- | |
| 92 | |
| 93 from (https://stackoverflow.com/questions/40513716/malloc-trim0-releases-fastbins-of-thread-arenas) | |
| 94 | |
| 95 If possible, gives memory back to the system (via negative | |
| 96 arguments to sbrk) if there is unused memory at the `high' end of | |
| 97 the malloc pool. You can call this after freeing large blocks of | |
| 98 memory to potentially reduce the system-level memory requirements | |
| 99 of a program. However, it cannot guarantee to reduce memory. Under | |
| 100 some allocation patterns, some large free blocks of memory will be | |
| 101 locked between two used chunks, so they cannot be given back to | |
| 102 the system. | |
| 103 | |
| 104 The `pad' argument to malloc_trim represents the amount of free | |
| 105 trailing space to leave untrimmed. If this argument is zero, | |
| 106 only the minimum amount of memory to maintain internal data | |
| 107 structures will be left (one page or less). Non-zero arguments | |
| 108 can be supplied to maintain enough trailing space to service | |
| 109 future expected allocations without having to re-obtain memory | |
| 110 from the system. | |
| 111 | |
| 112 Malloc_trim returns 1 if it actually released any memory, else 0. | |
| 113 On systems that do not support "negative sbrks", it will always | |
| 114 return 0. | |
|
5159
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
115 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
116 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
117 glibc internals |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
118 --------------- |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
119 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
120 Lots of useful info here: https://man7.org/linux/man-pages/man3/mallopt.3.html |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
121 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
122 summary: |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
123 - malloc uses sbrk() or mmap() to allocate memory. mmap() is used to allocate |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
124 large memory chunks, larger than M_MMAP_THRESHOLD. |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
125 - about mmap(): On the other hand, there are some disadvantages to |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
126 the use of mmap(2): deallocated space is not placed on the |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
127 free list for reuse by later allocations; memory may be |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
128 wasted because mmap(2) allocations must be page-aligned; |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
129 and the kernel must perform the expensive task of zeroing |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
130 out memory allocated via mmap(2). Balancing these factors |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
131 leads to a default setting of 128*1024 for the |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
132 M_MMAP_THRESHOLD parameter. |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
133 - free() employs sbrk() to release memory back to the system and M_TRIM_THRESHOLD |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
134 specifies the minimum size that is released. So, even without |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
135 malloc_trim, Orthanc is able to give back memory to the system. |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
136 - free() never gives back block allocated by mmap() to the system, only malloc_trim() does ! |
|
5346
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
137 |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
138 UPDATE on June 2023: |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
139 ------------------- |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
140 |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
141 Given this discussion: https://discourse.orthanc-server.org/t/onchange-callbacks-and-cpu-loads/3534, |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
142 changed the interval from 100ms to 30s. |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
143 We also added a metrics to monitor the duration: orthanc_memory_trimming_duration_ms |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
144 |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
145 Good reference article: |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
146 https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory/ |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
147 |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
148 |
|
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
149 |