Mercurial > hg > orthanc
annotate OrthancServer/Resources/ImplementationNotes/memory_consumption.txt @ 5802:816416425f2b Orthanc-1.12.4
fix unit tests
author | Alain Mazy <am@orthanc.team> |
---|---|
date | Fri, 20 Sep 2024 16:07:08 +0200 |
parents | 566e8d32bd3a |
children |
rev | line source |
---|---|
5153 | 1 In Orthanc 1.11.3, we have introduced a Housekeeper thread that |
2 tries to give back unused memory back to the system. This is implemented | |
5346
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
3 by calling malloc_trim every 30s (note: on 1.11.3 and 1.12.0, the interval |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
4 was 100ms which caused high idle CPU load). |
5153 | 5 |
5156 | 6 |
5153 | 7 Here is how we validated the effect of this new feature: |
8 ------------------------------------------------------- | |
9 | |
10 We compared the behaviour of 2 osimis/orthanc Docker images from the mainline | |
11 on Feb 1st 2023. One image without the call to malloc_trim and the other with | |
12 this call. | |
13 | |
5156 | 14 |
5153 | 15 1st test: unconstrained Docker containers |
16 ......................................... | |
17 | |
18 5 large studies are uploaded to each instance of Orthanc (around 1GB in total). | |
19 A script triggers anonymization of these studies as quick as possible. | |
20 We compare the memory used by the containers after 2 minutes of execution | |
21 (using `docker stats`): | |
22 - without malloc_trim: 1500 MB | |
23 - with malloc_trim: 410 MB | |
24 | |
5156 | 25 |
5153 | 26 2nd test: memory constrained Docker containers |
27 .............................................. | |
28 | |
29 Each Orthanc container is limited to 400MB (through the docker-compose configuration | |
30 `mem_limit: 400m`) | |
31 5 large studies are uploaded to each instance of Orthanc (around 1GB in total). | |
32 Each study is anonymized manually, one by one and then, we repeat the operation. | |
5157 | 33 We compare the memory used by the containers after each anonymization |
5153 | 34 (using `docker stats`): |
35 | |
36 # study without malloc_trim with_malloc_trim | |
37 0 ~ 50 MB ~ 50 MB | |
38 1 ~ 140 MB ~ 140 MB | |
39 2 ~ 390 MB ~ 340 MB | |
40 3 ~ 398 MB ~ 345 MB | |
41 4 out-of-memory crash ~ 345 MB | |
42 5..20 ~ 380 MB (stable) | |
43 | |
5156 | 44 |
45 3rd test: memory constrained Docker containers | |
5155 | 46 .............................................. |
47 | |
48 In this last test, we lowered the memory allocation to 300MB and have been able to | |
49 run the first test script for at least 7 minutes (we did not try longer !). The | |
50 consumed memory is most of the time around 99% but it seems that the memory constrain | |
51 is handled correctly. Note that, in this configuration, 128 MB are used by the Dicom | |
52 Cache. | |
53 | |
54 The same test without malloc_trim could never run for more than 35 seconds. | |
55 | |
56 | |
5156 | 57 4th test: performance impact of malloc_trim and available memory |
58 ................................................................ | |
59 | |
60 In this test, we have measured the time required to anonymize a 2000 instances study | |
61 with various configurations. It appears that malloc_trim or the total amount | |
5158 | 62 of memory available in the system has no significant impact on performance. |
5156 | 63 |
64 - No malloc trim, 300 MB in the system: ~ 38s | |
65 - No malloc trim, 1500 MB in the system: ~ 38s | |
66 - With malloc trim, 300 MB in the system: ~ 38s | |
67 - With malloc trim, 1500 MB in the system: ~ 38s | |
68 | |
69 | |
70 Conclusion: | |
71 ---------- | |
5155 | 72 |
5157 | 73 The use of malloc_trim reduces the overall memory consumption of Orthanc |
74 and avoids some of the out-of-memory situations. | |
75 | |
76 However, it does not guarantee that Orthanc will never reach a | |
5153 | 77 out-of-memory error, especially on very constrained systems. |
5157 | 78 |
5153 | 79 Depending on the allocation pattern, the Orthanc memory can get |
5157 | 80 very fragmented and increase regularly since malloc_trim only releases memory |
5153 | 81 at the end of each of malloc arena. However, note that, even long before the |
82 introduction of malloc_trim, we have observed Orthanc instances running for years | |
83 without ever reaching out-of-memory errors and Orthanc is usually considered as | |
84 very stable. | |
85 | |
5157 | 86 Moreover, before each release, Orthanc integration tests are run against Valgrind |
87 and no memory leaks have been identified. | |
5153 | 88 |
89 | |
90 malloc_trim documentation | |
91 ------------------------- | |
92 | |
93 from (https://stackoverflow.com/questions/40513716/malloc-trim0-releases-fastbins-of-thread-arenas) | |
94 | |
95 If possible, gives memory back to the system (via negative | |
96 arguments to sbrk) if there is unused memory at the `high' end of | |
97 the malloc pool. You can call this after freeing large blocks of | |
98 memory to potentially reduce the system-level memory requirements | |
99 of a program. However, it cannot guarantee to reduce memory. Under | |
100 some allocation patterns, some large free blocks of memory will be | |
101 locked between two used chunks, so they cannot be given back to | |
102 the system. | |
103 | |
104 The `pad' argument to malloc_trim represents the amount of free | |
105 trailing space to leave untrimmed. If this argument is zero, | |
106 only the minimum amount of memory to maintain internal data | |
107 structures will be left (one page or less). Non-zero arguments | |
108 can be supplied to maintain enough trailing space to service | |
109 future expected allocations without having to re-obtain memory | |
110 from the system. | |
111 | |
112 Malloc_trim returns 1 if it actually released any memory, else 0. | |
113 On systems that do not support "negative sbrks", it will always | |
114 return 0. | |
5159
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
115 |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
116 |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
117 glibc internals |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
118 --------------- |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
119 |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
120 Lots of useful info here: https://man7.org/linux/man-pages/man3/mallopt.3.html |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
121 |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
122 summary: |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
123 - malloc uses sbrk() or mmap() to allocate memory. mmap() is used to allocate |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
124 large memory chunks, larger than M_MMAP_THRESHOLD. |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
125 - about mmap(): On the other hand, there are some disadvantages to |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
126 the use of mmap(2): deallocated space is not placed on the |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
127 free list for reuse by later allocations; memory may be |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
128 wasted because mmap(2) allocations must be page-aligned; |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
129 and the kernel must perform the expensive task of zeroing |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
130 out memory allocated via mmap(2). Balancing these factors |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
131 leads to a default setting of 128*1024 for the |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
132 M_MMAP_THRESHOLD parameter. |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
133 - free() employs sbrk() to release memory back to the system and M_TRIM_THRESHOLD |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
134 specifies the minimum size that is released. So, even without |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
135 malloc_trim, Orthanc is able to give back memory to the system. |
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
136 - free() never gives back block allocated by mmap() to the system, only malloc_trim() does ! |
5346
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
137 |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
138 UPDATE on June 2023: |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
139 ------------------- |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
140 |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
141 Given this discussion: https://discourse.orthanc-server.org/t/onchange-callbacks-and-cpu-loads/3534, |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
142 changed the interval from 100ms to 30s. |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
143 We also added a metrics to monitor the duration: orthanc_memory_trimming_duration_ms |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
144 |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
145 Good reference article: |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
146 https://www.algolia.com/blog/engineering/when-allocators-are-hoarding-your-precious-memory/ |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
147 |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
148 |
566e8d32bd3a
Reduce the frequency of memory trimming from 100ms to 30s to avoid high idle CPU load
Alain Mazy <am@osimis.io>
parents:
5159
diff
changeset
|
149 |