Mercurial > hg > orthanc
annotate OrthancServer/Resources/ImplementationNotes/memory_consumption.txt @ 5294:e0e2aee4453e
Modality worklists plugin: allow searching on private tags (exact match only)
| author | Alain Mazy <am@osimis.io> |
|---|---|
| date | Wed, 10 May 2023 12:52:35 +0200 |
| parents | f5907aecbaed |
| children | 566e8d32bd3a |
| rev | line source |
|---|---|
| 5153 | 1 In Orthanc 1.11.3, we have introduced a Housekeeper thread that |
| 2 tries to give back unused memory back to the system. This is implemented | |
| 3 by calling malloc_trim every 100ms. | |
| 4 | |
| 5156 | 5 |
| 5153 | 6 Here is how we validated the effect of this new feature: |
| 7 ------------------------------------------------------- | |
| 8 | |
| 9 We compared the behaviour of 2 osimis/orthanc Docker images from the mainline | |
| 10 on Feb 1st 2023. One image without the call to malloc_trim and the other with | |
| 11 this call. | |
| 12 | |
| 5156 | 13 |
| 5153 | 14 1st test: unconstrained Docker containers |
| 15 ......................................... | |
| 16 | |
| 17 5 large studies are uploaded to each instance of Orthanc (around 1GB in total). | |
| 18 A script triggers anonymization of these studies as quick as possible. | |
| 19 We compare the memory used by the containers after 2 minutes of execution | |
| 20 (using `docker stats`): | |
| 21 - without malloc_trim: 1500 MB | |
| 22 - with malloc_trim: 410 MB | |
| 23 | |
| 5156 | 24 |
| 5153 | 25 2nd test: memory constrained Docker containers |
| 26 .............................................. | |
| 27 | |
| 28 Each Orthanc container is limited to 400MB (through the docker-compose configuration | |
| 29 `mem_limit: 400m`) | |
| 30 5 large studies are uploaded to each instance of Orthanc (around 1GB in total). | |
| 31 Each study is anonymized manually, one by one and then, we repeat the operation. | |
| 5157 | 32 We compare the memory used by the containers after each anonymization |
| 5153 | 33 (using `docker stats`): |
| 34 | |
| 35 # study without malloc_trim with_malloc_trim | |
| 36 0 ~ 50 MB ~ 50 MB | |
| 37 1 ~ 140 MB ~ 140 MB | |
| 38 2 ~ 390 MB ~ 340 MB | |
| 39 3 ~ 398 MB ~ 345 MB | |
| 40 4 out-of-memory crash ~ 345 MB | |
| 41 5..20 ~ 380 MB (stable) | |
| 42 | |
| 5156 | 43 |
| 44 3rd test: memory constrained Docker containers | |
| 5155 | 45 .............................................. |
| 46 | |
| 47 In this last test, we lowered the memory allocation to 300MB and have been able to | |
| 48 run the first test script for at least 7 minutes (we did not try longer !). The | |
| 49 consumed memory is most of the time around 99% but it seems that the memory constrain | |
| 50 is handled correctly. Note that, in this configuration, 128 MB are used by the Dicom | |
| 51 Cache. | |
| 52 | |
| 53 The same test without malloc_trim could never run for more than 35 seconds. | |
| 54 | |
| 55 | |
| 5156 | 56 4th test: performance impact of malloc_trim and available memory |
| 57 ................................................................ | |
| 58 | |
| 59 In this test, we have measured the time required to anonymize a 2000 instances study | |
| 60 with various configurations. It appears that malloc_trim or the total amount | |
| 5158 | 61 of memory available in the system has no significant impact on performance. |
| 5156 | 62 |
| 63 - No malloc trim, 300 MB in the system: ~ 38s | |
| 64 - No malloc trim, 1500 MB in the system: ~ 38s | |
| 65 - With malloc trim, 300 MB in the system: ~ 38s | |
| 66 - With malloc trim, 1500 MB in the system: ~ 38s | |
| 67 | |
| 68 | |
| 69 Conclusion: | |
| 70 ---------- | |
| 5155 | 71 |
| 5157 | 72 The use of malloc_trim reduces the overall memory consumption of Orthanc |
| 73 and avoids some of the out-of-memory situations. | |
| 74 | |
| 75 However, it does not guarantee that Orthanc will never reach a | |
| 5153 | 76 out-of-memory error, especially on very constrained systems. |
| 5157 | 77 |
| 5153 | 78 Depending on the allocation pattern, the Orthanc memory can get |
| 5157 | 79 very fragmented and increase regularly since malloc_trim only releases memory |
| 5153 | 80 at the end of each of malloc arena. However, note that, even long before the |
| 81 introduction of malloc_trim, we have observed Orthanc instances running for years | |
| 82 without ever reaching out-of-memory errors and Orthanc is usually considered as | |
| 83 very stable. | |
| 84 | |
| 5157 | 85 Moreover, before each release, Orthanc integration tests are run against Valgrind |
| 86 and no memory leaks have been identified. | |
| 5153 | 87 |
| 88 | |
| 89 malloc_trim documentation | |
| 90 ------------------------- | |
| 91 | |
| 92 from (https://stackoverflow.com/questions/40513716/malloc-trim0-releases-fastbins-of-thread-arenas) | |
| 93 | |
| 94 If possible, gives memory back to the system (via negative | |
| 95 arguments to sbrk) if there is unused memory at the `high' end of | |
| 96 the malloc pool. You can call this after freeing large blocks of | |
| 97 memory to potentially reduce the system-level memory requirements | |
| 98 of a program. However, it cannot guarantee to reduce memory. Under | |
| 99 some allocation patterns, some large free blocks of memory will be | |
| 100 locked between two used chunks, so they cannot be given back to | |
| 101 the system. | |
| 102 | |
| 103 The `pad' argument to malloc_trim represents the amount of free | |
| 104 trailing space to leave untrimmed. If this argument is zero, | |
| 105 only the minimum amount of memory to maintain internal data | |
| 106 structures will be left (one page or less). Non-zero arguments | |
| 107 can be supplied to maintain enough trailing space to service | |
| 108 future expected allocations without having to re-obtain memory | |
| 109 from the system. | |
| 110 | |
| 111 Malloc_trim returns 1 if it actually released any memory, else 0. | |
| 112 On systems that do not support "negative sbrks", it will always | |
| 113 return 0. | |
|
5159
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
114 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
115 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
116 glibc internals |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
117 --------------- |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
118 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
119 Lots of useful info here: https://man7.org/linux/man-pages/man3/mallopt.3.html |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
120 |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
121 summary: |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
122 - malloc uses sbrk() or mmap() to allocate memory. mmap() is used to allocate |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
123 large memory chunks, larger than M_MMAP_THRESHOLD. |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
124 - about mmap(): On the other hand, there are some disadvantages to |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
125 the use of mmap(2): deallocated space is not placed on the |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
126 free list for reuse by later allocations; memory may be |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
127 wasted because mmap(2) allocations must be page-aligned; |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
128 and the kernel must perform the expensive task of zeroing |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
129 out memory allocated via mmap(2). Balancing these factors |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
130 leads to a default setting of 128*1024 for the |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
131 M_MMAP_THRESHOLD parameter. |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
132 - free() employs sbrk() to release memory back to the system and M_TRIM_THRESHOLD |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
133 specifies the minimum size that is released. So, even without |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
134 malloc_trim, Orthanc is able to give back memory to the system. |
|
f5907aecbaed
conditional usage of malloc_trim
Alain Mazy <am@osimis.io>
parents:
5158
diff
changeset
|
135 - free() never gives back block allocated by mmap() to the system, only malloc_trim() does ! |