Mercurial > hg > orthanc-book
annotate Sphinx/source/faq/scalability.rst @ 490:933e28b6a64d
Orthanc 1.7.3
author | Sebastien Jodogne <s.jodogne@gmail.com> |
---|---|
date | Mon, 24 Aug 2020 08:05:10 +0200 |
parents | a10f0e5be459 |
children | f22b3743fd3f |
rev | line source |
---|---|
186 | 1 .. _scalability: |
2 | |
3 Scalability of Orthanc | |
4 ====================== | |
5 | |
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
6 .. contents:: |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
7 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
8 Overview |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
9 -------- |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
10 |
186 | 11 One of the most common question about Orthanc is: *"How many DICOM |
12 instances can be stored by Orthanc?"* | |
13 | |
317 | 14 The source code of Orthanc imposes no such hard limit by itself. |
15 | |
16 At the time of writing, we know that Orthanc is being used in | |
17 production in hospitals with more than 15TB of data, 125,000 studies | |
18 and around 50 millions of instances (please `get in touch with us | |
188 | 19 <https://www.orthanc-server.com/static.php?page=contact>`__ if you can |
317 | 20 share other testimonials). Other users have even reported more than |
21 28TB of data. Here are links to some testimonials that were published | |
22 on the `Orthanc Users | |
23 <https://groups.google.com/forum/#!forum/orthanc-users>`__ discussion | |
24 group: `1 | |
316 | 25 <https://groups.google.com/d/msg/orthanc-users/-L0D1c2y6rw/KmWnwEijAgAJ>`__, |
317 | 26 `2 |
27 <https://groups.google.com/d/msg/orthanc-users/-L0D1c2y6rw/nLXxtYzuCQAJ>`__, | |
316 | 28 `3 |
317 | 29 <https://groups.google.com/d/msg/orthanc-users/s5-XlgA2BEY/ZpYagqBwAAAJ>`__, |
316 | 30 `4 |
31 <https://groups.google.com/d/msg/orthanc-users/A4hPaJo439s/NwR6zk9FCgAJ>`__, | |
317 | 32 `5 |
33 <https://groups.google.com/d/msg/orthanc-users/Z5cLwbVgJc0/SxVzxF7ABgAJ>`__, | |
316 | 34 `6 |
35 <https://groups.google.com/d/msg/orthanc-users/6tGNOqlUk-Q/vppkAYnFAQAJ>`__... | |
188 | 36 |
37 The stress is actually put on the underlying database engine, and on | |
38 the storage area (check out :ref:`orthanc-storage`). As explained in | |
39 the :ref:`troubleshooting section <troubleshooting>`, the built-in | |
40 SQLite database engine should be replaced by an enterprise-ready | |
41 database engine once Orthanc must store several hundreds of thousands | |
42 of DICOM instances (check out the :ref:`postgresql` and | |
43 :ref:`mysql`). It is also true that the performance of Orthanc in the | |
44 presence of large databases has continuously improved over time, | |
45 especially when it comes to the speed of :ref:`DICOM C-FIND | |
46 <dicom-find>`. | |
186 | 47 |
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
48 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
49 .. _scalability-setup: |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
50 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
51 Recommended setup for best performance |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
52 -------------------------------------- |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
53 |
186 | 54 Here is a generic setup that should provide best performance in the |
55 presence of large databases: | |
56 | |
490 | 57 * Make sure to use the latest release of Orthanc (1.7.3 at the time of |
186 | 58 writing). |
59 | |
60 * We suggest to use the latest release of the :ref:`PostgreSQL plugin | |
219 | 61 <postgresql>` to store the database index (3.2 at the time of |
186 | 62 writing). Make sure that ``EnableIndex`` is set to ``true``. |
63 | |
187 | 64 * Make sure that :ref:`run-time debug assertions <troubleshooting>` |
65 are turned off. A warning will show in the logs if this is not the | |
188 | 66 case. Note that all pre-built binaries provided by Osimis are |
67 correctly configured in that respect. | |
187 | 68 |
186 | 69 * We suggest to use the default filesystem storage area. Of course, |
190
5444374c4202
faq: Fix typo: "backup" -> "backed up"
Thibault Nélis <tn@osimis.io>
parents:
189
diff
changeset
|
70 make sure that the filesystem is properly backed up, and that |
186 | 71 technologies such as RAID are enabled. Make sure that the option |
72 ``EnableStorage`` of the PostgreSQL plugins is set to ``false``. | |
73 | |
74 * Obviously, the PostgreSQL database should be stored on a high-speed | |
75 drive (SSD). This is less important for the storage area. | |
76 | |
315 | 77 * It may be useful to store the PostgreSQL database on another drive |
78 than the storage area. This should improve the use of the available | |
79 bandwidth to the disks. | |
80 | |
186 | 81 * The :ref:`Orthanc configuration file <configuration>` should have |
82 the following values for performance-related options (but make sure | |
189 | 83 to understand their implications): |
186 | 84 |
85 * ``StorageCompression = false`` | |
86 * ``LimitFindResults = 100`` | |
87 * ``LimitFindInstances = 100`` | |
88 * ``KeepAlive = true`` | |
89 * ``TcpNoDelay = true`` | |
90 * ``SaveJobs = false`` | |
91 * ``StorageAccessOnFind = Never`` | |
92 | |
93 * Make sure to carefully :ref:`read the logs <log>` in ``--verbose`` | |
94 mode, especially at the startup of Orthanc. The logs may contain | |
95 very important information regarding performance. | |
96 | |
187 | 97 * Make sure to read guides about the `tuning of PostgreSQL |
186 | 98 <https://wiki.postgresql.org/wiki/Performance_Optimization>`__. |
238 | 99 |
100 * You might also be interested in checking the options related to | |
101 :ref:`security <security>`. | |
309 | 102 |
103 * Consider using filesystems that are known to achieve high | |
104 performance, such as `XFS <https://en.wikipedia.org/wiki/XFS>`__ or | |
105 `Btrfs <https://en.wikipedia.org/wiki/Btrfs>`__ on GNU/Linux | |
106 distributions. | |
107 | |
108 * On GNU/Linux distributions, `LVM (Logical Volume Manager) | |
109 <https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)>`__ | |
110 can be used to dynamically and easily grow the storage area as more | |
111 space becomes needed. | |
318
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
112 |
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
113 * If using the :ref:`DICOMweb server plugin <dicomweb-server-config>`, |
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
114 consider setting configuration option ``StudiesMetadata`` to |
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
115 ``MainDicomTags``. |
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
116 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
117 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
118 .. _scalability-memory: |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
119 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
120 Controlling memory usage |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
121 ------------------------ |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
122 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
123 The absence of memory leaks in Orthanc is verified thanks to `valgrind |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
124 <https://valgrind.org/>`__. |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
125 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
126 On GNU/Linux systems, you might however `observe a large memory |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
127 consumption |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
128 <https://groups.google.com/d/msg/orthanc-users/qWqxpvCPv8g/47wnYyhOCAAJ>`__ |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
129 in the "resident set size" (VmRSS) of the application, notably if you |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
130 upload multiple large DICOM files using the REST API. |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
131 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
132 This large memory consumption comes from the fact that the embedded |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
133 HTTP server is heavily multi-threaded, and that many so-called `memory |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
134 arenas <https://sourceware.org/glibc/wiki/MallocInternals>`__ are |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
135 created by the glibc standard library (up to one per thread). As a |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
136 consequence, if each one of the 50 threads in the HTTP server of |
392 | 137 Orthanc (default value of the ``HttpThreadsCount`` option) allocates |
138 at some point, say, 50MB, the total memory usage reported as "VmRSS" | |
139 can grow up to 50 threads x 50MB = 2.5GB, even if the Orthanc threads | |
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
140 properly free all the buffers. |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
141 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
142 .. highlight:: bash |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
143 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
144 A possible solution to reducing this memory usage is to ask glibc to |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
145 limit the number of "memory arenas" that are used by the Orthanc |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
146 process. On GNU/Linux, this can be controlled by setting the |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
147 environment variable ``MALLOC_ARENA_MAX``. For instance, the following |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
148 bash command-line would use one single arena that is shared by all the |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
149 threads in Orthanc:: |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
150 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
151 $ MALLOC_ARENA_MAX=1 ./Orthanc |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
152 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
153 Obviously, this restrictive setting will use minimal memory, but will |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
154 result in contention among the threads. A good compromise might be to |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
155 use 5 arenas:: |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
156 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
157 $ MALLOC_ARENA_MAX=5 ./Orthanc |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
158 |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
159 Memory allocation on GNU/Linux is a complex topic. There are other |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
160 options available as environment variables that could also reduce |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
161 memory consumption (for instance, ``MALLOC_MMAP_THRESHOLD_`` would |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
162 bypass arenas for large memory blocks such as DICOM files). Check out |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
163 the `manpage <http://man7.org/linux/man-pages/man3/mallopt.3.html>`__ |
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
164 of ``mallopt()`` for more information. |
438 | 165 |
166 | |
167 .. _scalability-limitations: | |
168 | |
169 Known limitations | |
170 ----------------- | |
171 | |
172 Exclusive access to the DB | |
173 ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
174 | |
490 | 175 As of Orthanc 1.7.3, the internal code accessing the DB is still affected |
438 | 176 by limitations induced by the SQLite engine that was the only one originally |
177 available at the beginning of the project: inside a single Orthanc process, | |
440 | 178 there is no concurrent access to the DB. |
438 | 179 |
180 One solution to avoid this limitation is to have multiple Orthanc accessing | |
181 the same DB (works only for MySQL and PostgreSQL) as presented in this `sample | |
182 <https://bitbucket.org/osimis/orthanc-setup-samples/src/master/docker/multiple-orthancs-on-same-db/>`__. | |
183 | |
184 Also note that the core of Orthanc does not currently support the replay | |
185 of database transactions, which is necessary to deal with conflicts | |
186 between several instances of Orthanc that would simultaneously write | |
490 | 187 to the database. As a consequence, as of Orthanc 1.7.3, when connecting multiple |
438 | 188 Orthanc to a single database by setting ``Lock`` to ``false``, there |
189 should only be one instance of Orthanc acting as a writer and all the | |
190 other instances of Orthanc acting as readers only. Be careful to set | |
191 the option ``SaveJobs`` to ``false`` in the configuration file of all | |
192 the instances of Orthanc acting as readers. | |
193 | |
194 A refactoring is needed to improve the core of Orthanc in that | |
195 respect, for which we are looking for funding from the | |
196 industry. Some issues reported in our bug tracker call for this | |
197 refactoring: `issue 83 | |
445
987fbbc2b59e
leaving bitbucket wrt. bug tracker
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
441
diff
changeset
|
198 <https://bugs.orthanc-server.com/show_bug.cgi?id=83>`__, `issue 121 |
987fbbc2b59e
leaving bitbucket wrt. bug tracker
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
441
diff
changeset
|
199 <https://bugs.orthanc-server.com/show_bug.cgi?id=121>`__, `issue 151 |
987fbbc2b59e
leaving bitbucket wrt. bug tracker
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
441
diff
changeset
|
200 <https://bugs.orthanc-server.com/show_bug.cgi?id=151>`__. |
438 | 201 |
440 | 202 |
203 Latency | |
204 ^^^^^^^ | |
205 | |
490 | 206 As of Orthanc 1.7.3, Orthanc still performs quite a large number of small |
440 | 207 SQL requests. A simple request to a route like ``/studies/{id}`` can trigger |
208 6 SQL queries. | |
209 | |
210 This is not an ideal situation and this might be addressed | |
211 in a future larger DB refactoring (the most time-consuming queries have already | |
212 been optimized). Given the large number of round-trips | |
213 between Orthanc and the DB server, it's important that the latency is reduced | |
214 as much as possible. I.e, if deploying Orthanc in a cloud infrastructure, | |
215 make sure that the DB server and Orthanc VMs are located in the same datacenter. | |
216 | |
217 Typically, a latency of 1-4 ms is expected to have correct performances. If your | |
218 latency is 20ms, a simple request to ``/studies/{id}`` might spend 120ms in | |
219 round-trip alone. | |
220 | |
221 | |
222 | |
223 |