Mercurial > hg > orthanc-book
annotate Sphinx/source/faq/scalability.rst @ 445:987fbbc2b59e
leaving bitbucket wrt. bug tracker
| author | Sebastien Jodogne <s.jodogne@gmail.com> |
|---|---|
| date | Mon, 29 Jun 2020 15:40:27 +0200 |
| parents | c2ae93c562d2 |
| children | a10f0e5be459 |
| rev | line source |
|---|---|
| 186 | 1 .. _scalability: |
| 2 | |
| 3 Scalability of Orthanc | |
| 4 ====================== | |
| 5 | |
|
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
6 .. contents:: |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
7 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
8 Overview |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
9 -------- |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
10 |
| 186 | 11 One of the most common question about Orthanc is: *"How many DICOM |
| 12 instances can be stored by Orthanc?"* | |
| 13 | |
| 317 | 14 The source code of Orthanc imposes no such hard limit by itself. |
| 15 | |
| 16 At the time of writing, we know that Orthanc is being used in | |
| 17 production in hospitals with more than 15TB of data, 125,000 studies | |
| 18 and around 50 millions of instances (please `get in touch with us | |
| 188 | 19 <https://www.orthanc-server.com/static.php?page=contact>`__ if you can |
| 317 | 20 share other testimonials). Other users have even reported more than |
| 21 28TB of data. Here are links to some testimonials that were published | |
| 22 on the `Orthanc Users | |
| 23 <https://groups.google.com/forum/#!forum/orthanc-users>`__ discussion | |
| 24 group: `1 | |
| 316 | 25 <https://groups.google.com/d/msg/orthanc-users/-L0D1c2y6rw/KmWnwEijAgAJ>`__, |
| 317 | 26 `2 |
| 27 <https://groups.google.com/d/msg/orthanc-users/-L0D1c2y6rw/nLXxtYzuCQAJ>`__, | |
| 316 | 28 `3 |
| 317 | 29 <https://groups.google.com/d/msg/orthanc-users/s5-XlgA2BEY/ZpYagqBwAAAJ>`__, |
| 316 | 30 `4 |
| 31 <https://groups.google.com/d/msg/orthanc-users/A4hPaJo439s/NwR6zk9FCgAJ>`__, | |
| 317 | 32 `5 |
| 33 <https://groups.google.com/d/msg/orthanc-users/Z5cLwbVgJc0/SxVzxF7ABgAJ>`__, | |
| 316 | 34 `6 |
| 35 <https://groups.google.com/d/msg/orthanc-users/6tGNOqlUk-Q/vppkAYnFAQAJ>`__... | |
| 188 | 36 |
| 37 The stress is actually put on the underlying database engine, and on | |
| 38 the storage area (check out :ref:`orthanc-storage`). As explained in | |
| 39 the :ref:`troubleshooting section <troubleshooting>`, the built-in | |
| 40 SQLite database engine should be replaced by an enterprise-ready | |
| 41 database engine once Orthanc must store several hundreds of thousands | |
| 42 of DICOM instances (check out the :ref:`postgresql` and | |
| 43 :ref:`mysql`). It is also true that the performance of Orthanc in the | |
| 44 presence of large databases has continuously improved over time, | |
| 45 especially when it comes to the speed of :ref:`DICOM C-FIND | |
| 46 <dicom-find>`. | |
| 186 | 47 |
|
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
48 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
49 .. _scalability-setup: |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
50 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
51 Recommended setup for best performance |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
52 -------------------------------------- |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
53 |
| 186 | 54 Here is a generic setup that should provide best performance in the |
| 55 presence of large databases: | |
| 56 | |
| 441 | 57 * Make sure to use the latest release of Orthanc (1.7.1 at the time of |
| 186 | 58 writing). |
| 59 | |
| 60 * We suggest to use the latest release of the :ref:`PostgreSQL plugin | |
| 219 | 61 <postgresql>` to store the database index (3.2 at the time of |
| 186 | 62 writing). Make sure that ``EnableIndex`` is set to ``true``. |
| 63 | |
| 187 | 64 * Make sure that :ref:`run-time debug assertions <troubleshooting>` |
| 65 are turned off. A warning will show in the logs if this is not the | |
| 188 | 66 case. Note that all pre-built binaries provided by Osimis are |
| 67 correctly configured in that respect. | |
| 187 | 68 |
| 186 | 69 * We suggest to use the default filesystem storage area. Of course, |
|
190
5444374c4202
faq: Fix typo: "backup" -> "backed up"
Thibault Nélis <tn@osimis.io>
parents:
189
diff
changeset
|
70 make sure that the filesystem is properly backed up, and that |
| 186 | 71 technologies such as RAID are enabled. Make sure that the option |
| 72 ``EnableStorage`` of the PostgreSQL plugins is set to ``false``. | |
| 73 | |
| 74 * Obviously, the PostgreSQL database should be stored on a high-speed | |
| 75 drive (SSD). This is less important for the storage area. | |
| 76 | |
| 315 | 77 * It may be useful to store the PostgreSQL database on another drive |
| 78 than the storage area. This should improve the use of the available | |
| 79 bandwidth to the disks. | |
| 80 | |
| 186 | 81 * The :ref:`Orthanc configuration file <configuration>` should have |
| 82 the following values for performance-related options (but make sure | |
| 189 | 83 to understand their implications): |
| 186 | 84 |
| 85 * ``StorageCompression = false`` | |
| 86 * ``LimitFindResults = 100`` | |
| 87 * ``LimitFindInstances = 100`` | |
| 88 * ``KeepAlive = true`` | |
| 89 * ``TcpNoDelay = true`` | |
| 90 * ``SaveJobs = false`` | |
| 91 * ``StorageAccessOnFind = Never`` | |
| 92 | |
| 93 * Make sure to carefully :ref:`read the logs <log>` in ``--verbose`` | |
| 94 mode, especially at the startup of Orthanc. The logs may contain | |
| 95 very important information regarding performance. | |
| 96 | |
| 187 | 97 * Make sure to read guides about the `tuning of PostgreSQL |
| 186 | 98 <https://wiki.postgresql.org/wiki/Performance_Optimization>`__. |
| 238 | 99 |
| 100 * You might also be interested in checking the options related to | |
| 101 :ref:`security <security>`. | |
| 309 | 102 |
| 103 * Consider using filesystems that are known to achieve high | |
| 104 performance, such as `XFS <https://en.wikipedia.org/wiki/XFS>`__ or | |
| 105 `Btrfs <https://en.wikipedia.org/wiki/Btrfs>`__ on GNU/Linux | |
| 106 distributions. | |
| 107 | |
| 108 * On GNU/Linux distributions, `LVM (Logical Volume Manager) | |
| 109 <https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)>`__ | |
| 110 can be used to dynamically and easily grow the storage area as more | |
| 111 space becomes needed. | |
|
318
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
112 |
|
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
113 * If using the :ref:`DICOMweb server plugin <dicomweb-server-config>`, |
|
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
114 consider setting configuration option ``StudiesMetadata`` to |
|
83d822f11e78
SeriesMetadata and StudiesMetadata in DICOMweb
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
317
diff
changeset
|
115 ``MainDicomTags``. |
|
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
116 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
117 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
118 .. _scalability-memory: |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
119 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
120 Controlling memory usage |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
121 ------------------------ |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
122 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
123 The absence of memory leaks in Orthanc is verified thanks to `valgrind |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
124 <https://valgrind.org/>`__. |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
125 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
126 On GNU/Linux systems, you might however `observe a large memory |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
127 consumption |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
128 <https://groups.google.com/d/msg/orthanc-users/qWqxpvCPv8g/47wnYyhOCAAJ>`__ |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
129 in the "resident set size" (VmRSS) of the application, notably if you |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
130 upload multiple large DICOM files using the REST API. |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
131 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
132 This large memory consumption comes from the fact that the embedded |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
133 HTTP server is heavily multi-threaded, and that many so-called `memory |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
134 arenas <https://sourceware.org/glibc/wiki/MallocInternals>`__ are |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
135 created by the glibc standard library (up to one per thread). As a |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
136 consequence, if each one of the 50 threads in the HTTP server of |
| 392 | 137 Orthanc (default value of the ``HttpThreadsCount`` option) allocates |
| 138 at some point, say, 50MB, the total memory usage reported as "VmRSS" | |
| 139 can grow up to 50 threads x 50MB = 2.5GB, even if the Orthanc threads | |
|
384
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
140 properly free all the buffers. |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
141 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
142 .. highlight:: bash |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
143 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
144 A possible solution to reducing this memory usage is to ask glibc to |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
145 limit the number of "memory arenas" that are used by the Orthanc |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
146 process. On GNU/Linux, this can be controlled by setting the |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
147 environment variable ``MALLOC_ARENA_MAX``. For instance, the following |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
148 bash command-line would use one single arena that is shared by all the |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
149 threads in Orthanc:: |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
150 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
151 $ MALLOC_ARENA_MAX=1 ./Orthanc |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
152 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
153 Obviously, this restrictive setting will use minimal memory, but will |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
154 result in contention among the threads. A good compromise might be to |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
155 use 5 arenas:: |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
156 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
157 $ MALLOC_ARENA_MAX=5 ./Orthanc |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
158 |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
159 Memory allocation on GNU/Linux is a complex topic. There are other |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
160 options available as environment variables that could also reduce |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
161 memory consumption (for instance, ``MALLOC_MMAP_THRESHOLD_`` would |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
162 bypass arenas for large memory blocks such as DICOM files). Check out |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
163 the `manpage <http://man7.org/linux/man-pages/man3/mallopt.3.html>`__ |
|
e4b0a4d69f42
note about memory usage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
336
diff
changeset
|
164 of ``mallopt()`` for more information. |
| 438 | 165 |
| 166 | |
| 167 .. _scalability-limitations: | |
| 168 | |
| 169 Known limitations | |
| 170 ----------------- | |
| 171 | |
| 172 Exclusive access to the DB | |
| 173 ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 174 | |
| 441 | 175 As of Orthanc 1.7.1, the internal code accessing the DB is still affected |
| 438 | 176 by limitations induced by the SQLite engine that was the only one originally |
| 177 available at the beginning of the project: inside a single Orthanc process, | |
| 440 | 178 there is no concurrent access to the DB. |
| 438 | 179 |
| 180 One solution to avoid this limitation is to have multiple Orthanc accessing | |
| 181 the same DB (works only for MySQL and PostgreSQL) as presented in this `sample | |
| 182 <https://bitbucket.org/osimis/orthanc-setup-samples/src/master/docker/multiple-orthancs-on-same-db/>`__. | |
| 183 | |
| 184 Also note that the core of Orthanc does not currently support the replay | |
| 185 of database transactions, which is necessary to deal with conflicts | |
| 186 between several instances of Orthanc that would simultaneously write | |
| 441 | 187 to the database. As a consequence, as of Orthanc 1.7.1, when connecting multiple |
| 438 | 188 Orthanc to a single database by setting ``Lock`` to ``false``, there |
| 189 should only be one instance of Orthanc acting as a writer and all the | |
| 190 other instances of Orthanc acting as readers only. Be careful to set | |
| 191 the option ``SaveJobs`` to ``false`` in the configuration file of all | |
| 192 the instances of Orthanc acting as readers. | |
| 193 | |
| 194 A refactoring is needed to improve the core of Orthanc in that | |
| 195 respect, for which we are looking for funding from the | |
| 196 industry. Some issues reported in our bug tracker call for this | |
| 197 refactoring: `issue 83 | |
|
445
987fbbc2b59e
leaving bitbucket wrt. bug tracker
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
441
diff
changeset
|
198 <https://bugs.orthanc-server.com/show_bug.cgi?id=83>`__, `issue 121 |
|
987fbbc2b59e
leaving bitbucket wrt. bug tracker
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
441
diff
changeset
|
199 <https://bugs.orthanc-server.com/show_bug.cgi?id=121>`__, `issue 151 |
|
987fbbc2b59e
leaving bitbucket wrt. bug tracker
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
441
diff
changeset
|
200 <https://bugs.orthanc-server.com/show_bug.cgi?id=151>`__. |
| 438 | 201 |
| 440 | 202 |
| 203 Latency | |
| 204 ^^^^^^^ | |
| 205 | |
| 441 | 206 As of Orthanc 1.7.1, Orthanc still performs quite a large number of small |
| 440 | 207 SQL requests. A simple request to a route like ``/studies/{id}`` can trigger |
| 208 6 SQL queries. | |
| 209 | |
| 210 This is not an ideal situation and this might be addressed | |
| 211 in a future larger DB refactoring (the most time-consuming queries have already | |
| 212 been optimized). Given the large number of round-trips | |
| 213 between Orthanc and the DB server, it's important that the latency is reduced | |
| 214 as much as possible. I.e, if deploying Orthanc in a cloud infrastructure, | |
| 215 make sure that the DB server and Orthanc VMs are located in the same datacenter. | |
| 216 | |
| 217 Typically, a latency of 1-4 ms is expected to have correct performances. If your | |
| 218 latency is 20ms, a simple request to ``/studies/{id}`` might spend 120ms in | |
| 219 round-trip alone. | |
| 220 | |
| 221 | |
| 222 | |
| 223 |