Mercurial > hg > orthanc-book
comparison Sphinx/source/faq/scalability.rst @ 643:411e82bb3a9f
documenting revisions and multiple writers
author | Sebastien Jodogne <s.jodogne@gmail.com> |
---|---|
date | Fri, 23 Apr 2021 15:47:14 +0200 |
parents | 75ac001a48ac |
children | fe7b80fe3caf |
comparison
equal
deleted
inserted
replaced
642:a76d83a00c68 | 643:411e82bb3a9f |
---|---|
88 * ``KeepAlive = true`` | 88 * ``KeepAlive = true`` |
89 * ``TcpNoDelay = true`` | 89 * ``TcpNoDelay = true`` |
90 * ``SaveJobs = false`` | 90 * ``SaveJobs = false`` |
91 * ``StorageAccessOnFind = Never`` | 91 * ``StorageAccessOnFind = Never`` |
92 | 92 |
93 * Since Orthanc 1.9.2 and PostgreSQL plugins 4.0: By default, the | |
94 PostgreSQL index plugin uses 1 single connection to the PostgreSQL | |
95 database. You can have multiple connections by setting the | |
96 ``IndexConnectionsCount`` to a higher value (for instance ``5``) in | |
97 the ``PostgreSQL`` section of the configuration file. This will | |
98 improve concurrency. Check out :ref:`the explanation below <multiple-writers>`. | |
99 | |
100 * Since Orthanc 1.9.2 and PostgreSQL plugins 4.0: If you have an | |
101 hospital-wide VNA deployment, you could consider to deploy multiple | |
102 Orthanc servers sharing the same PostgreSQL database. A typical | |
103 scenario is having one "writer" Orthanc server that handles the | |
104 ingesting of DICOM instances, and multiple "reader" Orthanc servers | |
105 with features such as DICOMweb or viewers. | |
106 | |
93 * Make sure to carefully :ref:`read the logs <log>` in ``--verbose`` | 107 * Make sure to carefully :ref:`read the logs <log>` in ``--verbose`` |
94 mode, especially at the startup of Orthanc. The logs may contain | 108 mode, especially at the startup of Orthanc. The logs may contain |
95 very important information regarding performance. | 109 very important information regarding performance. |
96 | 110 |
97 * Make sure to read guides about the `tuning of PostgreSQL | 111 * Make sure to read guides about the `tuning of PostgreSQL |
173 memory consumption (for instance, ``MALLOC_MMAP_THRESHOLD_`` would | 187 memory consumption (for instance, ``MALLOC_MMAP_THRESHOLD_`` would |
174 bypass arenas for large memory blocks such as DICOM files). Check out | 188 bypass arenas for large memory blocks such as DICOM files). Check out |
175 the `manpage <http://man7.org/linux/man-pages/man3/mallopt.3.html>`__ | 189 the `manpage <http://man7.org/linux/man-pages/man3/mallopt.3.html>`__ |
176 of ``mallopt()`` for more information. | 190 of ``mallopt()`` for more information. |
177 | 191 |
178 **Status of the Docker images:** | 192 **Status:** |
193 | |
194 * Since **Orthanc 1.8.2**, the global configuration ``MallocArenaMax`` | |
195 automatically sets ``MALLOC_MMAP_THRESHOLD_`` (defaults to ``5``) | |
196 during the startup of Orthanc. | |
179 | 197 |
180 * The ``jodogne/orthanc`` and ``jodogne/orthanc-plugins`` Docker | 198 * The ``jodogne/orthanc`` and ``jodogne/orthanc-plugins`` Docker |
181 images automatically set ``MALLOC_ARENA_MAX`` to ``5`` **since | 199 images automatically set ``MALLOC_ARENA_MAX`` to ``5`` **since |
182 release 1.6.1** (cf. `changeset | 200 release 1.6.1** (cf. `changeset |
183 <https://github.com/jodogne/OrthancDocker/commit/bd7e9f4665ce8dd6892f82a148cabe8ebcf1c7d9>`__). | 201 <https://github.com/jodogne/OrthancDocker/commit/bd7e9f4665ce8dd6892f82a148cabe8ebcf1c7d9>`__). |
189 .. _scalability-limitations: | 207 .. _scalability-limitations: |
190 | 208 |
191 Known limitations | 209 Known limitations |
192 ----------------- | 210 ----------------- |
193 | 211 |
194 Exclusive access to the DB | 212 Exclusive access to the DB in Orthanc <= 1.9.1 |
195 ^^^^^^^^^^^^^^^^^^^^^^^^^^ | 213 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
196 | 214 |
197 As of Orthanc 1.9.1, the internal code accessing the DB is still affected | 215 Orthanc was originally designed as a mini-DICOM server in 1-to-1 |
198 by limitations induced by the SQLite engine that was the only one originally | 216 relation with a SQLite database. Until **Orthanc 1.9.1**, because of |
199 available at the beginning of the project: inside a single Orthanc process, | 217 this original design, the internal code accessing the DB was affected |
200 there is no concurrent access to the DB. | 218 by a strong limitation: Inside a single Orthanc process, there was no |
201 | 219 concurrent access to the DB. |
202 One solution to avoid this limitation is to have multiple Orthanc accessing | 220 |
203 the same DB (works only for MySQL and PostgreSQL) as presented in this `sample | 221 One solution to avoid this limitation was to have multiple Orthanc |
222 accessing the same DB (works only for MySQL and PostgreSQL) as | |
223 presented in this `sample | |
204 <https://bitbucket.org/osimis/orthanc-setup-samples/src/master/docker/multiple-orthancs-on-same-db/>`__. | 224 <https://bitbucket.org/osimis/orthanc-setup-samples/src/master/docker/multiple-orthancs-on-same-db/>`__. |
205 | 225 However, this solution was only robust if there was **one single |
206 Also note that the core of Orthanc does not currently support the replay | 226 "writer" Orthanc server** (i.e. only one Orthanc was modifying the |
207 of database transactions, which is necessary to deal with conflicts | 227 database). Indeed, the core of Orthanc <= 1.9.1 did not support the |
208 between several instances of Orthanc that would simultaneously write | 228 replay of database transactions, which is necessary to deal with |
209 to the database. As a consequence, as of Orthanc 1.9.1, when connecting multiple | 229 conflicts between several instances of Orthanc that would |
210 Orthanc to a single database by setting ``Lock`` to ``false``, there | 230 simultaneously write to the database. |
211 should only be one instance of Orthanc acting as a writer and all the | 231 |
212 other instances of Orthanc acting as readers only. Be careful to set | 232 Concretely, in Orthanc <= 1.9.1, when connecting multiple Orthanc to a |
213 the option ``SaveJobs`` to ``false`` in the configuration file of all | 233 single database by setting ``Lock`` to ``false``, there should only be |
214 the instances of Orthanc acting as readers. | 234 one instance of Orthanc acting as a writer and all the other instances |
215 | 235 of Orthanc acting as readers only. Be careful to set the option |
216 A refactoring is needed to improve the core of Orthanc in that | 236 ``SaveJobs`` to ``false`` in the configuration file of all the |
217 respect, for which we are looking for funding from the | 237 instances of Orthanc acting as readers (otherwise the readers would |
218 industry. Some issues reported in our bug tracker call for this | 238 also modify the database). |
219 refactoring: `issue 83 | 239 |
220 <https://bugs.orthanc-server.com/show_bug.cgi?id=83>`__, `issue 121 | 240 Some issues reported in our bug tracker are related this limitation: |
221 <https://bugs.orthanc-server.com/show_bug.cgi?id=121>`__, `issue 151 | 241 `issue 83 <https://bugs.orthanc-server.com/show_bug.cgi?id=83>`__, |
222 <https://bugs.orthanc-server.com/show_bug.cgi?id=151>`__. | 242 `issue 121 <https://bugs.orthanc-server.com/show_bug.cgi?id=121>`__, |
243 `issue 151 <https://bugs.orthanc-server.com/show_bug.cgi?id=151>`__. | |
244 | |
245 This limitation has disappeared with Orthanc 1.9.2 and | |
246 PostgreSQL/MySQL plugins 4.0, were the database engine was fully | |
247 rewritten. | |
248 | |
249 | |
250 .. _multiple-writers: | |
251 | |
252 Concurrent accesses to the DB in Orthanc >= 1.9.2 | |
253 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
254 | |
255 In **Orthanc 1.9.2 and PostgreSQL/MySQL plugins 4.0**, the database | |
256 engine of Orthanc was rewritten from scratch to allow multiple | |
257 writers/readers to share the same database. This new feature | |
258 necessitated a full refactoring of the database engine, so as to | |
259 replay transactions in the case of collisions between concurrent | |
260 transactions to the database. | |
261 | |
262 Furthermore, one Orthanc server can also manage several connections to | |
263 PostgreSQL or MySQL, in order to improve performance by adding | |
264 concurrency. Read-only database transactions are also distinguished | |
265 from read-write transactions in order for the database engine to | |
266 further optimize the patterns of access. | |
267 | |
268 Summarizing, the **multiple readers/writers** is now possible. Here | |
269 is a drawing representing a possible deployment: | |
270 | |
271 .. image:: ../images/2021-04-22-MultipleWriters.png | |
272 :align: center | |
273 :width: 500px | |
274 | |
275 Care must be taken to the following aspects: | |
276 | |
277 * Orthanc 1.9.2 must be combined with a database plugin that supports | |
278 multiple writers. This is the case of the PostgreSQL and MySQL | |
279 plugins with version >= 4.0. The built-in SQLite database **does | |
280 not** support multiple writers. | |
281 | |
282 * Concurrent access can result in so-called `non-serializable | |
283 transactions | |
284 <https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable>`__ | |
285 if two separate database transactions modify the database at the | |
286 same time (cf. ``ErrorCode_DatabaseCannotSerialize`` in the source | |
287 code of Orthanc). Orthanc will **automatically replay such | |
288 transactions** a certain number of times (waiting 100ms more between | |
289 each retry), until the transactions succeed. The plugins provide an | |
290 option to control the maximum number of retries. If the maximum | |
291 number of retries is exceeded, the ``503 Service Unavailable`` HTTP | |
292 error is raised (server overloaded because of unsuccessful retries | |
293 of concurrent transactions). | |
294 | |
295 * If a higher-level application **modifies metadata and/or | |
296 attachments** in the presence of multiple writers, Orthanc provides | |
297 a :ref:`revision mechanism <revisions>` to prevent concurrent | |
298 updates. | |
299 | |
223 | 300 |
224 | 301 |
225 Latency | 302 Latency |
226 ^^^^^^^ | 303 ^^^^^^^ |
227 | 304 |
228 As of Orthanc 1.9.1, Orthanc still performs quite a large number of small | 305 For some queries to the database, Orthanc performs several small SQL |
229 SQL requests. A simple request to a route like ``/studies/{id}`` can trigger | 306 requests. For instance, a request to a route like ``/studies/{id}`` |
230 6 SQL queries. | 307 can trigger 6 SQL queries. Given these round-trips between Orthanc and |
231 | 308 the DB server, it's important for the **network latency to be as small |
232 This is not an ideal situation and this might be addressed | 309 as possible**. For instance, if your latency is 20ms, a single request |
233 in a future larger DB refactoring (the most time-consuming queries have already | 310 to ``/studies/{id}`` might take 120ms. Typically, a latency of 1-4 ms |
234 been optimized). Given the large number of round-trips | 311 is expected to have correct performances. |
235 between Orthanc and the DB server, it's important that the latency is reduced | 312 |
236 as much as possible. I.e, if deploying Orthanc in a cloud infrastructure, | 313 As a consequence, if deploying Orthanc in a cloud infrastructure, make |
237 make sure that the DB server and Orthanc VMs are located in the same datacenter. | 314 sure that the DB server and Orthanc VMs are located in the **same |
238 | 315 datacenter**. Note that most of the time-consuming queries have |
239 Typically, a latency of 1-4 ms is expected to have correct performances. If your | 316 already been optimized, and that future versions of Orthanc SDK might |
240 latency is 20ms, a simple request to ``/studies/{id}`` might spend 120ms in | 317 aggregate even more SQL requests. |
241 round-trip alone. | 318 |
242 | 319 Starting with Orthanc 1.9.2, and PostgreSQL/MySQL index plugins 4.0, |
243 | 320 Orthanc can also be configured to handle **multiple connections to the |
244 | 321 database server** by setting the ``IndexConnectionsCount`` to a value |
245 | 322 greater than ``1``. This allows concurrent accesses to the database, |
323 which avoids to sequentially wait for a database transaction to be | |
324 concluded before starting another one. Having multiple connections | |
325 makes the latency problem much less important. | |
326 | |
327 | |
328 Slow deletions | |
329 ^^^^^^^^^^^^^^ | |
330 | |
331 Deleting large studies can take some time, because removing a large | |
332 number of files from a filesystem can be an expensive operation (which | |
333 might sound counterintuitive). | |
334 | |
335 It is possible to create an :ref:`storage area plugin | |
336 <creating-plugins>` that delays the actual deletion from the | |
337 filesystem. The plugin would maintain a queue (e.g. as a SQLite | |
338 database) of files to be removed. The actual deletion from the | |
339 filesystem would be done asynchronously in a separate thread. | |
340 | |
341 We are looking for funding from the industry to implement such a | |
342 plugin. |