Scalability of Orthanc

Overview

One of the most common question about Orthanc is: “How many DICOM instances can be stored by Orthanc?”

The source code of Orthanc imposes no such hard limit by itself.

At the time of writing, we know that Orthanc is being used in production in hospitals with more than 65TB of data, 340,000 studies and around 150 millions of instances (please get in touch with us if you can share other testimonials).

The stress is actually put on the underlying database engine, and on the storage area (check out How does Orthanc store its database?). As explained in the troubleshooting section, the built-in SQLite database engine should be replaced by an enterprise-ready database engine once Orthanc must store several hundreds of thousands of DICOM instances (check out the PostgreSQL plugins and MySQL/MariaDB plugins). It is also true that the performance of Orthanc in the presence of large databases has continuously improved over time, especially when it comes to the speed of DICOM C-FIND.

Controlling the threads

Orthanc uses multiple threads to perform operations in parallel. Depending on your infrastructure and usage, you can fine-tune many configurations options related to multithreading:

Option

Purpose

Default

Related metrics

Recommended adjustments

HttpThreadsCount

Threads for handling incoming requests to the REST API.

50

orthanc_available_http_threads_count

Increase when the number of available threads is too low. An alternative adjustment is to spin another Orthanc on another VM, connected to the same PostgreSQL database to spread the workload.

DicomThreadsCount

Threads for handling incoming DICOM SCP connections.

4

orthanc_available_dicom_threads

Increase when the number of available threads is too low, typically if Orthanc must handle many C-STORE and C-FIND in parallel. For instance: If Orthanc is connected to 100 DICOM modalities, statistically, there might be 10 modalities that will try to communicate with Orthanc at any given time, so you should increase this option to 10.

ConcurrentJobs

Maximum number of processing jobs that are simultaneously running at any given time.

2

orthanc_jobs_running orthanc_jobs_pending

Increase when the number of pending jobs is too high. Note that each job may itself use multiple threads to execute.

JobsEngineThreadsCount, ResourceModification

Number of threads that are used to perform a resource modification or anonymization.

1

Increase to 8-16 if you are using a distributed object storage as the Orthanc storage area.

StorageLoaderThreadsCount

Number of threads that are used to read files from the storage area.

4

orthanc_storage_available_threads

Increase when the number of available threads is too low. Typically, if you are using a distributed object storage for the Orthanc storage area, increase to, say, 20.

DicomParserThreadsCount

Number of threads that are used to parse DICOM files and access DICOM tags that are not indexed in the database.

2

orthanc_dicom_parser_available_threads

Increase when the number of available threads is too low.

TranscoderThreadsCount

Number of threads that are used to transcode DICOM files.

4

orthanc_transcoder_available_threads

Increase when the number of available threads is too low.

SequentialDicomReaderThreadsCount

Number of threads used for the sequential access to DICOM instances (for archive jobs, C-STORE SCU, C-GET SCP, and C-MOVE SCP).

4

orthanc_sequential_reader_available_threads

Increase when the number of available threads is too low. If not defined, the default value is identical to StorageLoaderThreadsCount, so you should probably never adjust this value.

Pay attention that all these threads might consume a lot of memory and you might need to limit the memory usage, so check out the next section.

Controlling memory usage

Starting with Orthanc 1.13.0, you can fine-tune many configuration options related to the memory consumption. However, you should always keep in mind that, when handling large DICOM files, Orthanc needs to store the full file in RAM, possibly multiple times during transcoding or compression. As a consequence, even if you set limits on memory usage, Orthanc may punctually exceed these limits, which can result in an out-of-memory (OOM) error. For instance, if, at some point, Orthanc must handle a 3GB DICOM file while there is only 2GB RAM on the system, the system would crash.

Option

Purpose

Default

Related metrics

Recommended adjustments

MaximumStorageCacheSize

Maximum size of the storage cache (in MB). The storage cache is stored in RAM and contains the content of the recently accessed attachments from the storage area, which notably include the DICOM files.

128

orthanc_storage_cache_miss_count orthanc_storage_cache_hit_count orthanc_storage_cache_count orthanc_storage_cache_size_mb

This cache is mainly useful when you use the received DICOM data immediately after receiving it, or when REST API clients access the same DICOM instances across multiple calls. For instance, when you are viewing data directly after acquisition, if you are using Orthanc as a router, or if multiple users access the same study in a viewer. In such situations, it might be interesting to have a large cache size (possibly of multiple GB) to avoid reading the same data multiple times from the disk or from the distributed filesystem.

DicomParserCacheSize

Maximum size of the cache of parsed DICOM files (in MB). This cache is stored in RAM and contains a parsed version of the most recently accessed DICOM instances.

256

orthanc_dicom_parser_cache_miss_count orthanc_dicom_parser_cache_hit_count orthanc_dicom_parser_cache_count orthanc_dicom_parser_cache_size_mb

This cache is relevant when, e.g., a viewer or the REST API successively request different DICOM tags from the same DICOM instance. However, it is expected that the cache miss is very high and you should probably not worry about it.

TranscoderCacheSize

Maximum size of the cache of transcoded DICOM files (in MB). This cache is stored in RAM and contains a parsed version of the most recently transcoded DICOM instances.

256

orthanc_transcoder_cache_miss_count orthanc_transcoder_cache_hit_count orthanc_transcoder_cache_count orthanc_transcoder_cache_size_mb

This cache is relevant when, e.g., a viewer or the REST API request the same transcoded DICOM instance multiple times. This cache is useful because trancsoding a DICOM instance can be a highly costly operation. However, it is expected that the cache miss is very high and you should probably not worry about it.

StorageMemoryCapacity

Peak amount of RAM (in MB) that can be allocated by the threads loading from the storage area. Note that this limit can be exceeded when a single file is larger than this option.

512

orthanc_storage_memory_usage_mb orthanc_storage_memory_max_usage_mb orthanc_storage_memory_count orthanc_storage_memory_capacity_mb

Monitor orthanc_storage_memory_usage_mb. If it is regularly close to the configured capacity, this means that some storage loader threads might have to wait until more RAM is available, which can degrade the performance.

DicomParserMemoryCapacity

Peak amount of RAM (in MB) that can be allocated by the threads that parse the DICOM files. Note that this limit can be exceeded when a single DICOM instance is larger than this option.

256

orthanc_dicom_parser_memory_usage_mb orthanc_dicom_parser_memory_max_usage_mb orthanc_dicom_parser_memory_count orthanc_dicom_parser_memory_capacity_mb

Monitor orthanc_dicom_parser_memory_usage_mb. If it is regularly close to the configured capacity, this means that some DICOM parser threads might have to wait until more RAM is available, which can degrade the performance.

TranscoderMemoryCapacity

Peak amount of RAM (in MB) that can be allocated by the threads that transcode the DICOM files. Note that this limit can be exceeded when a single transcoded DICOM instance is larger than this option.

256

orthanc_transcoder_memory_usage_mb orthanc_transcoder_memory_max_usage_mb orthanc_transcoder_memory_count orthanc_transcoder_memory_capacity_mb

Monitor orthanc_transcoder_memory_usage_mb. If it is regularly close to the configured capacity, this means that some transcoder threads might have to wait until more RAM is available, which can degrade the performance.

SequentialDicomReaderWindowCapacity

Maximum amount of RAM (in MB) allocated to each local sliding-window buffer for each thread accessing a set of DICOM instances in sequential order. See also the SequentialDicomReaderWindowSize configuration. Note that this limit can be exceeded when a single file is larger than this option.

128

There is currently no way to monitor these memory consumptions.

All the caches will progressively fill to their full capacity as soon as they are used, so you should always expect them to consume their allocated amount of RAM.

For the memory capacities, RAM is allocated only when a thread requires memory from the corresponding memory area, and is released as soon as the thread has finished its job with this particular data. If, at some point, a thread requires X MB of RAM and this amount of RAM is currently not available, the thread must wait until other threads have freed X MB.

For instance, if you have configured StorageMemoryCapacity to 512 MB and two threads have each reserved 200 MB to process a 200 MB file, a third thread that needs to reserve 200 MB for another file will have to wait until one of the first two threads has finished its work and at least 200 MB of memory is available in the corresponding memory capacity.

Conversely, if you have configured StorageMemoryCapacity to 512 MB and two threads have each reserved 200 MB to process a 200 MB file, a third thread that needs to reserve 600 MB for another file will have to wait until both threads have finished their work. Only then will the third thread be allocated 600 MB, punctually exceeding the declared maximum capacity. The maximum usage of this memory space will be recorded in the orthanc_*_memory_max_usage_mb metrics.

Note: Orthanc also uses RAM that is not part of the memory areas controlled by these memory capacities. It is difficult to estimate the amount of RAM used outside these controlled areas, as it depends on your usage patterns, but you should allow for an additional 128 MB of RAM. Also note that plugins can also consume RAM that is not taken into account either.

How can I compute the total amount of memory Orthanc will consume?

In this section, we will consider 2 scenarios:

  • In the first one, your Orthanc server only handles small DICOM instances that are less than 10-20 MB (MRI, CT, CR, DX, standard US,…).

  • In the second one, Orthanc also handles large DICOM instances between 300 MB and 4 GB (videos, cardiac US, whole-slide images,…).

For both scenario, consider we have these configurations:

{
  "ConcurrentJobs": 8,

  "StorageLoaderThreadsCount": 20,
  "StorageMemoryCapacity": 512,
  "MaximumStorageCacheSize": 128,

  "SequentialDicomReaderThreadsCount": 20,
  "SequentialDicomReaderWindowCapacity": 128,

  "DicomParserThreadsCount": 2,
  "DicomParserCacheSize": 256,
  "DicomParserMemoryCapacity": 256,

  "TranscoderThreadsCount": 4,
  "TranscoderCacheSize": 256,
  "TranscoderMemoryCapacity": 256
}

In the first scenario, where Orthanc handles small instances, Orthanc should never exceed the *MemoryCapacity configurations:

MaximumStorageCacheSize:                          128 MB
DicomParserCacheSize:                             256 MB
TranscoderCacheSize:                              256 MB
------------------------------------------------------------------
TOTAL CACHES                                                640 MB

StorageMemoryCapacity:                            512 MB
DicomParserMemoryCapacity:                        256 MB
TranscoderMemoryCapacity:                         256 MB
SequentialDicomReaderWindowCapacity
* SequentialDicomReaderThreadsCount:  128*20 =   2560 MB
------------------------------------------------------------------
TOTAL MEMORY CAPACITIES                                    3584 MB

EXTRA MARGIN FOR UNMANAGED MEMORY                           128 MB

==================================================================
TOTAL                                                      4352 MB

In the second scenario, let’s consider the worst case scenario where, at a given time, each active thread is handling a file that is 600 MB, therefore, exceeding the *MemoryCapacity configurations:

MaximumStorageCacheSize:                          128 MB
DicomParserCacheSize:                             256 MB
TranscoderCacheSize:                              256 MB
------------------------------------------------------------------
TOTAL CACHES                                                640 MB

StorageMemoryCapacity (overpassed):               600 MB
DicomParserMemoryCapacity (overpassed):           600 MB
TranscoderMemoryCapacity (overpassed):            600 MB
SequentialDicomReaderWindowCapacity (overpassed)
* SequentialDicomReaderThreadsCount:  600*20 =  12000 MB
------------------------------------------------------------------
TOTAL MEMORY CAPACITIES                                   13800 MB

EXTRA MARGIN FOR UNMANAGED MEMORY                           128 MB

==================================================================
TOTAL                                                     14568 MB

Running Orthanc on a system with limited RAM

If you are running Orthanc on a system with very limited memory, you should configure Orthanc accordingly. You should probably leave 64 MB to the OS, disable caches in Orthanc, reduce the number of SequentialDicomReaderThreadsCount, reduce all MemoryCapacities, and reduce the number of HTTP and DICOM threads. Here is an example:

MaximumStorageCacheSize:                            0 MB
DicomParserCacheSize:                               0 MB
TranscoderCacheSize:                                0 MB
------------------------------------------------------------------
TOTAL CACHES                                                  0 MB

StorageMemoryCapacity:                             32 MB
DicomParserMemoryCapacity:                         32 MB
TranscoderMemoryCapacity:                          32 MB
SequentialDicomReaderWindowCapacity
* SequentialDicomReaderThreadsCount:  16*2   =     32 MB
------------------------------------------------------------------
TOTAL MEMORY CAPACITIES                                     128 MB

EXTRA MARGIN FOR UNMANAGED MEMORY                           128 MB

==================================================================
TOTAL                                                       256 MB

For this scenario, your configuration would look like:

{
  "ConcurrentJobs": 2,
  "HttpThreadsCount": 8,
  "DicomThreadsCount": 2,

  "StorageLoaderThreadsCount": 2,
  "StorageMemoryCapacity": 32,
  "MaximumStorageCacheSize": 0,

  "SequentialDicomReaderThreadsCount": 2,
  "SequentialDicomReaderWindowCapacity": 16,

  "DicomParserThreadsCount": 2,
  "DicomParserCacheSize": 0,
  "DicomParserMemoryCapacity": 32,

  "TranscoderThreadsCount": 2,
  "TranscoderCacheSize": 0,
  "TranscoderMemoryCapacity": 32
}

Observed memory consumption

The absence of memory leaks in Orthanc is verified thanks to valgrind.

Note: It is not yet clear if the below section is still applicable to Orthanc 1.13.0 since we have introduced new memory handling configurations.

On GNU/Linux systems, you might however observe a large memory consumption in the “resident set size” (VmRSS) of the application, notably if you upload multiple large DICOM files using the REST API.

This large memory consumption comes from the fact that the embedded HTTP server is heavily multi-threaded, and that many so-called memory arenas are created by the glibc standard library (up to one per thread). As a consequence, if each one of the 50 threads in the HTTP server of Orthanc (default value of the HttpThreadsCount option) allocates at some point, say, 50MB, the total memory usage reported as “VmRSS” can grow up to 50 threads x 50MB = 2.5GB, even if the Orthanc threads properly free all the buffers.

A possible solution to reducing this memory usage is to ask glibc to limit the number of “memory arenas” that are used by the Orthanc process. On GNU/Linux, this can be controlled by setting the environment variable MALLOC_ARENA_MAX. For instance, the following bash command-line would use one single arena that is shared by all the threads in Orthanc:

$ MALLOC_ARENA_MAX=1 ./Orthanc

Obviously, this restrictive setting will use minimal memory, but will result in contention among the threads. A good compromise might be to use 5 arenas:

$ MALLOC_ARENA_MAX=5 ./Orthanc

Memory allocation on GNU/Linux is a complex topic. There are other options available as environment variables that could also reduce memory consumption (for instance, MALLOC_MMAP_THRESHOLD_ would bypass arenas for large memory blocks such as DICOM files). Check out the manpage of mallopt() for more information.

Status:

  • Since Orthanc 1.8.2, the global configuration MallocArenaMax automatically sets MALLOC_MMAP_THRESHOLD_ (defaults to 5) during the startup of Orthanc.

  • The jodogne/orthanc and jodogne/orthanc-plugins Docker images automatically set MALLOC_ARENA_MAX to 5 since release 1.6.1 (cf. changeset).

  • The orthancteam/orthanc images automatically set MALLOC_ARENA_MAX to 5 since release 20.12.2.

Known limitations

Exclusive access to the DB in Orthanc <= 1.9.1

Orthanc was originally designed as a mini-DICOM server in 1-to-1 relation with a SQLite database. Until Orthanc 1.9.1, because of this original design, the internal code accessing the DB was affected by a strong limitation: Inside a single Orthanc process, there was no concurrent access to the DB.

One solution to avoid this limitation was to have multiple Orthanc accessing the same DB (works only for MySQL and PostgreSQL) as presented in this sample. However, this solution was only robust if there was one single “writer” Orthanc server (i.e. only one Orthanc was modifying the database). Indeed, the core of Orthanc <= 1.9.1 did not support the replay of database transactions, which is necessary to deal with conflicts between several instances of Orthanc that would simultaneously write to the database.

Concretely, in Orthanc <= 1.9.1, when connecting multiple Orthanc to a single database by setting Lock to false, there should only be one instance of Orthanc acting as a writer and all the other instances of Orthanc acting as readers only. Be careful to set the option SaveJobs to false in the configuration file of all the instances of Orthanc acting as readers (otherwise the readers would also modify the database).

Some issues reported in our bug tracker are related this limitation: issue 83, issue 121, issue 151.

This limitation has disappeared with Orthanc 1.9.2 and PostgreSQL/MySQL plugins 4.0, were the database engine was fully rewritten.

Concurrent accesses to the DB in Orthanc >= 1.9.2

In Orthanc 1.9.2 and PostgreSQL/MySQL plugins 4.0, the database engine of Orthanc was rewritten from scratch to allow multiple writers/readers to share the same database. This new feature necessitated a full refactoring of the database engine, so as to replay transactions in the case of collisions between concurrent transactions to the database.

Furthermore, one Orthanc server can also manage several connections to PostgreSQL or MySQL, in order to improve performance by adding concurrency. Read-only database transactions are also distinguished from read-write transactions in order for the database engine to further optimize the patterns of access.

Summarizing, the multiple readers/writers is now possible. Here is a drawing representing a possible deployment with 4 Orthanc servers, all sharing the same DICOM images, with some servers handling multiple connections to a PostgreSQL database for higher throughput:

../_images/2021-04-22-MultipleWriters.png

Care must be taken to the following aspects:

  • Orthanc 1.9.2 must be combined with a database plugin that supports multiple writers. This is the case of the PostgreSQL and MySQL plugins with version >= 4.0. The built-in SQLite database does not support multiple writers.

  • Concurrent access can result in so-called non-serializable transactions if two separate database transactions modify the database at the same time (cf. ErrorCode_DatabaseCannotSerialize in the source code of Orthanc). Orthanc will automatically replay such transactions a certain number of times (waiting 100ms more between each retry), until the transactions succeed. The plugins provide an option to control the maximum number of retries. If the maximum number of retries is exceeded, the 503 Service Unavailable HTTP error is raised (server overloaded because of unsuccessful retries of concurrent transactions). Note that, since version 6.0 of the PostgreSQL plugin, it is now possible to configure the TransactionMode to ReadCommitted instead of the default Serializable mode to avoid most of the transactions collisions. This option is not (yet) available for the MySQL plugin.

  • If a higher-level application modifies metadata and/or attachments in the presence of multiple writers, Orthanc provides a revision mechanism to prevent concurrent updates.

  • Thanks to this support of concurrent accesses, it is possible to put a load balancer on the top of the REST API of Orthanc. All the DICOM resources (patients, studies, series and instances) are indeed shared by all the instances of Orthanc connected to the same underlying database. As an application, this might be of great help if multiple viewers must connect to Orthanc. In Kubernetes, concurrent accesses also make it possible to manage a set of replicas of Orthanc (e.g. as deployment).

    There are however some caveats if using a load balancer or Kubernetes replicas, notably:

    • Each Orthanc instance maintains its own list of jobs. Therefore, the /jobs route will return only the jobs of the responding Orthanc.

    • Similarly, each Orthanc instance maintains its own status for the resources it has received. Thus, the IsStable information is local to each Orthanc instance.

    • The /modalities or the /peers are also private to each instance of Orthanc in the cluster, as soon as the respective options DicomModalitiesInDatabase and OrthancPeersInDatabase are set to true.

    If you need to use such primitives in your application, you have three possibilities: (1) Introduce a distinguished Orthanc server that is responsible to take care of all the jobs (including modalities and peers) and/or to receive all the DICOM instances, (2) create an Orthanc plugin (e.g. using Python or Java) that queries all the Orthanc in the cluster and that aggregates all of their answers, or (3) do the same using a higher-level framework (such as Node.js).

Latency

Up to v 6.2, for some queries to the database, Orthanc performs several small SQL requests. For instance, a request to a route like /studies/{id} can trigger 6 SQL queries. Given these round-trips between Orthanc and the DB server, it’s important for the network latency to be as small as possible. For instance, if your latency is 20ms, a single request to /studies/{id} might take 120ms. Typically, a latency of 1-4 ms is expected to have correct performances.

As a consequence, if deploying Orthanc in a cloud infrastructure, make sure that the DB server and Orthanc VMs are located in the same datacenter. Note that most of the time-consuming queries have already been optimized in v 6.0 and a huge improvement has been implemented in v 7.0.

Starting with Orthanc 1.9.2, and PostgreSQL/MySQL index plugins 4.0, Orthanc can also be configured to handle multiple connections to the database server by setting the IndexConnectionsCount to a value greater than 1. This allows concurrent accesses to the database, which avoids to sequentially wait for a database transaction to be concluded before starting another one. Having multiple connections makes the latency problem much less important.

Slow deletions

Deleting large studies can take much time, because removing a large number of files from a filesystem can be an expensive operation (which might sound counter-intuitive). This is especially true with HDD drives, that can be much slower than SSD (an user has reported a 20 times speedup by switching from HDD to SSD).

If switching from HDD to SDD is not applicable, you may also use the Delayed Deletion plugin . The plugin would maintains a queue of files to be removed. The actual deletion from the filesystem is done asynchronously in a separate thread.