Mercurial > hg > orthanc-book

comparison Sphinx/source/plugins/object-storage.rst @ 1083:e1f7f1e2d148

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
document AWS_EC2_METADATA_DISABLED
author Sebastien Jodogne <s.jodogne@gmail.com>
date Tue, 25 Jun 2024 12:54:16 +0200
parents 5d4701d8fe28
children 337bbecf74c0
comparison
equal deleted inserted replaced
1082:202b378001f0 1083:e1f7f1e2d148
8 8
9 Release notes 9 Release notes
10 ------------- 10 -------------
11 11
12 Release notes are available `here 12 Release notes are available `here
13 <https://orthanc.uclouvain.be/hg/orthanc-object-storage/file/default/NEWS>`__ 13 <https://orthanc.uclouvain.be/hg/orthanc-object-storage/file/default/NEWS>`__.
14 14
15 Introduction 15 Introduction
16 ------------ 16 ------------
17 17
18 These 3 plugins enable storing the Orthanc files in `Object Storage <https://en.wikipedia.org/wiki/Object_storage>`__ 18 These 3 plugins enable storing the Orthanc files into `object storage
19 at the 3 main Cloud providers: `AWS <https://aws.amazon.com/s3/>`__, 19 <https://en.wikipedia.org/wiki/Object_storage>`__ at the 3 public
20 `Azure <https://azure.microsoft.com/en-us/services/storage/blobs/>`__ & 20 cloud providers: `AWS <https://aws.amazon.com/s3/>`__, `Azure
21 `Google Cloud <https://cloud.google.com/storage>`__ 21 <https://azure.microsoft.com/en-us/services/storage/blobs/>`__, and
22 22 `Google Cloud <https://cloud.google.com/storage>`__.
23 Storing Orthanc files in object storage and your index SQL in a 23
24 managed database allows you to have a stateless Orthanc that does 24 Storing Orthanc files in object storage and your index SQL in a
25 not store any data in its local file system which is highly recommended 25 managed database allows you to have a stateless Orthanc that does not
26 store any data in its local file system, which is highly recommended
26 when deploying an application in the cloud. 27 when deploying an application in the cloud.
27 28
28 29
29 Pre-compiled binaries 30 Pre-compiled binaries
30 --------------------- 31 ---------------------
37 <https://orthanc.uclouvain.be/downloads/windows-64/installers/index.html>`__ 38 <https://orthanc.uclouvain.be/downloads/windows-64/installers/index.html>`__
38 (only for 64bits platform). 39 (only for 64bits platform).
39 40
40 These plugins are used to interface Orthanc with commercial and 41 These plugins are used to interface Orthanc with commercial and
41 proprietary cloud services that you accept to pay. As a consequence, 42 proprietary cloud services that you accept to pay. As a consequence,
42 the Orthanc project usually doesn't freely update them or fix them unless 43 the Orthanc project usually doesn't freely update them or fix them
43 the requester purchases a support contract e.g. at `Orthanc Team <https://orthanc.team>`__. 44 unless the requester purchases a support contract, e.g., at `Orthanc
45 Team <https://orthanc.team>`__.
44 46
45 Although you are obviously free to compile these plugins by 47 Although you are obviously free to compile these plugins by
46 yourself (instructions are given below), purchasing such support 48 yourself (instructions are given below), purchasing such support
47 contracts makes the Orthanc project sustainable in the long term, to 49 contracts makes the Orthanc project sustainable in the long term, to
48 the benefit of the worldwide community of medical imaging. 50 the benefit of the worldwide community of medical imaging.
89 91
90 92
91 The **UseTransferManager** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client. 93 The **UseTransferManager** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client.
92 This option was introduced in version 2.3.0. If set to false (default value), the default "object" mode is used. 94 This option was introduced in version 2.3.0. If set to false (default value), the default "object" mode is used.
93 95
96 **Important:** On Microsoft Windows, it is recommended to set the
97 environment variable ``AWS_EC2_METADATA_DISABLED`` to ``true`` to
98 speed up the initialization of the plugin. The reasons are explained
99 in the `AWS official documentation
100 <https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html>`__.
101
94 102
95 .. _minio: 103 .. _minio:
96 104
97 Emulation of AWS S3 using MinIO 105 Emulation of AWS S3 using MinIO
98 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 106 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
207 215
208 Moving files between file-system and object-storage 216 Moving files between file-system and object-storage
209 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 217 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
210 218
211 When the ``HybridMode`` is set to ``WriteToFileSystem``, it is sometimes useful to move old files 219 When the ``HybridMode`` is set to ``WriteToFileSystem``, it is sometimes useful to move old files
212 to the object-storage for long term archive or to `pre-fetch` files from object-storage to file 220 to the object-storage for long term archive or to "pre-fetch" files from object-storage to file
213 system for improved performances e.g when before opening the study in a viewer. 221 system for improved performances e.g when before opening the study in a viewer.
214 222
215 When the ``HybridMode`` is set to ``WriteToObjectStorage``, it is useful to move file from the 223 When the ``HybridMode`` is set to ``WriteToObjectStorage``, it is useful to move file from the
216 file system to the object storage to perform a full data migration to object-storage. 224 file system to the object storage to perform a full data migration to object-storage.
217 225
234 --------------------------- 242 ---------------------------
235 243
236 The **StorageStructure** configuration allows you to select the way objects are organized 244 The **StorageStructure** configuration allows you to select the way objects are organized
237 within the storage (``flat`` or ``legacy``). 245 within the storage (``flat`` or ``legacy``).
238 Unlike the traditional file system in which Orthanc uses 2 levels 246 Unlike the traditional file system in which Orthanc uses 2 levels
239 of folders, object storages usually have no limit on the number of files per folder and 247 of folders, an object storage usually has no limit on the number of files per folder and
240 therefore all objects are stored at the root level of the object storage. This is the 248 therefore all objects are stored at the root level of the object storage. This is the
241 default ``flat`` behaviour. Note that, in the ``flat`` mode, an extension `.dcm` or `.json` 249 default ``flat`` behavior. Note that, in the ``flat`` mode, an extension ``.dcm`` or ``.json``
242 is added to the filename which is not the case in the legacy mode. 250 is added to the filename which is not the case in the legacy mode.
243 251
244 The ``legacy`` behaviour mimics the Orthanc File System convention. This is actually helpful 252 The ``legacy`` behavior mimics the Orthanc File System convention. This is actually helpful
245 when migrating your data from a file system to an object storage since you can copy all the file 253 when migrating your data from a file system to an object storage since you can copy all the file
246 hierarchy as is. 254 hierarchy as is.
247 255
248 *Note* : With AWS S3, there might be `some limitations <https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html>`__ 256 *Note* : With AWS S3, there might be `some limitations <https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html>`__
249 on the number of HTTP operations per *prefix*. Therefore, the ``legacy`` structure might 257 on the number of HTTP operations per *prefix*. Therefore, the ``legacy`` structure might
330 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV. 338 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV.
331 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file. 339 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file.
332 340
333 The format of data stored on disk is therefore the following: 341 The format of data stored on disk is therefore the following:
334 342
335 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently `A1` 343 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently ``A1``
336 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK 344 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK
337 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK 345 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK
338 - **EDEK**: 32 bytes: the DEK encrypted by the KEK. 346 - **EDEK**: 32 bytes: the DEK encrypted by the KEK.
339 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK 347 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK
340 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK) 348 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK)
346 354
347 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key:: 355 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key::
348 356
349 openssl rand -base64 -out /tmp/test.key 32 357 openssl rand -base64 -out /tmp/test.key 32
350 358
351 Each key must have a unique id that is a uint32 number. 359 Each key must have a unique id that is a ``uint32`` number.
352 360
353 .. highlight:: json 361 .. highlight:: json
354 362
355 Here's a sample configuration file of the `StorageEncryption` section of the plugins:: 363 Here's a sample configuration file of the ``StorageEncryption`` section of the plugins::
356 364
357 { 365 {
358 "GoogleCloudStorage" : { 366 "GoogleCloudStorage" : {
359 "StorageEncryption" : { 367 "StorageEncryption" : {
360 "Enable": true, 368 "Enable": true,