Mercurial > hg > orthanc-book
comparison Sphinx/source/plugins/object-storage.rst @ 1083:e1f7f1e2d148
document AWS_EC2_METADATA_DISABLED
author | Sebastien Jodogne <s.jodogne@gmail.com> |
---|---|
date | Tue, 25 Jun 2024 12:54:16 +0200 |
parents | 5d4701d8fe28 |
children | 337bbecf74c0 |
comparison
equal
deleted
inserted
replaced
1082:202b378001f0 | 1083:e1f7f1e2d148 |
---|---|
8 | 8 |
9 Release notes | 9 Release notes |
10 ------------- | 10 ------------- |
11 | 11 |
12 Release notes are available `here | 12 Release notes are available `here |
13 <https://orthanc.uclouvain.be/hg/orthanc-object-storage/file/default/NEWS>`__ | 13 <https://orthanc.uclouvain.be/hg/orthanc-object-storage/file/default/NEWS>`__. |
14 | 14 |
15 Introduction | 15 Introduction |
16 ------------ | 16 ------------ |
17 | 17 |
18 These 3 plugins enable storing the Orthanc files in `Object Storage <https://en.wikipedia.org/wiki/Object_storage>`__ | 18 These 3 plugins enable storing the Orthanc files into `object storage |
19 at the 3 main Cloud providers: `AWS <https://aws.amazon.com/s3/>`__, | 19 <https://en.wikipedia.org/wiki/Object_storage>`__ at the 3 public |
20 `Azure <https://azure.microsoft.com/en-us/services/storage/blobs/>`__ & | 20 cloud providers: `AWS <https://aws.amazon.com/s3/>`__, `Azure |
21 `Google Cloud <https://cloud.google.com/storage>`__ | 21 <https://azure.microsoft.com/en-us/services/storage/blobs/>`__, and |
22 | 22 `Google Cloud <https://cloud.google.com/storage>`__. |
23 Storing Orthanc files in object storage and your index SQL in a | 23 |
24 managed database allows you to have a stateless Orthanc that does | 24 Storing Orthanc files in object storage and your index SQL in a |
25 not store any data in its local file system which is highly recommended | 25 managed database allows you to have a stateless Orthanc that does not |
26 store any data in its local file system, which is highly recommended | |
26 when deploying an application in the cloud. | 27 when deploying an application in the cloud. |
27 | 28 |
28 | 29 |
29 Pre-compiled binaries | 30 Pre-compiled binaries |
30 --------------------- | 31 --------------------- |
37 <https://orthanc.uclouvain.be/downloads/windows-64/installers/index.html>`__ | 38 <https://orthanc.uclouvain.be/downloads/windows-64/installers/index.html>`__ |
38 (only for 64bits platform). | 39 (only for 64bits platform). |
39 | 40 |
40 These plugins are used to interface Orthanc with commercial and | 41 These plugins are used to interface Orthanc with commercial and |
41 proprietary cloud services that you accept to pay. As a consequence, | 42 proprietary cloud services that you accept to pay. As a consequence, |
42 the Orthanc project usually doesn't freely update them or fix them unless | 43 the Orthanc project usually doesn't freely update them or fix them |
43 the requester purchases a support contract e.g. at `Orthanc Team <https://orthanc.team>`__. | 44 unless the requester purchases a support contract, e.g., at `Orthanc |
45 Team <https://orthanc.team>`__. | |
44 | 46 |
45 Although you are obviously free to compile these plugins by | 47 Although you are obviously free to compile these plugins by |
46 yourself (instructions are given below), purchasing such support | 48 yourself (instructions are given below), purchasing such support |
47 contracts makes the Orthanc project sustainable in the long term, to | 49 contracts makes the Orthanc project sustainable in the long term, to |
48 the benefit of the worldwide community of medical imaging. | 50 the benefit of the worldwide community of medical imaging. |
89 | 91 |
90 | 92 |
91 The **UseTransferManager** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client. | 93 The **UseTransferManager** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client. |
92 This option was introduced in version 2.3.0. If set to false (default value), the default "object" mode is used. | 94 This option was introduced in version 2.3.0. If set to false (default value), the default "object" mode is used. |
93 | 95 |
96 **Important:** On Microsoft Windows, it is recommended to set the | |
97 environment variable ``AWS_EC2_METADATA_DISABLED`` to ``true`` to | |
98 speed up the initialization of the plugin. The reasons are explained | |
99 in the `AWS official documentation | |
100 <https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html>`__. | |
101 | |
94 | 102 |
95 .. _minio: | 103 .. _minio: |
96 | 104 |
97 Emulation of AWS S3 using MinIO | 105 Emulation of AWS S3 using MinIO |
98 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 106 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
207 | 215 |
208 Moving files between file-system and object-storage | 216 Moving files between file-system and object-storage |
209 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 217 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
210 | 218 |
211 When the ``HybridMode`` is set to ``WriteToFileSystem``, it is sometimes useful to move old files | 219 When the ``HybridMode`` is set to ``WriteToFileSystem``, it is sometimes useful to move old files |
212 to the object-storage for long term archive or to `pre-fetch` files from object-storage to file | 220 to the object-storage for long term archive or to "pre-fetch" files from object-storage to file |
213 system for improved performances e.g when before opening the study in a viewer. | 221 system for improved performances e.g when before opening the study in a viewer. |
214 | 222 |
215 When the ``HybridMode`` is set to ``WriteToObjectStorage``, it is useful to move file from the | 223 When the ``HybridMode`` is set to ``WriteToObjectStorage``, it is useful to move file from the |
216 file system to the object storage to perform a full data migration to object-storage. | 224 file system to the object storage to perform a full data migration to object-storage. |
217 | 225 |
234 --------------------------- | 242 --------------------------- |
235 | 243 |
236 The **StorageStructure** configuration allows you to select the way objects are organized | 244 The **StorageStructure** configuration allows you to select the way objects are organized |
237 within the storage (``flat`` or ``legacy``). | 245 within the storage (``flat`` or ``legacy``). |
238 Unlike the traditional file system in which Orthanc uses 2 levels | 246 Unlike the traditional file system in which Orthanc uses 2 levels |
239 of folders, object storages usually have no limit on the number of files per folder and | 247 of folders, an object storage usually has no limit on the number of files per folder and |
240 therefore all objects are stored at the root level of the object storage. This is the | 248 therefore all objects are stored at the root level of the object storage. This is the |
241 default ``flat`` behaviour. Note that, in the ``flat`` mode, an extension `.dcm` or `.json` | 249 default ``flat`` behavior. Note that, in the ``flat`` mode, an extension ``.dcm`` or ``.json`` |
242 is added to the filename which is not the case in the legacy mode. | 250 is added to the filename which is not the case in the legacy mode. |
243 | 251 |
244 The ``legacy`` behaviour mimics the Orthanc File System convention. This is actually helpful | 252 The ``legacy`` behavior mimics the Orthanc File System convention. This is actually helpful |
245 when migrating your data from a file system to an object storage since you can copy all the file | 253 when migrating your data from a file system to an object storage since you can copy all the file |
246 hierarchy as is. | 254 hierarchy as is. |
247 | 255 |
248 *Note* : With AWS S3, there might be `some limitations <https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html>`__ | 256 *Note* : With AWS S3, there might be `some limitations <https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html>`__ |
249 on the number of HTTP operations per *prefix*. Therefore, the ``legacy`` structure might | 257 on the number of HTTP operations per *prefix*. Therefore, the ``legacy`` structure might |
330 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV. | 338 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV. |
331 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file. | 339 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file. |
332 | 340 |
333 The format of data stored on disk is therefore the following: | 341 The format of data stored on disk is therefore the following: |
334 | 342 |
335 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently `A1` | 343 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently ``A1`` |
336 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK | 344 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK |
337 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK | 345 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK |
338 - **EDEK**: 32 bytes: the DEK encrypted by the KEK. | 346 - **EDEK**: 32 bytes: the DEK encrypted by the KEK. |
339 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK | 347 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK |
340 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK) | 348 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK) |
346 | 354 |
347 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key:: | 355 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key:: |
348 | 356 |
349 openssl rand -base64 -out /tmp/test.key 32 | 357 openssl rand -base64 -out /tmp/test.key 32 |
350 | 358 |
351 Each key must have a unique id that is a uint32 number. | 359 Each key must have a unique id that is a ``uint32`` number. |
352 | 360 |
353 .. highlight:: json | 361 .. highlight:: json |
354 | 362 |
355 Here's a sample configuration file of the `StorageEncryption` section of the plugins:: | 363 Here's a sample configuration file of the ``StorageEncryption`` section of the plugins:: |
356 | 364 |
357 { | 365 { |
358 "GoogleCloudStorage" : { | 366 "GoogleCloudStorage" : { |
359 "StorageEncryption" : { | 367 "StorageEncryption" : { |
360 "Enable": true, | 368 "Enable": true, |