Mercurial > hg > orthanc-book
annotate Sphinx/source/plugins/object-storage.rst @ 988:afea429661c2
S3 transfer mode
author | Alain Mazy <am@osimis.io> |
---|---|
date | Thu, 16 Nov 2023 08:37:28 +0100 |
parents | 5df222ddd7d1 |
children | 38909a00964c |
rev | line source |
---|---|
451 | 1 .. _object-storage: |
2 | |
3 | |
4 Cloud Object Storage plugins | |
5 ============================ | |
6 | |
7 .. contents:: | |
8 | |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
9 Release notes |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
10 ------------- |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
11 |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
12 Release notes are available `here |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
13 <https://hg.orthanc-server.com/orthanc-object-storage/file/default/NEWS>`__ |
451 | 14 |
15 Introduction | |
16 ------------ | |
17 | |
880 | 18 These 3 plugins enable storing the Orthanc files in `Object Storage <https://en.wikipedia.org/wiki/Object_storage>`__ |
19 at the 3 main Cloud providers: `AWS <https://aws.amazon.com/s3/>`__, | |
451 | 20 `Azure <https://azure.microsoft.com/en-us/services/storage/blobs/>`__ & |
21 `Google Cloud <https://cloud.google.com/storage>`__ | |
22 | |
23 Storing Orthanc files in object storage and your index SQL in a | |
24 managed database allows you to have a stateless Orthanc that does | |
25 not store any data in its local file system which is highly recommended | |
26 when deploying an application in the cloud. | |
27 | |
28 | |
459
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
29 Pre-compiled binaries |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
30 --------------------- |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
31 |
904 | 32 These 3 plugins are provided as part of the ``osimis/orthanc`` :ref:`Docker images <docker-osimis>`. |
881 | 33 The AWS plugin is available in the default Docker images while the Azure and Google plugins are available |
34 in the ``-full`` images. | |
880 | 35 |
904 | 36 The Azure plugin is also available as part of the |
37 `Windows Installer <https://orthanc.osimis.io/win-installer/OrthancInstaller-Win64-latest.exe>`__ | |
38 (only for 64bits platform). | |
39 | |
459
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
40 These plugins are used to interface Orthanc with commercial and |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
41 proprietary cloud services that you accept to pay. As a consequence, |
880 | 42 the Orthanc project usually doesn't freely update them or fix them unless |
43 the requester purchases a support contract e.g. at `Orthanc Team <https://orthanc.team>`__. | |
44 | |
45 Although you are obviously free to compile these plugins by | |
459
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
46 yourself (instructions are given below), purchasing such support |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
47 contracts makes the Orthanc project sustainable in the long term, to |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
48 the benefit of the worldwide community of medical imaging. |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
49 |
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
50 |
451 | 51 Configuration |
52 ------------- | |
53 | |
54 .. highlight:: json | |
55 | |
56 AWS S3 plugin | |
57 ^^^^^^^^^^^^^ | |
58 | |
59 Sample configuration:: | |
60 | |
61 "AwsS3Storage" : { | |
766
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
62 "BucketName": "test-orthanc-s3-plugin", |
451 | 63 "Region" : "eu-central-1", |
685 | 64 "AccessKey" : "AKXXX", // optional: if not specified, the plugin will use the default credentials manager (available from version 1.3.0) |
65 "SecretKey" : "RhYYYY", // optional: if not specified, the plugin will use the default credentials manager (available from version 1.3.0) | |
66 "Endpoint": "", // optional: custom endpoint | |
67 "ConnectionTimeout": 30, // optional: connection timeout in seconds | |
68 "RequestTimeout": 1200, // optional: request timeout in seconds (max time to upload/download a file) | |
69 "RootPath": "", // optional: see below | |
70 "MigrationFromFileSystemEnabled": false, // optional: see below | |
71 "StorageStructure": "flat", // optional: see below | |
766
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
72 "EnableLegacyUnknownFiles": true, // optional: see below |
685 | 73 "VirtualAddressing": true, // optional: see the section related to MinIO |
880 | 74 "StorageEncryption" : {}, // optional: see the section related to encryption |
988 | 75 "HybridMode": "Disabled", // optional: see the section related to Hybrid storage |
76 "UseTransferMode": true // optional: see below (available from version 2.3.0) | |
451 | 77 } |
78 | |
464 | 79 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway:: |
80 | |
81 "AwsS3Storage" : { | |
82 "BucketName": "test-orthanc", | |
83 "Region": "fr-par", | |
84 "AccessKey": "XXX", | |
85 "SecretKey": "YYY", | |
86 "Endpoint": "s3.fr-par.scw.cloud" | |
505
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
87 } |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
88 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
89 |
988 | 90 The **UseTransferMode** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client. |
91 This option was introduced in version 2.3.0 and is the default one. If set to false, the default mode | |
92 is used. | |
93 | |
94 | |
746 | 95 .. _minio: |
96 | |
505
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
97 Emulation of AWS S3 using MinIO |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
98 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
99 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
100 .. highlight:: bash |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
101 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
102 The `MinIO project <https://min.io/>`__ can be used to emulate AWS S3 |
507
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
103 for local testing/prototyping. Here is a sample command to start a |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
104 MinIO server on your local computer using Docker (evidently, make sure |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
105 to set different credentials):: |
505
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
106 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
107 $ docker run -p 9000:9000 \ |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
108 -e "MINIO_REGION=eu-west-1" \ |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
109 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \ |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
110 -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MNG/bPxRfiCYEXAMPLEKEY" \ |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
111 minio/minio server /data |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
112 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
113 .. highlight:: json |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
114 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
115 Note that the ``MINIO_REGION`` must be set to an arbitrary region that |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
116 is supported by AWS S3. |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
117 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
118 You can then open the URL `http://localhost:9000/ |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
119 <http://localhost:9000/>`__ with your Web browser to create a bucket, |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
120 say ``my-sample-bucket``. |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
121 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
122 Here is a corresponding full configuration for Orthanc:: |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
123 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
124 { |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
125 "Plugins" : [ <...> ], |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
126 "AwsS3Storage" : { |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
127 "BucketName": "my-sample-bucket", |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
128 "Region" : "eu-west-1", |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
129 "Endpoint": "http://localhost:9000/", |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
130 "AccessKey": "AKIAIOSFODNN7EXAMPLE", |
506 | 131 "SecretKey": "wJalrXUtnFEMI/K7MNG/bPxRfiCYEXAMPLEKEY", |
505
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
132 "VirtualAddressing" : false |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
133 } |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
134 } |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
135 |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
136 Note that the ``VirtualAddressing`` option must be set to ``false`` |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
137 for such a `local setup with MinIO to work |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
138 <https://github.com/aws/aws-sdk-cpp/issues/1425>`__. This option is |
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
139 **not** available in releases <= 1.1.0 of the AWS S3 plugin. |
507
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
140 |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
141 **Important:** If you get the cryptic error message |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
142 ``SignatureDoesNotMatch The request signature we calculated does not |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
143 match the signature you provided. Check your key and signing |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
144 method.``, this most probably indicates that your access key or your |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
145 secret key doesn't match the credentials that were used while starting |
a51542cfdfeb
warning about minio credentials
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
506
diff
changeset
|
146 the MinIO server. |
505
e4bea5b97890
Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
504
diff
changeset
|
147 |
464 | 148 |
451 | 149 Azure Blob Storage plugin |
150 ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
151 | |
152 Sample configuration:: | |
153 | |
154 "AzureBlobStorage" : { | |
155 "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxx;AccountKey=yyyyyyyy===;EndpointSuffix=core.windows.net", | |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
156 "ContainerName" : "test-orthanc-storage-plugin", |
647 | 157 "CreateContainerIfNotExists": true, // available from version 1.2.0 |
502 | 158 "RootPath": "", // see below |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
159 "MigrationFromFileSystemEnabled": false, // see below |
766
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
160 "StorageStructure": "flat", // see below |
880 | 161 "EnableLegacyUnknownFiles": true, // optional: see below |
162 "StorageEncryption" : {} // optional: see the section related to encryption | |
163 "HybridMode": "Disabled" // optional: see the section related to Hybrid storage | |
451 | 164 } |
165 | |
166 | |
167 Google Storage plugin | |
168 ^^^^^^^^^^^^^^^^^^^^^ | |
169 | |
170 Sample configuration:: | |
171 | |
172 "GoogleCloudStorage" : { | |
173 "ServiceAccountFile": "/path/to/googleServiceAccountFile.json", | |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
174 "BucketName": "test-orthanc-storage-plugin", |
502 | 175 "RootPath": "", // see below |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
176 "MigrationFromFileSystemEnabled": false, // see below |
766
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
177 "StorageStructure": "flat", // see below |
880 | 178 "EnableLegacyUnknownFiles": true, // optional: see below |
179 "StorageEncryption" : {} // optional: see the section related to encryption | |
180 "HybridMode": "Disabled" // optional: see the section related to Hybrid storage | |
451 | 181 } |
182 | |
183 | |
880 | 184 Migration & Hybrid mode Storage structure |
185 ----------------------------------------- | |
186 | |
187 Since version **2.1.0** of the plugins, an HybridMode as been introduced. | |
188 This mode allows reading/writing files from both/to the file system and the object-storage. | |
189 | |
190 By default, the ``HybridMode`` is ``Disabled``. This means that the plugins will access | |
191 only the object-storage. | |
192 | |
193 When the ``HybridMode`` is set to ``WriteToFileSystem``, it means that new files received | |
885 | 194 are stored on the file system. When accessing a file, it is first read from the file system |
880 | 195 and, if it is not found on the file system, it is read from the object-storage. |
196 | |
884 | 197 The ``WriteToFileSystem`` hybrid mode is useful for storing recent files on the file system for |
880 | 198 better performance and old files on the object-storage for lower cost and easier backups. |
199 | |
200 When the ``HybridMode`` is set to ``WriteToObjectStorage``, it means that new files received | |
885 | 201 are stored on the object storage. When accessing a file, it is first read from the object storage |
880 | 202 and, if it is not found on the object-storage, it is read from the file system. |
203 | |
884 | 204 The ``WriteToObjectStorage`` hybrid mode is useful mainly during a migration from file system to |
880 | 205 object-storage, e.g, if you have deployed a VM in a cloud with local file system storage and want |
206 to move your files to object-storage without interrupting your service. | |
207 | |
208 Moving files between file-system and object-storage | |
209 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
210 | |
884 | 211 When the ``HybridMode`` is set to ``WriteToFileSystem``, it is sometimes useful to move old files |
212 to the object-storage for long term archive or to `pre-fetch` files from object-storage to file | |
880 | 213 system for improved performances e.g when before opening the study in a viewer. |
214 | |
884 | 215 When the ``HybridMode`` is set to ``WriteToObjectStorage``, it is useful to move file from the |
880 | 216 file system to the object storage to perform a full data migration to object-storage. |
217 | |
218 To move files from one storage to the other, you should call the plugin Rest API:: | |
219 | |
220 $ curl -X POST http://localhost:8042/move-storage \ | |
221 --data '{ | |
222 "Resources": ["27f7126f-4f66fb14-03f4081b-f9341db2-53925988"], | |
223 "TargetStorage": "file-system", | |
224 "Asynchronous": true, | |
225 "Priority": 0 | |
226 }' | |
227 | |
228 This call creates a ``MoveStorageJob`` that can then be monitor to the ``/jobs`` route. | |
229 | |
230 The allowed values for ``TargetStorage`` are ``file-system`` or ``object-storage``. | |
231 | |
232 | |
233 Other configuration options | |
234 --------------------------- | |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
235 |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
236 The **StorageStructure** configuration allows you to select the way objects are organized |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
237 within the storage (``flat`` or ``legacy``). |
500 | 238 Unlike the traditional file system in which Orthanc uses 2 levels |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
239 of folders, object storages usually have no limit on the number of files per folder and |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
240 therefore all objects are stored at the root level of the object storage. This is the |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
241 default ``flat`` behaviour. Note that, in the ``flat`` mode, an extension `.dcm` or `.json` |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
242 is added to the filename which is not the case in the legacy mode. |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
243 |
500 | 244 The ``legacy`` behaviour mimics the Orthanc File System convention. This is actually helpful |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
245 when migrating your data from a file system to an object storage since you can copy all the file |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
246 hierarchy as is. |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
247 |
502 | 248 The **RootPath** allows you to store the files in another folder as the root level of the |
520 | 249 object storage. Note: it shall not start with a ``/``. |
502 | 250 |
251 Note that you can not change these configurations once you've uploaded the first files in Orthanc. | |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
252 |
880 | 253 The **MigrationFromFileSystemEnabled** configuration has been superseded by the **HybridMode** in v 2.1.0. |
508 | 254 |
766
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
255 The **EnableLegacyUnknownFiles** configuration has been introduced to allow recent version of the plugins (from 1.3.3) |
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
256 continue working with data that was saved with Orthanc version around 1.9.3 and plugins version around 1.2.0 (e.g. osimis/orthanc:21.5.1 docker images). |
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
257 With these specific versions, some ``.unk`` files were generated instead of ``.dcm.head`` files. With this configuration option enabled, |
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
258 when reading files, the plugin will try both file extensions. |
73ee8a489b24
object-storage: EnableLegacyUnknownFiles
Alain Mazy <am@osimis.io>
parents:
746
diff
changeset
|
259 If you have ``.unk`` files in your storage, you must enable this configuration. |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
260 |
452 | 261 Sample setups |
262 ------------- | |
263 | |
962 | 264 You'll find sample deployments and more info in the `Orthanc Setup Samples repository <https://github.com/orthanc-server/orthanc-setup-samples/tree/master/#markdown-header-for-software-integrators>`__ . |
452 | 265 |
511
5b574520a34c
performance tests for object-storage
Alain Mazy <alain@mazy.be>
parents:
509
diff
changeset
|
266 Performances |
5b574520a34c
performance tests for object-storage
Alain Mazy <alain@mazy.be>
parents:
509
diff
changeset
|
267 ------------ |
5b574520a34c
performance tests for object-storage
Alain Mazy <alain@mazy.be>
parents:
509
diff
changeset
|
268 |
962 | 269 You'll find some performance comparison between VM SSDs and object-storage `here <https://github.com/orthanc-server/orthanc-setup-samples/tree/master/docker/performance-tests/>`__ . |
511
5b574520a34c
performance tests for object-storage
Alain Mazy <alain@mazy.be>
parents:
509
diff
changeset
|
270 |
452 | 271 |
586
5f5519f1491a
securing the storage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
543
diff
changeset
|
272 .. _client-side-encryption: |
5f5519f1491a
securing the storage
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
543
diff
changeset
|
273 |
451 | 274 Client-side encryption |
275 ---------------------- | |
276 | |
277 Although all cloud providers already provide encryption at rest, the plugins provide | |
278 an optional layer of client-side encryption . It is very important that you understand | |
279 the scope and benefits of this additional layer of encryption. | |
280 | |
281 Rationale | |
282 ^^^^^^^^^ | |
283 | |
284 Encryption at rest provided by cloud providers basically compares with a file-system disk encryption. | |
285 If someone has access to the disk, he won't have access to your data without the encryption key. | |
286 | |
287 With cloud encryption at rest only, if someone has access to the "api-key" of your storage or if one | |
288 of your admin inadvertently make your storage public, `PHI <https://en.wikipedia.org/wiki/Protected_health_information>`__ will leak. | |
289 | |
290 Once you use client-side encryption, you'll basically store packets of meaningless bytes on the cloud infrastructure. | |
291 So, if an "api-key" leaks or if the storage is misconfigured, packets of bytes will leak but not PHI since | |
292 no one will be able to decrypt them. | |
293 | |
884 | 294 Another advantage is that these packets of bytes might eventually not be considered as PHI anymore and potentially |
451 | 295 help you meet your local regulations (Please check your local regulations). |
296 | |
297 However, note that, if you're running entirely in a cloud environment, your decryption keys will still | |
298 be stored on the cloud infrastructure (VM disks - process RAM) and an attacker could still eventually gain access to this keys. | |
299 | |
884 | 300 If Orthanc is running in your infrastructure with the Index DB on your infrastructure, and files are stored in the cloud, |
451 | 301 the master keys will remain on your infrastructure only and there's no way the data stored in the cloud could be decrypted outside your infrastructure. |
302 | |
303 Also note that, although the cloud providers also provide client-side encryption, we, as an open-source project, | |
304 wanted to provide our own implementation on which you'll have full control and extension capabilities. | |
305 This also allows us to implement the same logic on all cloud providers. | |
306 | |
307 Our encryption is based on well-known standards (see below). Since it is documented and the source code is open-source, | |
308 feel-free to have your security expert review it before using it in a production environment. | |
309 | |
310 Technical details | |
311 ^^^^^^^^^^^^^^^^^ | |
312 | |
313 Orthanc saves 2 kind of files: DICOM files and JSON summaries of DICOM files. Both files contain PHI. | |
314 | |
452 | 315 When configuring the plugin, you'll have to provide a **Master Key** that we can also call the **Key Encryption Key (KEK)**. |
451 | 316 |
452 | 317 For each file being saved, the plugin will generate a new **Data Encryption Key (DEK)**. This DEK, encrypted with the KEK will be pre-pended to the file. |
451 | 318 |
319 If, at any point, your KEK leaks or you want to rotate your KEKs, you'll be able to use a new one to encrypt new files that are being added | |
320 and still use the old ones to decrypt data. You could then eventually start a side script to remove usages of the leaked/obsolete KEKs. | |
321 | |
322 To summarize: | |
323 | |
452 | 324 - We use `Crypto++ <https://www.cryptopp.com/>`__ to perform all encryptions. |
451 | 325 - All keys (KEK and DEK) are AES-256 keys. |
326 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV. | |
327 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file. | |
328 | |
329 The format of data stored on disk is therefore the following: | |
330 | |
331 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently `A1` | |
332 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK | |
333 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK | |
334 - **EDEK**: 32 bytes: the DEK encrypted by the KEK. | |
335 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK | |
336 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK) | |
337 | |
338 Configuration | |
339 ^^^^^^^^^^^^^ | |
340 | |
341 .. highlight:: text | |
342 | |
343 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key:: | |
344 | |
345 openssl rand -base64 -out /tmp/test.key 32 | |
346 | |
347 Each key must have a unique id that is a uint32 number. | |
348 | |
349 .. highlight:: json | |
350 | |
351 Here's a sample configuration file of the `StorageEncryption` section of the plugins:: | |
352 | |
353 { | |
499
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
354 "GoogleCloudStorage" : { |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
355 "StorageEncryption" : { |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
356 "Enable": true, |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
357 "MasterKey": [3, "/path/to/master.key"], // key id - path to the base64 encoded key |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
358 "PreviousMasterKeys" : [ |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
359 [1, "/path/to/previous1.key"], |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
360 [2, "/path/to/previous2.key"] |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
361 ], |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
362 "MaxConcurrentInputSize" : 1024 // size in MB |
d255e02eb89d
updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents:
464
diff
changeset
|
363 } |
451 | 364 } |
365 } | |
366 | |
367 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more | |
368 than 2 times the input, we want to limit the number of threads doing concurrent processing according | |
369 to the available memory instead of the number of concurrent threads. Therefore, if you're currently | |
370 ingesting small files, you can have a lot of thread working together while, if you're ingesting large | |
371 files, threads might have to wait before receiving a "slot" to access the encryption module. | |
988 | 372 |
373 | |
374 Compilation | |
375 ----------- | |
376 | |
377 .. highlight:: text | |
378 | |
379 The procedure to compile the plugins is quite similar of that for the | |
380 :ref:`core of Orthanc <compiling>` although they usually require | |
381 some prerequisites. The documented procedure has been tested only | |
382 on a Debian Buster machine. | |
383 | |
384 The compilation of each plugin produces a shared library that contains | |
385 the plugin. | |
386 | |
387 | |
388 AWS S3 plugin | |
389 ^^^^^^^^^^^^^ | |
390 | |
391 Prerequisites: Compile the AWS C++ SDK:: | |
392 | |
393 $ mkdir ~/aws | |
394 $ cd ~/aws | |
395 $ git clone https://github.com/aws/aws-sdk-cpp.git | |
396 $ | |
397 $ mkdir -p ~/aws/builds/aws-sdk-cpp | |
398 $ cd ~/aws/builds/aws-sdk-cpp | |
399 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp | |
400 $ make -j 4 | |
401 $ make install | |
402 | |
403 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
404 | |
405 $ ./vcpkg install cryptopp | |
406 | |
407 Compile:: | |
408 | |
409 $ mkdir -p build/aws | |
410 $ cd build/aws | |
411 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws | |
412 | |
413 | |
414 **NB:** If you don't want to use vcpkg, you can use the following | |
415 command (this syntax is not compatible with Ninja yet):: | |
416 | |
417 $ cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_VCPKG_PACKAGES=OFF -DUSE_SYSTEM_GOOGLE_TEST=OFF ../../orthanc-object-storage/Aws | |
418 $ make | |
419 | |
420 Crypto++ must be installed (on Ubuntu, run ``sudo apt install libcrypto++-dev``). | |
421 | |
422 | |
423 Azure Blob Storage plugin | |
424 ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
425 | |
426 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
427 | |
428 $ ./vcpkg install cryptopp | |
429 $ ./vcpkg install azure-storage-cpp | |
430 | |
431 | |
432 Compile:: | |
433 | |
434 $ mkdir -p build/azure | |
435 $ cd build/azure | |
436 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure | |
437 | |
438 Google Storage plugin | |
439 ^^^^^^^^^^^^^^^^^^^^^ | |
440 | |
441 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
442 | |
443 $ ./vcpkg install cryptopp | |
444 $ ./vcpkg install google-cloud-cpp | |
445 | |
446 Compile:: | |
447 | |
448 $ mkdir -p build/google | |
449 $ cd build/google | |
450 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google |