Mercurial > hg > orthanc-book
annotate Sphinx/source/plugins/object-storage.rst @ 464:5ea70331c0be
AWS Endpoint doc
| author | Alain Mazy <alain@mazy.be> |
|---|---|
| date | Thu, 09 Jul 2020 11:15:28 +0200 |
| parents | fae16ea8b13c |
| children | d255e02eb89d |
| rev | line source |
|---|---|
| 451 | 1 .. _object-storage: |
| 2 | |
| 3 | |
| 4 Cloud Object Storage plugins | |
| 5 ============================ | |
| 6 | |
| 7 .. contents:: | |
| 8 | |
| 9 | |
| 10 Introduction | |
| 11 ------------ | |
| 12 | |
| 13 Osimis freely provides the `source code | |
| 14 <https://hg.orthanc-server.com/orthanc-object-storage/file/default/>`__ of 3 plugins | |
| 15 to store the Orthanc files in `Object Storage <https://en.wikipedia.org/wiki/Object_storage>`__ | |
| 16 at the 3 main providers: `AWS <https://aws.amazon.com/s3/>`__, | |
| 17 `Azure <https://azure.microsoft.com/en-us/services/storage/blobs/>`__ & | |
| 18 `Google Cloud <https://cloud.google.com/storage>`__ | |
| 19 | |
| 20 Storing Orthanc files in object storage and your index SQL in a | |
| 21 managed database allows you to have a stateless Orthanc that does | |
| 22 not store any data in its local file system which is highly recommended | |
| 23 when deploying an application in the cloud. | |
| 24 | |
| 25 | |
|
459
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
26 Pre-compiled binaries |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
27 --------------------- |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
28 |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
29 These plugins are used to interface Orthanc with commercial and |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
30 proprietary cloud services that you accept to pay. As a consequence, |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
31 the Orthanc project doesn't freely provide pre-compiled binaries for |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
32 Docker, Windows, Linux or OS X. These pre-compiled binaries do exist, |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
33 but are reserved to the companies who have subscribed to a |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
34 `professional support contract |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
35 <https://www.osimis.io/en/services.html#cloud-plugins>`__ by |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
36 Osimis. Although you are obviously free to compile these plugins by |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
37 yourself (instructions are given below), purchasing such support |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
38 contracts makes the Orthanc project sustainable in the long term, to |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
39 the benefit of the worldwide community of medical imaging. |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
40 |
|
a4ed4e883337
highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
453
diff
changeset
|
41 |
| 451 | 42 Compilation |
| 43 ----------- | |
| 44 | |
| 45 .. highlight:: text | |
| 46 | |
| 47 The procedure to compile the plugins is quite similar of that for the | |
| 48 :ref:`core of Orthanc <compiling>` although they usually require | |
| 49 some prerequisites. The documented procedure has been tested only | |
| 50 on a Debian Buster machine. | |
| 51 | |
| 52 The compilation of each plugin produces a shared library that contains | |
| 53 the plugin. | |
| 54 | |
| 55 | |
| 56 AWS S3 plugin | |
| 57 ^^^^^^^^^^^^^ | |
| 58 | |
| 59 Prerequisites: Compile the AWS C++ SDK:: | |
| 60 | |
| 61 $ mkdir ~/aws | |
| 62 $ cd ~/aws | |
| 63 $ git clone https://github.com/aws/aws-sdk-cpp.git | |
| 64 $ | |
| 65 $ mkdir -p ~/aws/builds/aws-sdk-cpp | |
| 66 $ cd ~/aws/builds/aws-sdk-cpp | |
| 67 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp | |
| 68 $ make -j 4 | |
| 69 $ make install | |
| 70 | |
| 71 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
| 72 | |
| 73 $ ./vcpkg install cryptopp | |
| 74 | |
| 75 Compile:: | |
| 76 | |
| 77 $ mkdir -p build/aws | |
| 78 $ cd build/aws | |
| 79 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws | |
| 80 | |
| 81 Azure Blob Storage plugin | |
| 82 ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 83 | |
| 84 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
| 85 | |
| 86 $ ./vcpkg install cpprestsdk | |
| 87 | |
| 88 | |
| 89 Compile:: | |
| 90 | |
| 91 $ mkdir -p build/azure | |
| 92 $ cd build/azure | |
| 93 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure | |
| 94 | |
| 95 Google Storage plugin | |
| 96 ^^^^^^^^^^^^^^^^^^^^^ | |
| 97 | |
| 98 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
| 99 | |
| 100 $ ./vcpkg install google-cloud-cpp | |
| 101 $ ./vcpkg install cryptopp | |
| 102 | |
| 103 Compile:: | |
| 104 | |
| 105 $ mkdir -p build/google | |
| 106 $ cd build/google | |
| 107 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google | |
| 108 | |
| 109 | |
| 110 Configuration | |
| 111 ------------- | |
| 112 | |
| 113 .. highlight:: json | |
| 114 | |
| 115 AWS S3 plugin | |
| 116 ^^^^^^^^^^^^^ | |
| 117 | |
| 118 Sample configuration:: | |
| 119 | |
| 120 "AwsS3Storage" : { | |
| 121 "BucketName": "test-orthanc-s3-plugin", | |
| 122 "Region" : "eu-central-1", | |
| 123 "AccessKey" : "AKXXX", | |
| 463 | 124 "SecretKey" : "RhYYYY", |
| 125 "Endpoint": "", // optional - currently in mainline version only: custom endpoint | |
| 126 "ConnectionTimeout": 30, // optional - currently in mainline version only: connection timeout in seconds | |
| 127 "RequestTimeout": 1200 // optional - currently in mainline version only: request timeout in seconds (max time to upload/download a file) | |
| 451 | 128 } |
| 129 | |
| 464 | 130 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway:: |
| 131 | |
| 132 "AwsS3Storage" : { | |
| 133 "BucketName": "test-orthanc", | |
| 134 "Region": "fr-par", | |
| 135 "AccessKey": "XXX", | |
| 136 "SecretKey": "YYY", | |
| 137 "Endpoint": "s3.fr-par.scw.cloud" | |
| 138 }, | |
| 139 | |
| 451 | 140 Azure Blob Storage plugin |
| 141 ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 142 | |
| 143 Sample configuration:: | |
| 144 | |
| 145 "AzureBlobStorage" : { | |
| 146 "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxx;AccountKey=yyyyyyyy===;EndpointSuffix=core.windows.net", | |
| 147 "ContainerName" : "test-orthanc-storage-plugin" | |
| 148 } | |
| 149 | |
| 150 | |
| 151 Google Storage plugin | |
| 152 ^^^^^^^^^^^^^^^^^^^^^ | |
| 153 | |
| 154 Sample configuration:: | |
| 155 | |
| 156 "GoogleCloudStorage" : { | |
| 157 "ServiceAccountFile": "/path/to/googleServiceAccountFile.json", | |
| 158 "BucketName": "test-orthanc-storage-plugin" | |
| 159 } | |
| 160 | |
| 161 | |
| 452 | 162 Sample setups |
| 163 ------------- | |
| 164 | |
| 165 You'll find sample deployments and more info in the `Orthanc Setup Samples repository <https://bitbucket.org/osimis/orthanc-setup-samples/src/master/#markdown-header-for-osimisorthanc-pro-image-users>`__ . | |
| 166 | |
| 167 | |
| 451 | 168 Client-side encryption |
| 169 ---------------------- | |
| 170 | |
| 171 Although all cloud providers already provide encryption at rest, the plugins provide | |
| 172 an optional layer of client-side encryption . It is very important that you understand | |
| 173 the scope and benefits of this additional layer of encryption. | |
| 174 | |
| 175 Rationale | |
| 176 ^^^^^^^^^ | |
| 177 | |
| 178 Encryption at rest provided by cloud providers basically compares with a file-system disk encryption. | |
| 179 If someone has access to the disk, he won't have access to your data without the encryption key. | |
| 180 | |
| 181 With cloud encryption at rest only, if someone has access to the "api-key" of your storage or if one | |
| 182 of your admin inadvertently make your storage public, `PHI <https://en.wikipedia.org/wiki/Protected_health_information>`__ will leak. | |
| 183 | |
| 184 Once you use client-side encryption, you'll basically store packets of meaningless bytes on the cloud infrastructure. | |
| 185 So, if an "api-key" leaks or if the storage is misconfigured, packets of bytes will leak but not PHI since | |
| 186 no one will be able to decrypt them. | |
| 187 | |
| 188 Another advantage is that these packets of bytes might eventually not be considered as PHI anymore and eventually | |
| 189 help you meet your local regulations (Please check your local regulations). | |
| 190 | |
| 191 However, note that, if you're running entirely in a cloud environment, your decryption keys will still | |
| 192 be stored on the cloud infrastructure (VM disks - process RAM) and an attacker could still eventually gain access to this keys. | |
| 193 | |
| 194 If Orthanc is running in your infrastructure with the Index DB on your infrastructure, and files are store in the cloud, | |
| 195 the master keys will remain on your infrastructure only and there's no way the data stored in the cloud could be decrypted outside your infrastructure. | |
| 196 | |
| 197 Also note that, although the cloud providers also provide client-side encryption, we, as an open-source project, | |
| 198 wanted to provide our own implementation on which you'll have full control and extension capabilities. | |
| 199 This also allows us to implement the same logic on all cloud providers. | |
| 200 | |
| 201 Our encryption is based on well-known standards (see below). Since it is documented and the source code is open-source, | |
| 202 feel-free to have your security expert review it before using it in a production environment. | |
| 203 | |
| 204 Technical details | |
| 205 ^^^^^^^^^^^^^^^^^ | |
| 206 | |
| 207 Orthanc saves 2 kind of files: DICOM files and JSON summaries of DICOM files. Both files contain PHI. | |
| 208 | |
| 452 | 209 When configuring the plugin, you'll have to provide a **Master Key** that we can also call the **Key Encryption Key (KEK)**. |
| 451 | 210 |
| 452 | 211 For each file being saved, the plugin will generate a new **Data Encryption Key (DEK)**. This DEK, encrypted with the KEK will be pre-pended to the file. |
| 451 | 212 |
| 213 If, at any point, your KEK leaks or you want to rotate your KEKs, you'll be able to use a new one to encrypt new files that are being added | |
| 214 and still use the old ones to decrypt data. You could then eventually start a side script to remove usages of the leaked/obsolete KEKs. | |
| 215 | |
| 216 To summarize: | |
| 217 | |
| 452 | 218 - We use `Crypto++ <https://www.cryptopp.com/>`__ to perform all encryptions. |
| 451 | 219 - All keys (KEK and DEK) are AES-256 keys. |
| 220 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV. | |
| 221 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file. | |
| 222 | |
| 223 The format of data stored on disk is therefore the following: | |
| 224 | |
| 225 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently `A1` | |
| 226 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK | |
| 227 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK | |
| 228 - **EDEK**: 32 bytes: the DEK encrypted by the KEK. | |
| 229 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK | |
| 230 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK) | |
| 231 | |
| 232 Configuration | |
| 233 ^^^^^^^^^^^^^ | |
| 234 | |
| 235 .. highlight:: text | |
| 236 | |
| 237 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key:: | |
| 238 | |
| 239 openssl rand -base64 -out /tmp/test.key 32 | |
| 240 | |
| 241 Each key must have a unique id that is a uint32 number. | |
| 242 | |
| 243 .. highlight:: json | |
| 244 | |
| 245 Here's a sample configuration file of the `StorageEncryption` section of the plugins:: | |
| 246 | |
| 247 { | |
| 248 "StorageEncryption" : { | |
| 249 "Enable": true, | |
| 250 "MasterKey": [3, "/path/to/master.key"], // key id - path to the base64 encoded key | |
| 251 "PreviousMasterKeys" : [ | |
| 252 [1, "/path/to/previous1.key"], | |
| 253 [2, "/path/to/previous2.key"] | |
| 254 ], | |
| 255 "MaxConcurrentInputSize" : 1024 // size in MB | |
| 256 } | |
| 257 } | |
| 258 | |
| 259 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more | |
| 260 than 2 times the input, we want to limit the number of threads doing concurrent processing according | |
| 261 to the available memory instead of the number of concurrent threads. Therefore, if you're currently | |
| 262 ingesting small files, you can have a lot of thread working together while, if you're ingesting large | |
| 263 files, threads might have to wait before receiving a "slot" to access the encryption module. |