Mercurial > hg > orthanc-book

comparison Sphinx/source/plugins/object-storage.rst @ 451:938206110483

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
added object storage
author Alain Mazy <alain@mazy.be>
date Fri, 03 Jul 2020 12:52:11 +0200
parents
children aef5c8b74381
comparison
equal deleted inserted replaced
450:235a5541e06e 451:938206110483
1 .. _object-storage:
2
3
4 Cloud Object Storage plugins
5 ============================
6
7 .. contents::
8
9
10 Introduction
11 ------------
12
13 Osimis freely provides the `source code
14 <https://hg.orthanc-server.com/orthanc-object-storage/file/default/>`__ of 3 plugins
15 to store the Orthanc files in `Object Storage <https://en.wikipedia.org/wiki/Object_storage>`__
16 at the 3 main providers: `AWS <https://aws.amazon.com/s3/>`__,
17 `Azure <https://azure.microsoft.com/en-us/services/storage/blobs/>`__ &
18 `Google Cloud <https://cloud.google.com/storage>`__
19
20 Storing Orthanc files in object storage and your index SQL in a
21 managed database allows you to have a stateless Orthanc that does
22 not store any data in its local file system which is highly recommended
23 when deploying an application in the cloud.
24
25
26 Compilation
27 -----------
28
29 .. highlight:: text
30
31 The procedure to compile the plugins is quite similar of that for the
32 :ref:`core of Orthanc <compiling>` although they usually require
33 some prerequisites. The documented procedure has been tested only
34 on a Debian Buster machine.
35
36 The compilation of each plugin produces a shared library that contains
37 the plugin.
38
39 Given thes plugins are used to interface with a commercial & proprietary
40 service, pre-compiled Windows/Docker binaries are available only for
41 companies who have subscribed for a `support contract <https://www.osimis.io/en/services.html#cloud-plugins>`__ at Osimis.
42
43
44 AWS S3 plugin
45 ^^^^^^^^^^^^^
46
47 Prerequisites: Compile the AWS C++ SDK::
48
49 $ mkdir ~/aws
50 $ cd ~/aws
51 $ git clone https://github.com/aws/aws-sdk-cpp.git
52 $
53 $ mkdir -p ~/aws/builds/aws-sdk-cpp
54 $ cd ~/aws/builds/aws-sdk-cpp
55 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp
56 $ make -j 4
57 $ make install
58
59 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
60
61 $ ./vcpkg install cryptopp
62
63 Compile::
64
65 $ mkdir -p build/aws
66 $ cd build/aws
67 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws
68
69 Azure Blob Storage plugin
70 ^^^^^^^^^^^^^^^^^^^^^^^^^
71
72 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
73
74 $ ./vcpkg install cpprestsdk
75
76
77 Compile::
78
79 $ mkdir -p build/azure
80 $ cd build/azure
81 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure
82
83 Google Storage plugin
84 ^^^^^^^^^^^^^^^^^^^^^
85
86 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
87
88 $ ./vcpkg install google-cloud-cpp
89 $ ./vcpkg install cryptopp
90
91 Compile::
92
93 $ mkdir -p build/google
94 $ cd build/google
95 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google
96
97
98 Configuration
99 -------------
100
101 .. highlight:: json
102
103 AWS S3 plugin
104 ^^^^^^^^^^^^^
105
106 Sample configuration::
107
108 "AwsS3Storage" : {
109 "BucketName": "test-orthanc-s3-plugin",
110 "Region" : "eu-central-1",
111 "AccessKey" : "AKXXX",
112 "SecretKey" : "RhYYYY"
113 }
114
115 Azure Blob Storage plugin
116 ^^^^^^^^^^^^^^^^^^^^^^^^^
117
118 Sample configuration::
119
120 "AzureBlobStorage" : {
121 "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxx;AccountKey=yyyyyyyy===;EndpointSuffix=core.windows.net",
122 "ContainerName" : "test-orthanc-storage-plugin"
123 }
124
125
126 Google Storage plugin
127 ^^^^^^^^^^^^^^^^^^^^^
128
129 Sample configuration::
130
131 "GoogleCloudStorage" : {
132 "ServiceAccountFile": "/path/to/googleServiceAccountFile.json",
133 "BucketName": "test-orthanc-storage-plugin"
134 }
135
136
137 Client-side encryption
138 ----------------------
139
140 Although all cloud providers already provide encryption at rest, the plugins provide
141 an optional layer of client-side encryption . It is very important that you understand
142 the scope and benefits of this additional layer of encryption.
143
144 Rationale
145 ^^^^^^^^^
146
147 Encryption at rest provided by cloud providers basically compares with a file-system disk encryption.
148 If someone has access to the disk, he won't have access to your data without the encryption key.
149
150 With cloud encryption at rest only, if someone has access to the "api-key" of your storage or if one
151 of your admin inadvertently make your storage public, `PHI <https://en.wikipedia.org/wiki/Protected_health_information>`__ will leak.
152
153 Once you use client-side encryption, you'll basically store packets of meaningless bytes on the cloud infrastructure.
154 So, if an "api-key" leaks or if the storage is misconfigured, packets of bytes will leak but not PHI since
155 no one will be able to decrypt them.
156
157 Another advantage is that these packets of bytes might eventually not be considered as PHI anymore and eventually
158 help you meet your local regulations (Please check your local regulations).
159
160 However, note that, if you're running entirely in a cloud environment, your decryption keys will still
161 be stored on the cloud infrastructure (VM disks - process RAM) and an attacker could still eventually gain access to this keys.
162 Furthermore, in the scope of the `Cloud Act <https://en.wikipedia.org/wiki/CLOUD_Act>`__ , the cloud provider might still have
163 the possibility to retrieve your data and encryption key (while it will still be more complex than with standard encryption at rest).
164
165 If Orthanc is running in your infrastructure with the Index DB on your infrastructure, and files are store in the cloud,
166 the master keys will remain on your infrastructure only and there's no way the data stored in the cloud could be decrypted outside your infrastructure.
167
168 Also note that, although the cloud providers also provide client-side encryption, we, as an open-source project,
169 wanted to provide our own implementation on which you'll have full control and extension capabilities.
170 This also allows us to implement the same logic on all cloud providers.
171
172 Our encryption is based on well-known standards (see below). Since it is documented and the source code is open-source,
173 feel-free to have your security expert review it before using it in a production environment.
174
175 Technical details
176 ^^^^^^^^^^^^^^^^^
177
178 Orthanc saves 2 kind of files: DICOM files and JSON summaries of DICOM files. Both files contain PHI.
179
180 When configuring the plugin, you'll have to provide a `Master Key` that we can also call the `Key Encryption Key` (KEK).
181
182 For each file being saved, the plugin will generate a new `Data Encryption Key` (DEK). This DEK, encrypted with the KEK will be pre-pended to the file.
183
184 If, at any point, your KEK leaks or you want to rotate your KEKs, you'll be able to use a new one to encrypt new files that are being added
185 and still use the old ones to decrypt data. You could then eventually start a side script to remove usages of the leaked/obsolete KEKs.
186
187 To summarize:
188
189 - We use `Crypto++<https://www.cryptopp.com/>`__ to perform all encryptions.
190 - All keys (KEK and DEK) are AES-256 keys.
191 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV.
192 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file.
193
194 The format of data stored on disk is therefore the following:
195
196 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently `A1`
197 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK
198 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK
199 - **EDEK**: 32 bytes: the DEK encrypted by the KEK.
200 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK
201 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK)
202
203 Configuration
204 ^^^^^^^^^^^^^
205
206 .. highlight:: text
207
208 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key::
209
210 openssl rand -base64 -out /tmp/test.key 32
211
212 Each key must have a unique id that is a uint32 number.
213
214 .. highlight:: json
215
216 Here's a sample configuration file of the `StorageEncryption` section of the plugins::
217
218 {
219 "StorageEncryption" : {
220 "Enable": true,
221 "MasterKey": [3, "/path/to/master.key"], // key id - path to the base64 encoded key
222 "PreviousMasterKeys" : [
223 [1, "/path/to/previous1.key"],
224 [2, "/path/to/previous2.key"]
225 ],
226 "MaxConcurrentInputSize" : 1024 // size in MB
227 }
228 }
229
230 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more
231 than 2 times the input, we want to limit the number of threads doing concurrent processing according
232 to the available memory instead of the number of concurrent threads. Therefore, if you're currently
233 ingesting small files, you can have a lot of thread working together while, if you're ingesting large
234 files, threads might have to wait before receiving a "slot" to access the encryption module.