annotate Sphinx/source/plugins/object-storage.rst @ 505:e4bea5b97890

Emulation of AWS S3 using MinIO
author Sebastien Jodogne <s.jodogne@gmail.com>
date Mon, 07 Sep 2020 17:30:47 +0200
parents 2845ac3adad2
children 30d415f2b8ee
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
1 .. _object-storage:
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
2
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
3
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
4 Cloud Object Storage plugins
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
5 ============================
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
6
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
7 .. contents::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
8
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
9 Release notes
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
10 -------------
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
11
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
12 Release notes are available `here
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
13 <https://hg.orthanc-server.com/orthanc-object-storage/file/default/NEWS>`__
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
14
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
15 Introduction
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
16 ------------
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
17
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
18 Osimis freely provides the `source code
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
19 <https://hg.orthanc-server.com/orthanc-object-storage/file/default/>`__ of 3 plugins
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
20 to store the Orthanc files in `Object Storage <https://en.wikipedia.org/wiki/Object_storage>`__
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
21 at the 3 main providers: `AWS <https://aws.amazon.com/s3/>`__,
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
22 `Azure <https://azure.microsoft.com/en-us/services/storage/blobs/>`__ &
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
23 `Google Cloud <https://cloud.google.com/storage>`__
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
24
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
25 Storing Orthanc files in object storage and your index SQL in a
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
26 managed database allows you to have a stateless Orthanc that does
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
27 not store any data in its local file system which is highly recommended
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
28 when deploying an application in the cloud.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
29
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
30
459
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
31 Pre-compiled binaries
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
32 ---------------------
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
33
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
34 These plugins are used to interface Orthanc with commercial and
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
35 proprietary cloud services that you accept to pay. As a consequence,
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
36 the Orthanc project doesn't freely provide pre-compiled binaries for
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
37 Docker, Windows, Linux or OS X. These pre-compiled binaries do exist,
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
38 but are reserved to the companies who have subscribed to a
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
39 `professional support contract
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
40 <https://www.osimis.io/en/services.html#cloud-plugins>`__ by
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
41 Osimis. Although you are obviously free to compile these plugins by
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
42 yourself (instructions are given below), purchasing such support
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
43 contracts makes the Orthanc project sustainable in the long term, to
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
44 the benefit of the worldwide community of medical imaging.
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
45
a4ed4e883337 highlighting the pre-compiled binaries for google, aws and azure
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 453
diff changeset
46
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
47 Compilation
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
48 -----------
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
49
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
50 .. highlight:: text
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
51
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
52 The procedure to compile the plugins is quite similar of that for the
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
53 :ref:`core of Orthanc <compiling>` although they usually require
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
54 some prerequisites. The documented procedure has been tested only
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
55 on a Debian Buster machine.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
56
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
57 The compilation of each plugin produces a shared library that contains
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
58 the plugin.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
59
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
60
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
61 AWS S3 plugin
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
62 ^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
63
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
64 Prerequisites: Compile the AWS C++ SDK::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
65
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
66 $ mkdir ~/aws
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
67 $ cd ~/aws
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
68 $ git clone https://github.com/aws/aws-sdk-cpp.git
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
69 $
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
70 $ mkdir -p ~/aws/builds/aws-sdk-cpp
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
71 $ cd ~/aws/builds/aws-sdk-cpp
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
72 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
73 $ make -j 4
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
74 $ make install
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
75
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
76 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
77
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
78 $ ./vcpkg install cryptopp
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
79
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
80 Compile::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
81
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
82 $ mkdir -p build/aws
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
83 $ cd build/aws
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
84 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
85
504
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
86
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
87 **NB:** If you don't want to use vcpkg, you can use the following
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
88 command (this syntax is not compatible with Ninja yet)::
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
89
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
90 $ cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_VCPKG_PACKAGES=OFF ../../orthanc-object-storage/Aws
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
91 $ make
2845ac3adad2 USE_VCPKG_PACKAGES
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 502
diff changeset
92
505
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
93
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
94 Azure Blob Storage plugin
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
95 ^^^^^^^^^^^^^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
96
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
97 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
98
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
99 $ ./vcpkg install cpprestsdk
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
100
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
101
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
102 Compile::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
103
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
104 $ mkdir -p build/azure
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
105 $ cd build/azure
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
106 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
107
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
108 Google Storage plugin
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
109 ^^^^^^^^^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
110
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
111 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
112
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
113 $ ./vcpkg install google-cloud-cpp
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
114 $ ./vcpkg install cryptopp
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
115
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
116 Compile::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
117
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
118 $ mkdir -p build/google
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
119 $ cd build/google
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
120 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
121
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
122
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
123 Configuration
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
124 -------------
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
125
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
126 .. highlight:: json
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
127
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
128 AWS S3 plugin
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
129 ^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
130
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
131 Sample configuration::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
132
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
133 "AwsS3Storage" : {
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
134 "BucketName": "test-orthanc-s3-plugin",
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
135 "Region" : "eu-central-1",
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
136 "AccessKey" : "AKXXX",
463
fae16ea8b13c new AWS params
Alain Mazy <alain@mazy.be>
parents: 459
diff changeset
137 "SecretKey" : "RhYYYY",
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
138 "Endpoint": "", // custom endpoint
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
139 "ConnectionTimeout": 30, // connection timeout in seconds
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
140 "RequestTimeout": 1200, // request timeout in seconds (max time to upload/download a file)
502
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
141 "RootPath": "", // see below
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
142 "MigrationFromFileSystemEnabled": false, // see below
505
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
143 "StorageStructure": "flat", // see below
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
144 "VirtualAddressing": true // see the section related to MinIO
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
145 }
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
146
464
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
147 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway::
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
148
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
149 "AwsS3Storage" : {
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
150 "BucketName": "test-orthanc",
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
151 "Region": "fr-par",
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
152 "AccessKey": "XXX",
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
153 "SecretKey": "YYY",
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
154 "Endpoint": "s3.fr-par.scw.cloud"
505
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
155 }
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
156
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
157
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
158 Emulation of AWS S3 using MinIO
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
159 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
160
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
161 .. highlight:: bash
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
162
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
163 The `MinIO project <https://min.io/>`__ can be used to emulate AWS S3
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
164 for local testing/prototyping. Here is a sample command to start MinIO
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
165 on your local computer using Docker (evidently, make sure to adapt
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
166 your credentials)::
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
167
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
168 $ docker run -p 9000:9000 \
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
169 -e "MINIO_REGION=eu-west-1" \
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
170 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
171 -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MNG/bPxRfiCYEXAMPLEKEY" \
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
172 minio/minio server /data
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
173
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
174 .. highlight:: json
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
175
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
176 Note that the ``MINIO_REGION`` must be set to an arbitrary region that
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
177 is supported by AWS S3.
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
178
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
179 You can then open the URL `http://localhost:9000/
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
180 <http://localhost:9000/>`__ with your Web browser to create a bucket,
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
181 say ``my-sample-bucket``.
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
182
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
183 Here is a corresponding full configuration for Orthanc::
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
184
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
185 {
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
186 "Plugins" : [ <...> ],
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
187 "AwsS3Storage" : {
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
188 "BucketName": "my-sample-bucket",
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
189 "Region" : "eu-west-1",
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
190 "Endpoint": "http://localhost:9000/",
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
191 "AccessKey": "AKIAIOSFODNN7EXAMPLE",
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
192 "SecretKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
193 "VirtualAddressing" : false
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
194 }
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
195 }
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
196
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
197 Note that the ``VirtualAddressing`` option must be set to ``false``
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
198 for such a `local setup with MinIO to work
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
199 <https://github.com/aws/aws-sdk-cpp/issues/1425>`__. This option is
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
200 **not** available in releases <= 1.1.0 of the AWS S3 plugin.
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
201
e4bea5b97890 Emulation of AWS S3 using MinIO
Sebastien Jodogne <s.jodogne@gmail.com>
parents: 504
diff changeset
202
464
5ea70331c0be AWS Endpoint doc
Alain Mazy <alain@mazy.be>
parents: 463
diff changeset
203
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
204 Azure Blob Storage plugin
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
205 ^^^^^^^^^^^^^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
206
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
207 Sample configuration::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
208
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
209 "AzureBlobStorage" : {
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
210 "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxx;AccountKey=yyyyyyyy===;EndpointSuffix=core.windows.net",
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
211 "ContainerName" : "test-orthanc-storage-plugin",
502
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
212 "RootPath": "", // see below
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
213 "MigrationFromFileSystemEnabled": false, // see below
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
214 "StorageStructure": "flat" // see below
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
215 }
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
216
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
217
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
218 Google Storage plugin
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
219 ^^^^^^^^^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
220
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
221 Sample configuration::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
222
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
223 "GoogleCloudStorage" : {
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
224 "ServiceAccountFile": "/path/to/googleServiceAccountFile.json",
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
225 "BucketName": "test-orthanc-storage-plugin",
502
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
226 "RootPath": "", // see below
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
227 "MigrationFromFileSystemEnabled": false, // see below
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
228 "StorageStructure": "flat" // see below
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
229 }
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
230
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
231
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
232 Migration & Storage structure
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
233 -----------------------------
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
234
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
235 The **StorageStructure** configuration allows you to select the way objects are organized
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
236 within the storage (``flat`` or ``legacy``).
500
Alain Mazy <alain@mazy.be>
parents: 499
diff changeset
237 Unlike the traditional file system in which Orthanc uses 2 levels
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
238 of folders, object storages usually have no limit on the number of files per folder and
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
239 therefore all objects are stored at the root level of the object storage. This is the
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
240 default ``flat`` behaviour. Note that, in the ``flat`` mode, an extension `.dcm` or `.json`
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
241 is added to the filename which is not the case in the legacy mode.
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
242
500
Alain Mazy <alain@mazy.be>
parents: 499
diff changeset
243 The ``legacy`` behaviour mimics the Orthanc File System convention. This is actually helpful
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
244 when migrating your data from a file system to an object storage since you can copy all the file
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
245 hierarchy as is.
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
246
502
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
247 The **RootPath** allows you to store the files in another folder as the root level of the
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
248 object storage.
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
249
4e426dec4fee Object-storage: RootPath
Alain Mazy <alain@mazy.be>
parents: 500
diff changeset
250 Note that you can not change these configurations once you've uploaded the first files in Orthanc.
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
251
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
252 The **MigrationFromFileSystemEnabled** configuration has been for Orthanc to continue working
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
253 while you're migrating your data from the file system to the object storage. While this option is enabled,
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
254 Orthanc will store all new files into the object storage but will try to read/delete files
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
255 from both the file system and the object storage.
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
256
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
257 This option can be disabled as soon as all files have been copied from the file system to the
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
258 object storage. Note that Orthanc is not copying the files from one storage to the other; you'll
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
259 have to use a standard ``sync`` command from the object-storage provider.
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
260
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
261
452
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
262 Sample setups
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
263 -------------
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
264
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
265 You'll find sample deployments and more info in the `Orthanc Setup Samples repository <https://bitbucket.org/osimis/orthanc-setup-samples/src/master/#markdown-header-for-osimisorthanc-pro-image-users>`__ .
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
266
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
267
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
268 Client-side encryption
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
269 ----------------------
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
270
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
271 Although all cloud providers already provide encryption at rest, the plugins provide
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
272 an optional layer of client-side encryption . It is very important that you understand
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
273 the scope and benefits of this additional layer of encryption.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
274
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
275 Rationale
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
276 ^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
277
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
278 Encryption at rest provided by cloud providers basically compares with a file-system disk encryption.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
279 If someone has access to the disk, he won't have access to your data without the encryption key.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
280
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
281 With cloud encryption at rest only, if someone has access to the "api-key" of your storage or if one
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
282 of your admin inadvertently make your storage public, `PHI <https://en.wikipedia.org/wiki/Protected_health_information>`__ will leak.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
283
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
284 Once you use client-side encryption, you'll basically store packets of meaningless bytes on the cloud infrastructure.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
285 So, if an "api-key" leaks or if the storage is misconfigured, packets of bytes will leak but not PHI since
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
286 no one will be able to decrypt them.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
287
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
288 Another advantage is that these packets of bytes might eventually not be considered as PHI anymore and eventually
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
289 help you meet your local regulations (Please check your local regulations).
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
290
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
291 However, note that, if you're running entirely in a cloud environment, your decryption keys will still
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
292 be stored on the cloud infrastructure (VM disks - process RAM) and an attacker could still eventually gain access to this keys.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
293
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
294 If Orthanc is running in your infrastructure with the Index DB on your infrastructure, and files are store in the cloud,
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
295 the master keys will remain on your infrastructure only and there's no way the data stored in the cloud could be decrypted outside your infrastructure.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
296
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
297 Also note that, although the cloud providers also provide client-side encryption, we, as an open-source project,
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
298 wanted to provide our own implementation on which you'll have full control and extension capabilities.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
299 This also allows us to implement the same logic on all cloud providers.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
300
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
301 Our encryption is based on well-known standards (see below). Since it is documented and the source code is open-source,
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
302 feel-free to have your security expert review it before using it in a production environment.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
303
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
304 Technical details
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
305 ^^^^^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
306
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
307 Orthanc saves 2 kind of files: DICOM files and JSON summaries of DICOM files. Both files contain PHI.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
308
452
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
309 When configuring the plugin, you'll have to provide a **Master Key** that we can also call the **Key Encryption Key (KEK)**.
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
310
452
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
311 For each file being saved, the plugin will generate a new **Data Encryption Key (DEK)**. This DEK, encrypted with the KEK will be pre-pended to the file.
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
312
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
313 If, at any point, your KEK leaks or you want to rotate your KEKs, you'll be able to use a new one to encrypt new files that are being added
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
314 and still use the old ones to decrypt data. You could then eventually start a side script to remove usages of the leaked/obsolete KEKs.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
315
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
316 To summarize:
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
317
452
aef5c8b74381 fix + link to sample
Alain Mazy <alain@mazy.be>
parents: 451
diff changeset
318 - We use `Crypto++ <https://www.cryptopp.com/>`__ to perform all encryptions.
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
319 - All keys (KEK and DEK) are AES-256 keys.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
320 - DEKs and IVs are encrypted by KEK using CTR block cipher using a null IV.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
321 - data is encrypted by DEK using GCM block cipher that will also perform integrity check on the whole file.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
322
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
323 The format of data stored on disk is therefore the following:
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
324
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
325 - **VERSION HEADER**: 2 bytes: identify the structure of the following data currently `A1`
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
326 - **MASTER KEY ID**: 4 bytes: a numerical ID of the KEK that was used to encrypt the DEK
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
327 - **EIV**: 32 bytes: IV used by DEK for data encryption; encrypted by KEK
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
328 - **EDEK**: 32 bytes: the DEK encrypted by the KEK.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
329 - **CIPHER TEXT**: variable length: the DICOM/JSON file encrypted by the DEK
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
330 - **TAG**: 16 bytes: integrity check performed on the whole encrypted file (including header, master key id, EIV and EDEK)
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
331
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
332 Configuration
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
333 ^^^^^^^^^^^^^
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
334
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
335 .. highlight:: text
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
336
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
337 AES Keys shall be 32 bytes long (256 bits) and encoded in base64. Here's a sample OpenSSL command to generate such a key::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
338
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
339 openssl rand -base64 -out /tmp/test.key 32
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
340
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
341 Each key must have a unique id that is a uint32 number.
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
342
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
343 .. highlight:: json
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
344
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
345 Here's a sample configuration file of the `StorageEncryption` section of the plugins::
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
346
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
347 {
499
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
348 "GoogleCloudStorage" : {
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
349 "StorageEncryption" : {
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
350 "Enable": true,
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
351 "MasterKey": [3, "/path/to/master.key"], // key id - path to the base64 encoded key
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
352 "PreviousMasterKeys" : [
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
353 [1, "/path/to/previous1.key"],
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
354 [2, "/path/to/previous2.key"]
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
355 ],
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
356 "MaxConcurrentInputSize" : 1024 // size in MB
d255e02eb89d updated object-storage doc for 1.0.0
Alain Mazy <alain@mazy.be>
parents: 464
diff changeset
357 }
451
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
358 }
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
359 }
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
360
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
361 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
362 than 2 times the input, we want to limit the number of threads doing concurrent processing according
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
363 to the available memory instead of the number of concurrent threads. Therefore, if you're currently
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
364 ingesting small files, you can have a lot of thread working together while, if you're ingesting large
938206110483 added object storage
Alain Mazy <alain@mazy.be>
parents:
diff changeset
365 files, threads might have to wait before receiving a "slot" to access the encryption module.