Mercurial > hg > orthanc-book
comparison Sphinx/source/plugins/object-storage.rst @ 988:afea429661c2
S3 transfer mode
author | Alain Mazy <am@osimis.io> |
---|---|
date | Thu, 16 Nov 2023 08:37:28 +0100 |
parents | 5df222ddd7d1 |
children | 38909a00964c |
comparison
equal
deleted
inserted
replaced
987:759f7d5aca94 | 988:afea429661c2 |
---|---|
44 | 44 |
45 Although you are obviously free to compile these plugins by | 45 Although you are obviously free to compile these plugins by |
46 yourself (instructions are given below), purchasing such support | 46 yourself (instructions are given below), purchasing such support |
47 contracts makes the Orthanc project sustainable in the long term, to | 47 contracts makes the Orthanc project sustainable in the long term, to |
48 the benefit of the worldwide community of medical imaging. | 48 the benefit of the worldwide community of medical imaging. |
49 | |
50 | |
51 Compilation | |
52 ----------- | |
53 | |
54 .. highlight:: text | |
55 | |
56 The procedure to compile the plugins is quite similar of that for the | |
57 :ref:`core of Orthanc <compiling>` although they usually require | |
58 some prerequisites. The documented procedure has been tested only | |
59 on a Debian Buster machine. | |
60 | |
61 The compilation of each plugin produces a shared library that contains | |
62 the plugin. | |
63 | |
64 | |
65 AWS S3 plugin | |
66 ^^^^^^^^^^^^^ | |
67 | |
68 Prerequisites: Compile the AWS C++ SDK:: | |
69 | |
70 $ mkdir ~/aws | |
71 $ cd ~/aws | |
72 $ git clone https://github.com/aws/aws-sdk-cpp.git | |
73 $ | |
74 $ mkdir -p ~/aws/builds/aws-sdk-cpp | |
75 $ cd ~/aws/builds/aws-sdk-cpp | |
76 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp | |
77 $ make -j 4 | |
78 $ make install | |
79 | |
80 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
81 | |
82 $ ./vcpkg install cryptopp | |
83 | |
84 Compile:: | |
85 | |
86 $ mkdir -p build/aws | |
87 $ cd build/aws | |
88 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws | |
89 | |
90 | |
91 **NB:** If you don't want to use vcpkg, you can use the following | |
92 command (this syntax is not compatible with Ninja yet):: | |
93 | |
94 $ cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_VCPKG_PACKAGES=OFF -DUSE_SYSTEM_GOOGLE_TEST=OFF ../../orthanc-object-storage/Aws | |
95 $ make | |
96 | |
97 Crypto++ must be installed (on Ubuntu, run ``sudo apt install libcrypto++-dev``). | |
98 | |
99 | |
100 Azure Blob Storage plugin | |
101 ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
102 | |
103 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
104 | |
105 $ ./vcpkg install cryptopp | |
106 $ ./vcpkg install azure-storage-cpp | |
107 | |
108 | |
109 Compile:: | |
110 | |
111 $ mkdir -p build/azure | |
112 $ cd build/azure | |
113 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure | |
114 | |
115 Google Storage plugin | |
116 ^^^^^^^^^^^^^^^^^^^^^ | |
117 | |
118 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
119 | |
120 $ ./vcpkg install cryptopp | |
121 $ ./vcpkg install google-cloud-cpp | |
122 | |
123 Compile:: | |
124 | |
125 $ mkdir -p build/google | |
126 $ cd build/google | |
127 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google | |
128 | 49 |
129 | 50 |
130 Configuration | 51 Configuration |
131 ------------- | 52 ------------- |
132 | 53 |
149 "MigrationFromFileSystemEnabled": false, // optional: see below | 70 "MigrationFromFileSystemEnabled": false, // optional: see below |
150 "StorageStructure": "flat", // optional: see below | 71 "StorageStructure": "flat", // optional: see below |
151 "EnableLegacyUnknownFiles": true, // optional: see below | 72 "EnableLegacyUnknownFiles": true, // optional: see below |
152 "VirtualAddressing": true, // optional: see the section related to MinIO | 73 "VirtualAddressing": true, // optional: see the section related to MinIO |
153 "StorageEncryption" : {}, // optional: see the section related to encryption | 74 "StorageEncryption" : {}, // optional: see the section related to encryption |
154 "HybridMode": "Disabled" // optional: see the section related to Hybrid storage | 75 "HybridMode": "Disabled", // optional: see the section related to Hybrid storage |
76 "UseTransferMode": true // optional: see below (available from version 2.3.0) | |
155 } | 77 } |
156 | 78 |
157 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway:: | 79 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway:: |
158 | 80 |
159 "AwsS3Storage" : { | 81 "AwsS3Storage" : { |
161 "Region": "fr-par", | 83 "Region": "fr-par", |
162 "AccessKey": "XXX", | 84 "AccessKey": "XXX", |
163 "SecretKey": "YYY", | 85 "SecretKey": "YYY", |
164 "Endpoint": "s3.fr-par.scw.cloud" | 86 "Endpoint": "s3.fr-par.scw.cloud" |
165 } | 87 } |
88 | |
89 | |
90 The **UseTransferMode** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client. | |
91 This option was introduced in version 2.3.0 and is the default one. If set to false, the default mode | |
92 is used. | |
166 | 93 |
167 | 94 |
168 .. _minio: | 95 .. _minio: |
169 | 96 |
170 Emulation of AWS S3 using MinIO | 97 Emulation of AWS S3 using MinIO |
440 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more | 367 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more |
441 than 2 times the input, we want to limit the number of threads doing concurrent processing according | 368 than 2 times the input, we want to limit the number of threads doing concurrent processing according |
442 to the available memory instead of the number of concurrent threads. Therefore, if you're currently | 369 to the available memory instead of the number of concurrent threads. Therefore, if you're currently |
443 ingesting small files, you can have a lot of thread working together while, if you're ingesting large | 370 ingesting small files, you can have a lot of thread working together while, if you're ingesting large |
444 files, threads might have to wait before receiving a "slot" to access the encryption module. | 371 files, threads might have to wait before receiving a "slot" to access the encryption module. |
372 | |
373 | |
374 Compilation | |
375 ----------- | |
376 | |
377 .. highlight:: text | |
378 | |
379 The procedure to compile the plugins is quite similar of that for the | |
380 :ref:`core of Orthanc <compiling>` although they usually require | |
381 some prerequisites. The documented procedure has been tested only | |
382 on a Debian Buster machine. | |
383 | |
384 The compilation of each plugin produces a shared library that contains | |
385 the plugin. | |
386 | |
387 | |
388 AWS S3 plugin | |
389 ^^^^^^^^^^^^^ | |
390 | |
391 Prerequisites: Compile the AWS C++ SDK:: | |
392 | |
393 $ mkdir ~/aws | |
394 $ cd ~/aws | |
395 $ git clone https://github.com/aws/aws-sdk-cpp.git | |
396 $ | |
397 $ mkdir -p ~/aws/builds/aws-sdk-cpp | |
398 $ cd ~/aws/builds/aws-sdk-cpp | |
399 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp | |
400 $ make -j 4 | |
401 $ make install | |
402 | |
403 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
404 | |
405 $ ./vcpkg install cryptopp | |
406 | |
407 Compile:: | |
408 | |
409 $ mkdir -p build/aws | |
410 $ cd build/aws | |
411 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws | |
412 | |
413 | |
414 **NB:** If you don't want to use vcpkg, you can use the following | |
415 command (this syntax is not compatible with Ninja yet):: | |
416 | |
417 $ cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_VCPKG_PACKAGES=OFF -DUSE_SYSTEM_GOOGLE_TEST=OFF ../../orthanc-object-storage/Aws | |
418 $ make | |
419 | |
420 Crypto++ must be installed (on Ubuntu, run ``sudo apt install libcrypto++-dev``). | |
421 | |
422 | |
423 Azure Blob Storage plugin | |
424 ^^^^^^^^^^^^^^^^^^^^^^^^^ | |
425 | |
426 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
427 | |
428 $ ./vcpkg install cryptopp | |
429 $ ./vcpkg install azure-storage-cpp | |
430 | |
431 | |
432 Compile:: | |
433 | |
434 $ mkdir -p build/azure | |
435 $ cd build/azure | |
436 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure | |
437 | |
438 Google Storage plugin | |
439 ^^^^^^^^^^^^^^^^^^^^^ | |
440 | |
441 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies:: | |
442 | |
443 $ ./vcpkg install cryptopp | |
444 $ ./vcpkg install google-cloud-cpp | |
445 | |
446 Compile:: | |
447 | |
448 $ mkdir -p build/google | |
449 $ cd build/google | |
450 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google |