Mercurial > hg > orthanc-book

comparison Sphinx/source/plugins/object-storage.rst @ 988:afea429661c2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
S3 transfer mode
author Alain Mazy <am@osimis.io>
date Thu, 16 Nov 2023 08:37:28 +0100
parents 5df222ddd7d1
children 38909a00964c
comparison
equal deleted inserted replaced
987:759f7d5aca94 988:afea429661c2
44 44
45 Although you are obviously free to compile these plugins by 45 Although you are obviously free to compile these plugins by
46 yourself (instructions are given below), purchasing such support 46 yourself (instructions are given below), purchasing such support
47 contracts makes the Orthanc project sustainable in the long term, to 47 contracts makes the Orthanc project sustainable in the long term, to
48 the benefit of the worldwide community of medical imaging. 48 the benefit of the worldwide community of medical imaging.
49
50
51 Compilation
52 -----------
53
54 .. highlight:: text
55
56 The procedure to compile the plugins is quite similar of that for the
57 :ref:`core of Orthanc <compiling>` although they usually require
58 some prerequisites. The documented procedure has been tested only
59 on a Debian Buster machine.
60
61 The compilation of each plugin produces a shared library that contains
62 the plugin.
63
64
65 AWS S3 plugin
66 ^^^^^^^^^^^^^
67
68 Prerequisites: Compile the AWS C++ SDK::
69
70 $ mkdir ~/aws
71 $ cd ~/aws
72 $ git clone https://github.com/aws/aws-sdk-cpp.git
73 $
74 $ mkdir -p ~/aws/builds/aws-sdk-cpp
75 $ cd ~/aws/builds/aws-sdk-cpp
76 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp
77 $ make -j 4
78 $ make install
79
80 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
81
82 $ ./vcpkg install cryptopp
83
84 Compile::
85
86 $ mkdir -p build/aws
87 $ cd build/aws
88 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws
89
90
91 **NB:** If you don't want to use vcpkg, you can use the following
92 command (this syntax is not compatible with Ninja yet)::
93
94 $ cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_VCPKG_PACKAGES=OFF -DUSE_SYSTEM_GOOGLE_TEST=OFF ../../orthanc-object-storage/Aws
95 $ make
96
97 Crypto++ must be installed (on Ubuntu, run ``sudo apt install libcrypto++-dev``).
98
99
100 Azure Blob Storage plugin
101 ^^^^^^^^^^^^^^^^^^^^^^^^^
102
103 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
104
105 $ ./vcpkg install cryptopp
106 $ ./vcpkg install azure-storage-cpp
107
108
109 Compile::
110
111 $ mkdir -p build/azure
112 $ cd build/azure
113 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure
114
115 Google Storage plugin
116 ^^^^^^^^^^^^^^^^^^^^^
117
118 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
119
120 $ ./vcpkg install cryptopp
121 $ ./vcpkg install google-cloud-cpp
122
123 Compile::
124
125 $ mkdir -p build/google
126 $ cd build/google
127 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google
128 49
129 50
130 Configuration 51 Configuration
131 ------------- 52 -------------
132 53
149 "MigrationFromFileSystemEnabled": false, // optional: see below 70 "MigrationFromFileSystemEnabled": false, // optional: see below
150 "StorageStructure": "flat", // optional: see below 71 "StorageStructure": "flat", // optional: see below
151 "EnableLegacyUnknownFiles": true, // optional: see below 72 "EnableLegacyUnknownFiles": true, // optional: see below
152 "VirtualAddressing": true, // optional: see the section related to MinIO 73 "VirtualAddressing": true, // optional: see the section related to MinIO
153 "StorageEncryption" : {}, // optional: see the section related to encryption 74 "StorageEncryption" : {}, // optional: see the section related to encryption
154 "HybridMode": "Disabled" // optional: see the section related to Hybrid storage 75 "HybridMode": "Disabled", // optional: see the section related to Hybrid storage
76 "UseTransferMode": true // optional: see below (available from version 2.3.0)
155 } 77 }
156 78
157 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway:: 79 The **EndPoint** configuration is used when accessing an S3 compatible cloud provider. I.e. here is a configuration to store data on Scaleway::
158 80
159 "AwsS3Storage" : { 81 "AwsS3Storage" : {
161 "Region": "fr-par", 83 "Region": "fr-par",
162 "AccessKey": "XXX", 84 "AccessKey": "XXX",
163 "SecretKey": "YYY", 85 "SecretKey": "YYY",
164 "Endpoint": "s3.fr-par.scw.cloud" 86 "Endpoint": "s3.fr-par.scw.cloud"
165 } 87 }
88
89
90 The **UseTransferMode** configuration is used to select the `Transfer Manager <https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html>`__ mode in the AWS SDK client.
91 This option was introduced in version 2.3.0 and is the default one. If set to false, the default mode
92 is used.
166 93
167 94
168 .. _minio: 95 .. _minio:
169 96
170 Emulation of AWS S3 using MinIO 97 Emulation of AWS S3 using MinIO
440 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more 367 **MaxConcurrentInputSize**: Since the memory used during encryption/decryption can grow up to a bit more
441 than 2 times the input, we want to limit the number of threads doing concurrent processing according 368 than 2 times the input, we want to limit the number of threads doing concurrent processing according
442 to the available memory instead of the number of concurrent threads. Therefore, if you're currently 369 to the available memory instead of the number of concurrent threads. Therefore, if you're currently
443 ingesting small files, you can have a lot of thread working together while, if you're ingesting large 370 ingesting small files, you can have a lot of thread working together while, if you're ingesting large
444 files, threads might have to wait before receiving a "slot" to access the encryption module. 371 files, threads might have to wait before receiving a "slot" to access the encryption module.
372
373
374 Compilation
375 -----------
376
377 .. highlight:: text
378
379 The procedure to compile the plugins is quite similar of that for the
380 :ref:`core of Orthanc <compiling>` although they usually require
381 some prerequisites. The documented procedure has been tested only
382 on a Debian Buster machine.
383
384 The compilation of each plugin produces a shared library that contains
385 the plugin.
386
387
388 AWS S3 plugin
389 ^^^^^^^^^^^^^
390
391 Prerequisites: Compile the AWS C++ SDK::
392
393 $ mkdir ~/aws
394 $ cd ~/aws
395 $ git clone https://github.com/aws/aws-sdk-cpp.git
396 $
397 $ mkdir -p ~/aws/builds/aws-sdk-cpp
398 $ cd ~/aws/builds/aws-sdk-cpp
399 $ cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp
400 $ make -j 4
401 $ make install
402
403 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
404
405 $ ./vcpkg install cryptopp
406
407 Compile::
408
409 $ mkdir -p build/aws
410 $ cd build/aws
411 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws
412
413
414 **NB:** If you don't want to use vcpkg, you can use the following
415 command (this syntax is not compatible with Ninja yet)::
416
417 $ cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_VCPKG_PACKAGES=OFF -DUSE_SYSTEM_GOOGLE_TEST=OFF ../../orthanc-object-storage/Aws
418 $ make
419
420 Crypto++ must be installed (on Ubuntu, run ``sudo apt install libcrypto++-dev``).
421
422
423 Azure Blob Storage plugin
424 ^^^^^^^^^^^^^^^^^^^^^^^^^
425
426 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
427
428 $ ./vcpkg install cryptopp
429 $ ./vcpkg install azure-storage-cpp
430
431
432 Compile::
433
434 $ mkdir -p build/azure
435 $ cd build/azure
436 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure
437
438 Google Storage plugin
439 ^^^^^^^^^^^^^^^^^^^^^
440
441 Prerequisites: Install `vcpkg <https://github.com/Microsoft/vcpkg>`__ dependencies::
442
443 $ ./vcpkg install cryptopp
444 $ ./vcpkg install google-cloud-cpp
445
446 Compile::
447
448 $ mkdir -p build/google
449 $ cd build/google
450 $ cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/google