Mercurial > hg > orthanc-object-storage
annotate README.md @ 165:76f702443ced
integration 2.1.2: back to mainline
| author | Sebastien Jodogne <s.jodogne@gmail.com> |
|---|---|
| date | Tue, 25 Jun 2024 12:04:38 +0200 |
| parents | 6aa3ec905f58 |
| children |
| rev | line source |
|---|---|
| 1 | 1 # README # |
| 2 | |
| 3 Orthanc object-storages plugin for main cloud providers (Google/Azure/AWS) | |
| 4 | |
|
124
6aa3ec905f58
migration to UCLouvain servers
Sebastien Jodogne <s.jodogne@gmail.com>
parents:
101
diff
changeset
|
5 Check the [Orthanc book](https://orthanc.uclouvain.be/book/plugins/object-storage.html) for complete documentation. |
| 1 | 6 |
| 5 | 7 ## info for developers ## |
| 8 | |
| 9 Here's a sample configuration file of the `StorageEncryption` section of the plugins: | |
| 10 | |
| 11 ``` | |
| 12 { | |
| 15 | 13 "GoogleCloudStorage" : { |
| 14 "StorageEncryption" : { | |
| 15 "Enable": true, | |
| 16 "MasterKey": [3, "/path/to/master.key"], // key id - path to the base64 encoded key | |
| 17 "PreviousMasterKeys" : [ | |
| 18 [ 1, "/path/to/previous1.key"], | |
| 19 [ 2, "/path/to/previous2.key"] | |
| 20 ], | |
| 21 "MaxConcurrentInputSize" : 1024 // size in MB | |
| 22 } | |
| 5 | 23 } |
| 24 } | |
| 25 ``` | |
| 26 | |
| 27 ### Compile Google plugin ### | |
| 28 | |
|
101
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
29 On Linux, with vcpkg version `2023.06.20`: |
|
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
30 |
|
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
31 * `./vcpkg install google-cloud-cpp[storage]` |
| 5 | 32 * `./vcpkg install cryptopp` |
| 33 * `hg clone ...` | |
| 34 * `mkdir -p build/google` | |
| 35 * `cd build/google` | |
| 15 | 36 * `cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Google` |
| 5 | 37 |
| 38 ### Google plugin configuration ### | |
| 39 | |
| 40 ``` | |
| 41 "GoogleCloudStorage" : { | |
| 15 | 42 "ServiceAccountFile" : "/.../googleServiceAccountFile.json", |
| 43 "BucketName": "test-orthanc-storage-plugin", | |
| 20 | 44 "RootPath": "", // optional: folder in which files are stored (ex: my/path/to/myfolder) |
| 15 | 45 "StorageEncryption" : {...}, |
| 46 "StorageStructure" : "flat", | |
| 47 "MigrationFromFileSystemEnabled" : false | |
| 5 | 48 } |
| 49 | |
| 50 ``` | |
| 51 | |
| 52 ## Azure Blob Storage plugin ## | |
| 53 | |
| 54 ### Prerequisites ### | |
| 55 | |
| 56 * Install [vcpkg](https://github.com/Microsoft/vcpkg) | |
| 57 | |
| 58 ### Compile Azure plugin ### | |
| 59 | |
|
101
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
60 On Linux, with vcpkg version `2023.06.20`: |
| 34 | 61 |
|
101
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
62 * `./vcpkg install azure-core-cpp azure-storage-blobs-cpp` |
| 34 | 63 * `./vcpkg install cryptopp` |
| 5 | 64 * `hg clone ...` |
| 65 * `mkdir -p build/azure` | |
| 66 * `cd build/azure` | |
| 67 * `cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Azure` | |
| 68 | |
|
101
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
69 On Windows, with vcpkg version `2023.06.20` : |
| 34 | 70 |
|
101
57be7094d6c5
updated Google plugin for latest SDK
Alain Mazy <am@osimis.io>
parents:
90
diff
changeset
|
71 * `.\vcpkg.exe install azure-storage-blobs-cpp:x64-windows-static` |
| 34 | 72 * `.\vcpkg.exe install cryptopp:x64-windows-static` |
| 73 * `hg clone ...` | |
| 74 * `mkdir -p build/azure` | |
| 75 * `cd build/azure` | |
| 76 * `cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows-static -DSTATIC_BUILD=ON -DCMAKE_BUILD_TYPE="Release" ../../orthanc-object-storage/Azure` | |
| 77 * `cmake --build . --config Release` | |
| 78 | |
| 79 | |
| 5 | 80 ### Azure plugin configuration ### |
| 81 | |
| 82 ``` | |
| 83 "AzureBlobStorage" : { | |
| 84 "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=xxxxxxxxx;AccountKey=yyyyyyyy===;EndpointSuffix=core.windows.net", | |
| 15 | 85 "ContainerName" : "test-orthanc-storage-plugin", |
| 50 | 86 "CreateContainerIfNotExists": true, // available from version 1.2.0 |
| 20 | 87 "RootPath": "", // optional: folder in which files are stored (ex: my/path/to/myfolder) |
| 15 | 88 "StorageEncryption" : {...}, |
| 89 "StorageStructure" : "flat", | |
| 90 "MigrationFromFileSystemEnabled" : false | |
| 5 | 91 } |
| 92 ``` | |
| 93 | |
| 94 ## AWS S3 Storage plugin ## | |
| 95 | |
| 96 ### Prerequisites ### | |
| 97 | |
| 98 * Install [vcpkg](https://github.com/Microsoft/vcpkg) | |
| 99 | |
| 100 * compile the AWS C++ SDK | |
| 101 | |
| 102 ``` | |
| 103 | |
| 104 mkdir ~/aws | |
| 105 cd ~/aws | |
| 106 git clone https://github.com/aws/aws-sdk-cpp.git | |
| 107 | |
| 108 mkdir -p ~/aws/builds/aws-sdk-cpp | |
| 109 cd ~/aws/builds/aws-sdk-cpp | |
| 110 cmake -DBUILD_ONLY="s3;transfer" ~/aws/aws-sdk-cpp | |
| 111 make -j 4 | |
| 112 make install | |
| 113 ``` | |
| 114 | |
| 115 ### Compile AWS S3 plugin ### | |
| 116 | |
| 117 * `./vcpkg install cryptopp` | |
| 118 * `hg clone ...` | |
| 119 * `mkdir -p build/aws` | |
| 120 * `cd build/aws` | |
| 121 * `cmake -DCMAKE_TOOLCHAIN_FILE=[vcpkg root]\scripts\buildsystems\vcpkg.cmake ../../orthanc-object-storage/Aws` | |
| 122 | |
| 54 | 123 ### AWS S3 plugin configuration ### |
| 5 | 124 |
| 125 ``` | |
| 126 "AwsS3Storage" : { | |
| 127 "BucketName": "test-orthanc-s3-plugin", | |
| 128 "Region" : "eu-central-1", | |
| 54 | 129 "AccessKey" : "AKXXX", // optional: if not specified, the plugin will use the default credentials manager (from version 1.3.0) |
| 130 "SecretKey" : "RhYYYY", // optional: if not specified, the plugin will use the default credentials manager (from version 1.3.0) | |
|
6
393fcf337462
AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout
Alain Mazy
parents:
5
diff
changeset
|
131 "Endpoint": "", // optional: custom endpoint |
|
393fcf337462
AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout
Alain Mazy
parents:
5
diff
changeset
|
132 "ConnectionTimeout": 30, // optional: connection timeout in seconds |
| 15 | 133 "RequestTimeout": 1200, // optional: request timeout in seconds (max time to upload/download a file) |
| 20 | 134 "RootPath": "", // optional: folder in which files are stored (ex: my/path/to/myfolder) |
| 135 "StorageEncryption" : {...}, // optional | |
| 136 "StorageStructure" : "flat", // optional | |
| 77 | 137 "MigrationFromFileSystemEnabled" : false, // optional (deprecated, is now equivalent to "HybridMode": "WriteToObjectStorage") |
| 138 "HybridMode": "WriteToDisk" // "WriteToDisk", "WriteToObjectStorage", "Disabled" | |
| 5 | 139 } |
| 77 | 140 ``` |
| 141 | |
| 142 ### Testing the S3 plugin with minio | |
| 143 | |
| 144 ``` | |
| 145 docker run -p 9000:9000 -p 9001:9001 -e MINIO_REGION=eu-west-1 -e MINIO_ROOT_USER=minio -e MINIO_ROOT_PASSWORD=miniopwd minio/minio server /data --console-address ":9001" | |
| 146 ``` | |
| 147 | |
| 148 config file: | |
| 149 ``` | |
| 150 "AwsS3Storage" : { | |
| 151 "BucketName": "orthanc", | |
| 152 "Region": "eu-west-1", | |
| 153 "Endpoint": "http://127.0.0.1:9000/", | |
| 154 "AccessKey": "minio", | |
| 155 "SecretKey": "miniopwd", | |
| 156 "VirtualAddressing": false | |
| 157 | |
| 158 // "StorageEncryption" : { | |
| 159 // "Enable": true, | |
| 160 // "MasterKey": [1, "/home/test/encryption/test.key"], | |
| 161 // "MaxConcurrentInputSize": 1024, | |
| 162 // "Verbose": true | |
| 163 // } // optional: see the section related to encryption | |
| 164 } | |
| 165 | |
| 166 ``` | |
| 167 | |
| 168 Test the hybrid mode | |
| 169 - start in "HybridMode": "WriteToFileSystem", | |
| 170 - upload instances 1 & 2 | |
| 171 - restart in "HybridMode": "WriteToObjectStorage", | |
| 172 - check that you can read instance 1 and that you can delete it | |
| 173 - upload instances 3 & 4 | |
| 174 - restart in "HybridMode": "WriteToFileSystem", | |
| 175 - check that you can read instance 3 and that you can delete it | |
| 176 - final check: | |
| 177 - there should be only one file in the disk storage | |
| 178 - there should be only one file in the S3 bucket | |
| 179 | |
|
83
431ab61b5760
/move-storage when HybridMode is enabled
Alain Mazy <am@osimis.io>
parents:
77
diff
changeset
|
180 test moving a study to file-system storage |
|
431ab61b5760
/move-storage when HybridMode is enabled
Alain Mazy <am@osimis.io>
parents:
77
diff
changeset
|
181 curl http://localhost:8043/move-storage -d '{"Resources": ["737c0c8d-ea890b4d-e36a43bb-fb8c8d41-aa0ed0a8"], "TargetStorage" : "file-system"}' |
|
431ab61b5760
/move-storage when HybridMode is enabled
Alain Mazy <am@osimis.io>
parents:
77
diff
changeset
|
182 curl http://localhost:8043/move-storage -d '{"Resources": ["737c0c8d-ea890b4d-e36a43bb-fb8c8d41-aa0ed0a8"], "TargetStorage" : "object-storage"}' |