orthanc-object-storage: Aws/AwsS3StoragePlugin.cpp annotate

annotate Aws/AwsS3StoragePlugin.cpp @ 197:df0b6998fd27

integration 2.3.0: back to mainline

author	Sebastien Jodogne <s.jodogne@gmail.com>
date	Tue, 25 Jun 2024 12:38:31 +0200
parents	6dd8bb916573 addccb7ad900
children

rev	line source
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	1 /**
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	2 * Cloud storage plugins for Orthanc
145 3c7e0374f28e updated copyright, as Orthanc Team now replaces Osimis Sebastien Jodogne <s.jodogne@gmail.com> parents: 141 diff changeset	3 * Copyright (C) 2020-2023 Osimis S.A., Belgium
3c7e0374f28e updated copyright, as Orthanc Team now replaces Osimis Sebastien Jodogne <s.jodogne@gmail.com> parents: 141 diff changeset	4 * Copyright (C) 2024-2024 Orthanc Team SRL, Belgium
3c7e0374f28e updated copyright, as Orthanc Team now replaces Osimis Sebastien Jodogne <s.jodogne@gmail.com> parents: 141 diff changeset	5 * Copyright (C) 2021-2024 Sebastien Jodogne, ICTEAM UCLouvain, Belgium
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	6 *
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	7 * This program is free software: you can redistribute it and/or
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	8 * modify it under the terms of the GNU Affero General Public License
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	9 * as published by the Free Software Foundation, either version 3 of
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	10 * the License, or (at your option) any later version.
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	11 *
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	12 * This program is distributed in the hope that it will be useful, but
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	13 * WITHOUT ANY WARRANTY; without even the implied warranty of
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	14 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	15 * Affero General Public License for more details.
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	16 *
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	17 * You should have received a copy of the GNU Affero General Public License
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	18 * along with this program. If not, see <http://www.gnu.org/licenses/>.
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	19 **/
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	20
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	21 #include "AwsS3StoragePlugin.h"
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	22 #include <Logging.h>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	23
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	24 #include <aws/core/Aws.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	25 #include <aws/s3/S3Client.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	26 #include <aws/s3/model/PutObjectRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	27 #include <aws/s3/model/GetObjectRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	28 #include <aws/s3/model/ListObjectsRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	29 #include <aws/s3/model/DeleteObjectRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	30 #include <aws/core/auth/AWSCredentialsProvider.h>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	31 #include <aws/core/utils/HashingUtils.h>
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	32 #include <aws/core/utils/logging/DefaultLogSystem.h>
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	33 #include <aws/core/utils/logging/DefaultCRTLogSystem.h>
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	34 #include <aws/core/utils/logging/AWSLogging.h>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	35 #include <aws/core/utils/memory/stl/AWSStreamFwd.h>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	36 #include <aws/core/utils/memory/stl/AWSStringStream.h>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	37 #include <aws/core/utils/memory/AWSMemory.h>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	38 #include <aws/core/utils/stream/PreallocatedStreamBuf.h>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	39 #include <aws/core/utils/StringUtils.h>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	40 #include <aws/transfer/TransferManager.h>
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	41 #include <aws/crt/Api.h>
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	42 #include <iostream>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	43 #include <fstream>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	44
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	45 #include <boost/lexical_cast.hpp>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	46 #include <boost/interprocess/streams/bufferstream.hpp>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	47 #include <iostream>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	48 #include <fstream>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	49
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	50 const char* ALLOCATION_TAG = "OrthancS3";
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	51
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	52 class AwsS3StoragePlugin : public BaseStorage
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	53 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	54 public:
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	55
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	56 std::string bucketName_;
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	57 bool storageContainsUnknownFiles_;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	58 bool useTransferManager_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	59 std::shared_ptr<Aws::S3::S3Client> client_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	60 std::shared_ptr<Aws::Utils::Threading::Executor> executor_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	61 std::shared_ptr<Aws::Transfer::TransferManager> transferManager_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	62
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	63 public:
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	64
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	65 AwsS3StoragePlugin(const std::string& nameForLogs, std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, bool enableLegacyStorageStructure, bool storageContainsUnknownFiles, bool useTransferManager, unsigned int transferThreadPoolSize, unsigned int transferBufferSizeMB);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	66
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	67 virtual ~AwsS3StoragePlugin();
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	68
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	69 virtual IWriter* GetWriterForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled) ORTHANC_OVERRIDE;
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	70 virtual IReader* GetReaderForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled) ORTHANC_OVERRIDE;
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	71 virtual void DeleteObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled) ORTHANC_OVERRIDE;
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	72
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	73 virtual bool HasFileExists() ORTHANC_OVERRIDE
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	74 {
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	75 return false;
6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	76 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	77 };
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	78
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	79
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	80 class DirectWriter : public IStorage::IWriter
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	81 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	82 std::string path_;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	83 std::shared_ptr<Aws::S3::S3Client> client_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	84 std::string bucketName_;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	85
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	86 public:
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	87 DirectWriter(std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, const std::string& path)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	88 : path_(path),
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	89 client_(client),
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	90 bucketName_(bucketName)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	91 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	92 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	93
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	94 virtual ~DirectWriter()
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	95 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	96 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	97
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	98 virtual void Write(const char* data, size_t size) ORTHANC_OVERRIDE
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	99 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	100 Aws::S3::Model::PutObjectRequest putObjectRequest;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	101
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	102 putObjectRequest.SetBucket(bucketName_.c_str());
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	103 putObjectRequest.SetKey(path_.c_str());
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	104
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	105 std::shared_ptr<Aws::StringStream> stream = Aws::MakeShared<Aws::StringStream>(ALLOCATION_TAG, std::ios_base::in \| std::ios_base::binary);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	106
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	107 stream->rdbuf()->pubsetbuf(const_cast<char*>(data), size);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	108 stream->rdbuf()->pubseekpos(size);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	109 stream->seekg(0);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	110
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	111 putObjectRequest.SetBody(stream);
74 a25b4140e7e4 AWS: added the content MD5 in the request when writing Alain Mazy <am@osimis.io> parents: 70 diff changeset	112 putObjectRequest.SetContentMD5(Aws::Utils::HashingUtils::Base64Encode(Aws::Utils::HashingUtils::CalculateMD5(*stream)));
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	113
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	114 auto result = client_->PutObject(putObjectRequest);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	115
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	116 if (!result.IsSuccess())
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	117 {
62 d10696f94959 more detailed AWS errors Alain Mazy <am@osimis.io> parents: 57 diff changeset	118 throw StoragePluginException(std::string("error while writing file ") + path_ + ": response code = " + boost::lexical_cast<std::string>((int)result.GetError().GetResponseCode()) + " " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	119 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	120 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	121 };
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	122
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	123
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	124 class DirectReader : public IStorage::IReader
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	125 {
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	126 protected:
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	127 std::shared_ptr<Aws::S3::S3Client> client_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	128 std::string bucketName_;
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	129 std::list<std::string> paths_;
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	130 std::string uuid_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	131
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	132 public:
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	133 DirectReader(std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, const std::list<std::string>& paths, const char* uuid)
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	134 : client_(client),
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	135 bucketName_(bucketName),
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	136 paths_(paths),
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	137 uuid_(uuid)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	138 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	139 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	140
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	141 virtual ~DirectReader()
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	142 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	143
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	144 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	145
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	146 virtual size_t GetSize() ORTHANC_OVERRIDE
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	147 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	148 std::string firstExceptionMessage;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	149
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	150 for (auto& path: paths_)
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	151 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	152 try
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	153 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	154 return _GetSize(path);
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	155 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	156 catch (StoragePluginException& ex)
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	157 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	158 if (firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	159 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	160 firstExceptionMessage = ex.what();
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	161 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	162 //ignore to retry
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	163 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	164 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	165 throw StoragePluginException(firstExceptionMessage);
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	166 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	167
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	168 virtual void ReadWhole(char* data, size_t size) ORTHANC_OVERRIDE
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	169 {
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	170 _Read(data, size, 0, false);
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	171 }
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	172
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	173 virtual void ReadRange(char* data, size_t size, size_t fromOffset) ORTHANC_OVERRIDE
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	174 {
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	175 _Read(data, size, fromOffset, true);
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	176 }
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	177
70 16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	178 private:
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	179
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	180 size_t _GetSize(const std::string& path)
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	181 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	182 Aws::S3::Model::ListObjectsRequest listObjectRequest;
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	183 listObjectRequest.SetBucket(bucketName_.c_str());
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	184 listObjectRequest.SetPrefix(path.c_str());
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	185
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	186 auto result = client_->ListObjects(listObjectRequest);
70 16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	187
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	188 if (result.IsSuccess())
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	189 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	190 Aws::Vector<Aws::S3::Model::Object> objectList =
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	191 result.GetResult().GetContents();
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	192
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	193 if (objectList.size() == 1)
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	194 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	195 return objectList[0].GetSize();
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	196 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	197 else if (objectList.size() > 1)
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	198 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	199 throw StoragePluginException(std::string("error while reading file ") + path + ": multiple objet with same name !");
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	200 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	201 throw StoragePluginException(std::string("error while reading file ") + path + ": object not found !");
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	202 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	203 else
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	204 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	205 throw StoragePluginException(std::string("error while reading file ") + path + ": " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str());
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	206 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	207 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	208
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	209
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	210 void _Read(char* data, size_t size, size_t fromOffset, bool useRange)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	211 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	212 std::string firstExceptionMessage;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	213
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	214 for (auto& path: paths_)
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	215 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	216 try
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	217 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	218 return __Read(path, data, size, fromOffset, useRange);
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	219 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	220 catch (StoragePluginException& ex)
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	221 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	222 if (firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	223 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	224 firstExceptionMessage = ex.what();
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	225 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	226 //ignore to retry
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	227 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	228 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	229 throw StoragePluginException(firstExceptionMessage);
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	230 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	231
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	232 void __Read(const std::string& path, char* data, size_t size, size_t fromOffset, bool useRange)
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	233 {
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	234 Aws::S3::Model::GetObjectRequest getObjectRequest;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	235 getObjectRequest.SetBucket(bucketName_.c_str());
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	236 getObjectRequest.SetKey(path.c_str());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	237
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	238 if (useRange)
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	239 {
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	240 // https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	241 std::string range = std::string("bytes=") + boost::lexical_cast<std::string>(fromOffset) + "-" + boost::lexical_cast<std::string>(fromOffset + size -1);
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	242 getObjectRequest.SetRange(range.c_str());
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	243 }
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	244
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	245 getObjectRequest.SetResponseStreamFactory(
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	246 [data, size]()
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	247 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	248 std::unique_ptr<Aws::StringStream>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	249 istream(Aws::New<Aws::StringStream>(ALLOCATION_TAG));
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	250
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	251 istream->rdbuf()->pubsetbuf(static_cast<char*>(data),
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	252 size);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	253
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	254 return istream.release();
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	255 });
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	256
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	257 // Get the object
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	258 auto result = client_->GetObject(getObjectRequest);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	259 if (result.IsSuccess())
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	260 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	261 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	262 else
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	263 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	264 throw StoragePluginException(std::string("error while reading file ") + path + ": response code = " + boost::lexical_cast<std::string>((int)result.GetError().GetResponseCode()) + " " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	265 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	266 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	267
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	268 };
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	269
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	270
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	271
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	272 class TransferWriter : public IStorage::IWriter
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	273 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	274 std::string path_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	275 std::shared_ptr<Aws::Transfer::TransferManager> transferManager_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	276 std::string bucketName_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	277
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	278 public:
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	279 TransferWriter(std::shared_ptr<Aws::Transfer::TransferManager> transferManager, const std::string& bucketName, const std::string& path)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	280 : path_(path),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	281 transferManager_(transferManager),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	282 bucketName_(bucketName)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	283 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	284 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	285
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	286 virtual ~TransferWriter()
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	287 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	288 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	289
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	290 virtual void Write(const char* data, size_t size) ORTHANC_OVERRIDE
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	291 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	292 boost::interprocess::bufferstream buffer(const_cast<char>(static_cast<const char>(data)), static_cast<size_t>(size));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	293 std::shared_ptr<Aws::IOStream> body = Aws::MakeShared<Aws::IOStream>(ALLOCATION_TAG, buffer.rdbuf());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	294
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	295 std::shared_ptr<Aws::Transfer::TransferHandle> transferHandle = transferManager_->UploadFile(body, bucketName_, path_.c_str(), "application/binary", Aws::Map<Aws::String, Aws::String>());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	296 transferHandle->WaitUntilFinished();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	297
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	298 if (transferHandle->GetStatus() != Aws::Transfer::TransferStatus::COMPLETED)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	299 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	300 throw StoragePluginException(std::string("error while writing file ") + path_ + ": response code = " + boost::lexical_cast<std::string>(static_cast<int>(transferHandle->GetLastError().GetResponseCode())) + " " + transferHandle->GetLastError().GetMessage());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	301 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	302 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	303 };
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	304
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	305
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	306 class TransferReader : public DirectReader
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	307 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	308 std::shared_ptr<Aws::Transfer::TransferManager> transferManager_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	309
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	310 public:
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	311 TransferReader(std::shared_ptr<Aws::Transfer::TransferManager> transferManager, std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, const std::list<std::string>& paths, const char* uuid)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	312 : DirectReader(client, bucketName, paths, uuid),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	313 transferManager_(transferManager)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	314 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	315 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	316
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	317 virtual ~TransferReader()
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	318 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	319
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	320 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	321
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	322 virtual void ReadWhole(char* data, size_t size) ORTHANC_OVERRIDE
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	323 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	324 std::string firstExceptionMessage;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	325
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	326 for (auto& path: paths_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	327 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	328 try
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	329 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	330 // The local variable 'streamBuffer' is captured by reference in a lambda.
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	331 // It must persist until all downloading by the 'transfer_manager' is complete.
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	332 Aws::Utils::Stream::PreallocatedStreamBuf streamBuffer(reinterpret_cast<unsigned char*>(data), size);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	333
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	334 std::shared_ptr<Aws::Transfer::TransferHandle> downloadHandler = transferManager_->DownloadFile(bucketName_, path, [&]() { //Define a lambda expression for the callback method parameter to stream back the data.
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	335 return Aws::New<Aws::IOStream>(ALLOCATION_TAG, &streamBuffer);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	336 });
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	337
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	338 downloadHandler->WaitUntilFinished();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	339
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	340 if (downloadHandler->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	341 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	342 return;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	343 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	344 else if (firstExceptionMessage.empty())
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	345 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	346 firstExceptionMessage = downloadHandler->GetLastError().GetMessage();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	347 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	348 // getObjectRequest.SetResponseStreamFactory(
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	349 // [data, size]()
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	350 // {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	351 // std::unique_ptr<Aws::StringStream>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	352 // istream(Aws::New<Aws::StringStream>(ALLOCATION_TAG));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	353
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	354 // istream->rdbuf()->pubsetbuf(static_cast<char*>(data),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	355 // size);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	356
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	357 // return istream.release();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	358 // });
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	359
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	360
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	361 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	362 catch (StoragePluginException& ex)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	363 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	364 if (firstExceptionMessage.empty())
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	365 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	366 firstExceptionMessage = ex.what();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	367 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	368 //ignore to retry
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	369 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	370 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	371 throw StoragePluginException(firstExceptionMessage);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	372 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	373
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	374 };
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	375
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	376
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	377
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	378
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	379 const char* AwsS3StoragePluginFactory::GetStoragePluginName()
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	380 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	381 return "AWS S3 Storage";
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	382 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	383
94 1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	384 const char* AwsS3StoragePluginFactory::GetStorageDescription()
1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	385 {
1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	386 return "Stores the Orthanc storage area in AWS S3";
1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	387 }
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	388
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	389 static std::unique_ptr<Aws::Crt::ApiHandle> api_;
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	390 static std::unique_ptr<Aws::SDKOptions> sdkOptions_;
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	391
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	392 #include <stdarg.h>
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	393
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	394 class AwsOrthancLogger : public Aws::Utils::Logging::LogSystemInterface
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	395 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	396 public:
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	397 virtual ~AwsOrthancLogger() {}
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	398
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	399 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	400 * Gets the currently configured log level for this logger.
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	401 */
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	402 virtual Aws::Utils::Logging::LogLevel GetLogLevel() const ORTHANC_OVERRIDE
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	403 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	404 return Aws::Utils::Logging::LogLevel::Trace;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	405 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	406 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	407 * Does a printf style output to the output stream. Don't use this, it's unsafe. See LogStream
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	408 */
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	409 virtual void Log(Aws::Utils::Logging::LogLevel logLevel, const char* tag, const char* formatStr, ...) ORTHANC_OVERRIDE
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	410 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	411 Aws::StringStream ss;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	412
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	413 va_list args;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	414 va_start(args, formatStr);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	415
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	416 va_list tmp_args; //unfortunately you cannot consume a va_list twice
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	417 va_copy(tmp_args, args); //so we have to copy it
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	418 #ifdef _WIN32
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	419 const int requiredLength = _vscprintf(formatStr, tmp_args) + 1;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	420 #else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	421 const int requiredLength = vsnprintf(nullptr, 0, formatStr, tmp_args) + 1;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	422 #endif
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	423 va_end(tmp_args);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	424
141 3e9cced85a5b fix build for MSVC 2015 Sebastien Jodogne <s.jodogne@gmail.com> parents: 119 diff changeset	425 assert(requiredLength > 0);
3e9cced85a5b fix build for MSVC 2015 Sebastien Jodogne <s.jodogne@gmail.com> parents: 119 diff changeset	426 std::string outputBuff;
3e9cced85a5b fix build for MSVC 2015 Sebastien Jodogne <s.jodogne@gmail.com> parents: 119 diff changeset	427 outputBuff.resize(requiredLength);
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	428 #ifdef _WIN32
141 3e9cced85a5b fix build for MSVC 2015 Sebastien Jodogne <s.jodogne@gmail.com> parents: 119 diff changeset	429 vsnprintf_s(&outputBuff[0], requiredLength, _TRUNCATE, formatStr, args);
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	430 #else
141 3e9cced85a5b fix build for MSVC 2015 Sebastien Jodogne <s.jodogne@gmail.com> parents: 119 diff changeset	431 vsnprintf(&outputBuff[0], requiredLength, formatStr, args);
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	432 #endif // _WIN32
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	433
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	434 if (logLevel == Aws::Utils::Logging::LogLevel::Debug \|\| logLevel == Aws::Utils::Logging::LogLevel::Trace)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	435 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	436 LOG(INFO) << outputBuff;
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	437 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	438 else if (logLevel == Aws::Utils::Logging::LogLevel::Warn)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	439 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	440 LOG(WARNING) << outputBuff;
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	441 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	442 else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	443 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	444 LOG(ERROR) << outputBuff;
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	445 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	446
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	447 va_end(args);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	448 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	449 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	450 * Writes the stream to the output stream.
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	451 */
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	452 virtual void LogStream(Aws::Utils::Logging::LogLevel logLevel, const char* tag, const Aws::OStringStream &messageStream) ORTHANC_OVERRIDE
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	453 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	454 if (logLevel == Aws::Utils::Logging::LogLevel::Debug \|\| logLevel == Aws::Utils::Logging::LogLevel::Trace)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	455 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	456 LOG(INFO) << tag << messageStream.str();
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	457 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	458 else if (logLevel == Aws::Utils::Logging::LogLevel::Warn)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	459 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	460 LOG(WARNING) << tag << messageStream.str();
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	461 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	462 else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	463 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	464 LOG(ERROR) << tag << messageStream.str();
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	465 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	466
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	467 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	468 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	469 * Writes any buffered messages to the underlying device if the logger supports buffering.
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	470 */
153 6dd8bb916573 cppcheck Sebastien Jodogne <s.jodogne@gmail.com> parents: 152 diff changeset	471 virtual void Flush() ORTHANC_OVERRIDE {}
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	472 };
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	473
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	474 IStorage* AwsS3StoragePluginFactory::CreateStorage(const std::string& nameForLogs, const OrthancPlugins::OrthancConfiguration& orthancConfig)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	475 {
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	476 if (sdkOptions_.get() != NULL)
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	477 {
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	478 throw Orthanc::OrthancException(Orthanc::ErrorCode_BadSequenceOfCalls, "Cannot initialize twice");
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	479 }
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	480
15 2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	481 bool enableLegacyStorageStructure;
70 16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	482 bool storageContainsUnknownFiles;
15 2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	483
77 80792bb9600e new HybridMode Alain Mazy <am@osimis.io> parents: 74 diff changeset	484 if (!orthancConfig.IsSection(GetConfigurationSectionName()))
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	485 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	486 LOG(WARNING) << GetStoragePluginName() << " plugin, section missing. Plugin is not enabled.";
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	487 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	488 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	489
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	490 OrthancPlugins::OrthancConfiguration pluginSection;
77 80792bb9600e new HybridMode Alain Mazy <am@osimis.io> parents: 74 diff changeset	491 orthancConfig.GetSection(pluginSection, GetConfigurationSectionName());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	492
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	493 if (!BaseStorage::ReadCommonConfiguration(enableLegacyStorageStructure, storageContainsUnknownFiles, pluginSection))
15 2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	494 {
2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	495 return nullptr;
2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	496 }
2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	497
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	498 std::string bucketName;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	499 std::string region;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	500 std::string accessKey;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	501 std::string secretKey;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	502
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	503 if (!pluginSection.LookupStringValue(bucketName, "BucketName"))
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	504 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	505 LOG(ERROR) << "AwsS3Storage/BucketName configuration missing. Unable to initialize plugin";
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	506 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	507 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	508
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	509 if (!pluginSection.LookupStringValue(region, "Region"))
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	510 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	511 LOG(ERROR) << "AwsS3Storage/Region configuration missing. Unable to initialize plugin";
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	512 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	513 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	514
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	515 const std::string endpoint = pluginSection.GetStringValue("Endpoint", "");
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	516 const unsigned int connectTimeout = pluginSection.GetUnsignedIntegerValue("ConnectTimeout", 30);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	517 const unsigned int requestTimeout = pluginSection.GetUnsignedIntegerValue("RequestTimeout", 1200);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	518 const bool virtualAddressing = pluginSection.GetBooleanValue("VirtualAddressing", true);
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	519 const bool enableAwsSdkLogs = pluginSection.GetBooleanValue("EnableAwsSdkLogs", false);
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	520 const std::string caFile = orthancConfig.GetStringValue("HttpsCACertificates", "");
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	521
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	522
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	523 api_.reset(new Aws::Crt::ApiHandle);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	524
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	525 sdkOptions_.reset(new Aws::SDKOptions);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	526 sdkOptions_->cryptoOptions.initAndCleanupOpenSSL = false; // Done by the Orthanc framework
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	527 sdkOptions_->httpOptions.initAndCleanupCurl = false; // Done by the Orthanc framework
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	528
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	529 if (enableAwsSdkLogs)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	530 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	531 // Set up logging
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	532 Aws::Utils::Logging::InitializeAWSLogging(Aws::MakeShared<AwsOrthancLogger>(ALLOCATION_TAG));
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	533 // strangely, this seems to disable logging !!!! sdkOptions_->loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	534 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	535
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	536 Aws::InitAPI(*sdkOptions_);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	537
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	538
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	539 try
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	540 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	541 Aws::Client::ClientConfiguration configuration;
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	542
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	543 configuration.region = region.c_str();
6 393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	544 configuration.scheme = Aws::Http::Scheme::HTTPS;
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	545 configuration.connectTimeoutMs = connectTimeout * 1000;
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	546 configuration.requestTimeoutMs = requestTimeout * 1000;
27 e1f52b851827 Added "VirtualAddressing" configuration option in the AWS S3 plugin (for compatibility with minio) Sebastien Jodogne <s.jodogne@gmail.com> parents: 15 diff changeset	547 configuration.httpRequestTimeoutMs = requestTimeout * 1000;
6 393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	548
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	549 if (!endpoint.empty())
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	550 {
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	551 configuration.endpointOverride = endpoint.c_str();
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	552 }
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	553
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	554 if (!caFile.empty())
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	555 {
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	556 configuration.caFile = caFile;
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	557 }
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	558
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	559 bool useTransferManager = false; // new in v 2.3.0
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	560 unsigned int transferPoolSize = 10;
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	561 unsigned int transferBufferSizeMB = 5;
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	562
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	563 pluginSection.LookupBooleanValue(useTransferManager, "UseTransferManager");
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	564 pluginSection.LookupUnsignedIntegerValue(transferPoolSize, "TransferPoolSize");
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	565 pluginSection.LookupUnsignedIntegerValue(transferBufferSizeMB, "TransferBufferSize");
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	566
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	567
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	568 std::shared_ptr<Aws::S3::S3Client> client;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	569
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	570 if (pluginSection.LookupStringValue(accessKey, "AccessKey") && pluginSection.LookupStringValue(secretKey, "SecretKey"))
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	571 {
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	572 LOG(INFO) << "AWS S3 Storage: using credentials from the configuration file";
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	573 Aws::Auth::AWSCredentials credentials(accessKey.c_str(), secretKey.c_str());
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	574
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	575 client = Aws::MakeShared<Aws::S3::S3Client>(ALLOCATION_TAG, credentials, configuration, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, virtualAddressing);
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	576 }
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	577 else
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	578 {
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	579 // when using default credentials, credentials are not checked at startup but only the first time you try to access the bucket !
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	580 LOG(INFO) << "AWS S3 Storage: using default credentials provider";
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	581 client = Aws::MakeShared<Aws::S3::S3Client>(ALLOCATION_TAG, configuration, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, virtualAddressing);
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	582 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	583
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	584 LOG(INFO) << "AWS S3 storage initialized";
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	585
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	586 return new AwsS3StoragePlugin(nameForLogs, client, bucketName, enableLegacyStorageStructure, storageContainsUnknownFiles, useTransferManager, transferPoolSize, transferBufferSizeMB);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	587 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	588 catch (const std::exception& e)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	589 {
151 00cd1f01dd5d fix initialization and finalization Sebastien Jodogne <s.jodogne@gmail.com> parents: 145 diff changeset	590 Aws::ShutdownAPI(*sdkOptions_);
152 d62f52be1943 use Orthanc frameworking for logging Sebastien Jodogne <s.jodogne@gmail.com> parents: 151 diff changeset	591 LOG(ERROR) << "AWS S3 Storage plugin: failed to initialize plugin: " << e.what();
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	592 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	593 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	594 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	595
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	596
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	597 AwsS3StoragePlugin::~AwsS3StoragePlugin()
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	598 {
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	599 assert(sdkOptions_.get() != NULL);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	600 Aws::ShutdownAPI(*sdkOptions_);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	601 api_.reset();
151 00cd1f01dd5d fix initialization and finalization Sebastien Jodogne <s.jodogne@gmail.com> parents: 145 diff changeset	602 sdkOptions_.reset();
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	603 }
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	604
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	605
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	606 AwsS3StoragePlugin::AwsS3StoragePlugin(const std::string& nameForLogs,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	607 std::shared_ptr<Aws::S3::S3Client> client,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	608 const std::string& bucketName,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	609 bool enableLegacyStorageStructure,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	610 bool storageContainsUnknownFiles,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	611 bool useTransferManager,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	612 unsigned int transferThreadPoolSize,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	613 unsigned int transferBufferSizeMB)
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	614 : BaseStorage(nameForLogs, enableLegacyStorageStructure),
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	615 bucketName_(bucketName),
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	616 storageContainsUnknownFiles_(storageContainsUnknownFiles),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	617 useTransferManager_(useTransferManager),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	618 client_(client)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	619 {
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	620 if (useTransferManager_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	621 {
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	622 executor_ = Aws::MakeShared<Aws::Utils::Threading::PooledThreadExecutor>(ALLOCATION_TAG, transferThreadPoolSize);
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	623 Aws::Transfer::TransferManagerConfiguration transferConfig(executor_.get());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	624 transferConfig.s3Client = client_;
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	625 transferConfig.bufferSize = static_cast<uint64_t>(transferBufferSizeMB) * 1024 * 1024;
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	626 transferConfig.transferBufferMaxHeapSize = static_cast<uint64_t>(transferBufferSizeMB) * 1024 * 1024 * transferThreadPoolSize;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	627
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	628 transferManager_ = Aws::Transfer::TransferManager::Create(transferConfig);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	629 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	630 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	631
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	632 IStorage::IWriter* AwsS3StoragePlugin::GetWriterForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	633 {
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	634 if (useTransferManager_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	635 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	636 return new TransferWriter(transferManager_, bucketName_, GetPath(uuid, type, encryptionEnabled));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	637 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	638 else
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	639 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	640 return new DirectWriter(client_, bucketName_, GetPath(uuid, type, encryptionEnabled));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	641 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	642 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	643
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	644 IStorage::IReader* AwsS3StoragePlugin::GetReaderForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	645 {
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	646 std::list<std::string> paths;
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	647 paths.push_back(GetPath(uuid, type, encryptionEnabled, false));
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	648 if (storageContainsUnknownFiles_)
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	649 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	650 paths.push_back(GetPath(uuid, type, encryptionEnabled, true));
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	651 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	652
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	653 if (useTransferManager_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	654 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	655 return new TransferReader(transferManager_, client_, bucketName_, paths, uuid);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	656 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	657 else
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	658 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	659 return new DirectReader(client_, bucketName_, paths, uuid);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	660 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	661 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	662
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	663 void AwsS3StoragePlugin::DeleteObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	664 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	665 std::string firstExceptionMessage;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	666
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	667 std::list<std::string> paths;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	668 paths.push_back(GetPath(uuid, type, encryptionEnabled, false));
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	669 if (storageContainsUnknownFiles_)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	670 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	671 paths.push_back(GetPath(uuid, type, encryptionEnabled, true));
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	672 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	673
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	674 // DeleteObject succeeds even if the file does not exist -> we need to try to delete every path
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	675 for (auto& path: paths)
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	676 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	677 Aws::S3::Model::DeleteObjectRequest deleteObjectRequest;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	678 deleteObjectRequest.SetBucket(bucketName_.c_str());
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	679 deleteObjectRequest.SetKey(path.c_str());
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	680
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	681 auto result = client_->DeleteObject(deleteObjectRequest);
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	682
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	683 if (!result.IsSuccess() && firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	684 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	685 firstExceptionMessage = std::string("error while deleting file ") + path + ": response code = " + boost::lexical_cast<std::string>((int)result.GetError().GetResponseCode()) + " " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str();
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	686 }
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	687 }
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	688
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	689 if (!firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	690 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	691 throw StoragePluginException(firstExceptionMessage);
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	692 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	693 }

Mercurial > hg > orthanc-object-storage

annotate Aws/AwsS3StoragePlugin.cpp @ 197:df0b6998fd27