orthanc-object-storage: Aws/AwsS3StoragePlugin.cpp annotate

annotate Aws/AwsS3StoragePlugin.cpp @ 119:e39aa88ec20e 2.3.0

AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations

author	Alain Mazy <am@osimis.io>
date	Mon, 20 Nov 2023 17:03:08 +0100
parents	78c075412ab4
children	addccb7ad900 3e9cced85a5b

rev	line source
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	1 /**
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	2 * Cloud storage plugins for Orthanc
37 f55b2afdf53d upgrade to year 2021 Sebastien Jodogne <s.jodogne@gmail.com> parents: 33 diff changeset	3 * Copyright (C) 2020-2021 Osimis S.A., Belgium
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	4 *
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	5 * This program is free software: you can redistribute it and/or
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	6 * modify it under the terms of the GNU Affero General Public License
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	7 * as published by the Free Software Foundation, either version 3 of
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	8 * the License, or (at your option) any later version.
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	9 *
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	10 * This program is distributed in the hope that it will be useful, but
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	11 * WITHOUT ANY WARRANTY; without even the implied warranty of
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	12 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	13 * Affero General Public License for more details.
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	14 *
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	15 * You should have received a copy of the GNU Affero General Public License
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	16 * along with this program. If not, see <http://www.gnu.org/licenses/>.
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	17 **/
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	18
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	19 #include "AwsS3StoragePlugin.h"
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	20 #include <Logging.h>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	21
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	22 #include <aws/core/Aws.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	23 #include <aws/s3/S3Client.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	24 #include <aws/s3/model/PutObjectRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	25 #include <aws/s3/model/GetObjectRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	26 #include <aws/s3/model/ListObjectsRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	27 #include <aws/s3/model/DeleteObjectRequest.h>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	28 #include <aws/core/auth/AWSCredentialsProvider.h>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	29 #include <aws/core/utils/HashingUtils.h>
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	30 #include <aws/core/utils/logging/DefaultLogSystem.h>
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	31 #include <aws/core/utils/logging/DefaultCRTLogSystem.h>
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	32 #include <aws/core/utils/logging/AWSLogging.h>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	33 #include <aws/core/utils/memory/stl/AWSStreamFwd.h>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	34 #include <aws/core/utils/memory/stl/AWSStringStream.h>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	35 #include <aws/core/utils/memory/AWSMemory.h>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	36 #include <aws/core/utils/stream/PreallocatedStreamBuf.h>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	37 #include <aws/core/utils/StringUtils.h>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	38 #include <aws/transfer/TransferManager.h>
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	39 #include <aws/crt/Api.h>
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	40 #include <iostream>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	41 #include <fstream>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	42
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	43 #include <boost/lexical_cast.hpp>
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	44 #include <boost/interprocess/streams/bufferstream.hpp>
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	45 #include <iostream>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	46 #include <fstream>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	47
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	48 const char* ALLOCATION_TAG = "OrthancS3";
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	49
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	50 class AwsS3StoragePlugin : public BaseStorage
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	51 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	52 public:
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	53
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	54 std::string bucketName_;
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	55 bool storageContainsUnknownFiles_;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	56 bool useTransferManager_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	57 std::shared_ptr<Aws::S3::S3Client> client_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	58 std::shared_ptr<Aws::Utils::Threading::Executor> executor_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	59 std::shared_ptr<Aws::Transfer::TransferManager> transferManager_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	60
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	61 public:
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	62
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	63 AwsS3StoragePlugin(const std::string& nameForLogs, std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, bool enableLegacyStorageStructure, bool storageContainsUnknownFiles, bool useTransferManager, unsigned int transferThreadPoolSize, unsigned int transferBufferSizeMB);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	64
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	65 virtual ~AwsS3StoragePlugin();
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	66
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	67 virtual IWriter* GetWriterForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	68 virtual IReader* GetReaderForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	69 virtual void DeleteObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled);
111 407bd022b0cf in /move-storage: now detecting if file should be moved or not Alain Mazy <am@osimis.io> parents: 107 diff changeset	70 virtual bool HasFileExists() {return false;};
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	71 };
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	72
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	73
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	74 class DirectWriter : public IStorage::IWriter
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	75 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	76 std::string path_;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	77 std::shared_ptr<Aws::S3::S3Client> client_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	78 std::string bucketName_;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	79
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	80 public:
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	81 DirectWriter(std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, const std::string& path)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	82 : path_(path),
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	83 client_(client),
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	84 bucketName_(bucketName)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	85 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	86 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	87
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	88 virtual ~DirectWriter()
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	89 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	90 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	91
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	92 virtual void Write(const char* data, size_t size)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	93 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	94 Aws::S3::Model::PutObjectRequest putObjectRequest;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	95
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	96 putObjectRequest.SetBucket(bucketName_.c_str());
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	97 putObjectRequest.SetKey(path_.c_str());
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	98
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	99 std::shared_ptr<Aws::StringStream> stream = Aws::MakeShared<Aws::StringStream>(ALLOCATION_TAG, std::ios_base::in \| std::ios_base::binary);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	100
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	101 stream->rdbuf()->pubsetbuf(const_cast<char*>(data), size);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	102 stream->rdbuf()->pubseekpos(size);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	103 stream->seekg(0);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	104
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	105 putObjectRequest.SetBody(stream);
74 a25b4140e7e4 AWS: added the content MD5 in the request when writing Alain Mazy <am@osimis.io> parents: 70 diff changeset	106 putObjectRequest.SetContentMD5(Aws::Utils::HashingUtils::Base64Encode(Aws::Utils::HashingUtils::CalculateMD5(*stream)));
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	107
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	108 auto result = client_->PutObject(putObjectRequest);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	109
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	110 if (!result.IsSuccess())
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	111 {
62 d10696f94959 more detailed AWS errors Alain Mazy <am@osimis.io> parents: 57 diff changeset	112 throw StoragePluginException(std::string("error while writing file ") + path_ + ": response code = " + boost::lexical_cast<std::string>((int)result.GetError().GetResponseCode()) + " " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	113 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	114 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	115 };
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	116
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	117
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	118 class DirectReader : public IStorage::IReader
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	119 {
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	120 protected:
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	121 std::shared_ptr<Aws::S3::S3Client> client_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	122 std::string bucketName_;
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	123 std::list<std::string> paths_;
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	124 std::string uuid_;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	125
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	126 public:
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	127 DirectReader(std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, const std::list<std::string>& paths, const char* uuid)
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	128 : client_(client),
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	129 bucketName_(bucketName),
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	130 paths_(paths),
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	131 uuid_(uuid)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	132 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	133 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	134
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	135 virtual ~DirectReader()
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	136 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	137
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	138 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	139
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	140 virtual size_t GetSize()
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	141 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	142 std::string firstExceptionMessage;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	143
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	144 for (auto& path: paths_)
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	145 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	146 try
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	147 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	148 return _GetSize(path);
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	149 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	150 catch (StoragePluginException& ex)
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	151 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	152 if (firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	153 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	154 firstExceptionMessage = ex.what();
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	155 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	156 //ignore to retry
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	157 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	158 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	159 throw StoragePluginException(firstExceptionMessage);
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	160 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	161
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	162 virtual void ReadWhole(char* data, size_t size)
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	163 {
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	164 _Read(data, size, 0, false);
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	165 }
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	166
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	167 virtual void ReadRange(char* data, size_t size, size_t fromOffset)
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	168 {
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	169 _Read(data, size, fromOffset, true);
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	170 }
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	171
70 16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	172 private:
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	173
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	174 size_t _GetSize(const std::string& path)
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	175 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	176 Aws::S3::Model::ListObjectsRequest listObjectRequest;
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	177 listObjectRequest.SetBucket(bucketName_.c_str());
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	178 listObjectRequest.SetPrefix(path.c_str());
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	179
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	180 auto result = client_->ListObjects(listObjectRequest);
70 16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	181
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	182 if (result.IsSuccess())
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	183 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	184 Aws::Vector<Aws::S3::Model::Object> objectList =
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	185 result.GetResult().GetContents();
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	186
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	187 if (objectList.size() == 1)
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	188 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	189 return objectList[0].GetSize();
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	190 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	191 else if (objectList.size() > 1)
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	192 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	193 throw StoragePluginException(std::string("error while reading file ") + path + ": multiple objet with same name !");
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	194 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	195 throw StoragePluginException(std::string("error while reading file ") + path + ": object not found !");
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	196 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	197 else
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	198 {
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	199 throw StoragePluginException(std::string("error while reading file ") + path + ": " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str());
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	200 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	201 }
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	202
16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	203
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	204 void _Read(char* data, size_t size, size_t fromOffset, bool useRange)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	205 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	206 std::string firstExceptionMessage;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	207
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	208 for (auto& path: paths_)
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	209 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	210 try
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	211 {
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	212 return __Read(path, data, size, fromOffset, useRange);
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	213 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	214 catch (StoragePluginException& ex)
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	215 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	216 if (firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	217 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	218 firstExceptionMessage = ex.what();
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	219 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	220 //ignore to retry
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	221 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	222 }
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	223 throw StoragePluginException(firstExceptionMessage);
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	224 }
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	225
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	226 void __Read(const std::string& path, char* data, size_t size, size_t fromOffset, bool useRange)
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	227 {
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	228 Aws::S3::Model::GetObjectRequest getObjectRequest;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	229 getObjectRequest.SetBucket(bucketName_.c_str());
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	230 getObjectRequest.SetKey(path.c_str());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	231
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	232 if (useRange)
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	233 {
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	234 // https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	235 std::string range = std::string("bytes=") + boost::lexical_cast<std::string>(fromOffset) + "-" + boost::lexical_cast<std::string>(fromOffset + size -1);
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	236 getObjectRequest.SetRange(range.c_str());
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	237 }
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	238
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	239 getObjectRequest.SetResponseStreamFactory(
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	240 [data, size]()
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	241 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	242 std::unique_ptr<Aws::StringStream>
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	243 istream(Aws::New<Aws::StringStream>(ALLOCATION_TAG));
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	244
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	245 istream->rdbuf()->pubsetbuf(static_cast<char*>(data),
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	246 size);
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	247
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	248 return istream.release();
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	249 });
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	250
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	251 // Get the object
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	252 auto result = client_->GetObject(getObjectRequest);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	253 if (result.IsSuccess())
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	254 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	255 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	256 else
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	257 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	258 throw StoragePluginException(std::string("error while reading file ") + path + ": response code = " + boost::lexical_cast<std::string>((int)result.GetError().GetResponseCode()) + " " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	259 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	260 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	261
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	262 };
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	263
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	264
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	265
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	266 class TransferWriter : public IStorage::IWriter
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	267 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	268 std::string path_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	269 std::shared_ptr<Aws::Transfer::TransferManager> transferManager_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	270 std::string bucketName_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	271
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	272 public:
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	273 TransferWriter(std::shared_ptr<Aws::Transfer::TransferManager> transferManager, const std::string& bucketName, const std::string& path)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	274 : path_(path),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	275 transferManager_(transferManager),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	276 bucketName_(bucketName)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	277 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	278 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	279
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	280 virtual ~TransferWriter()
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	281 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	282 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	283
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	284 virtual void Write(const char* data, size_t size)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	285 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	286 boost::interprocess::bufferstream buffer(const_cast<char>(static_cast<const char>(data)), static_cast<size_t>(size));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	287 std::shared_ptr<Aws::IOStream> body = Aws::MakeShared<Aws::IOStream>(ALLOCATION_TAG, buffer.rdbuf());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	288
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	289 std::shared_ptr<Aws::Transfer::TransferHandle> transferHandle = transferManager_->UploadFile(body, bucketName_, path_.c_str(), "application/binary", Aws::Map<Aws::String, Aws::String>());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	290 transferHandle->WaitUntilFinished();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	291
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	292 if (transferHandle->GetStatus() != Aws::Transfer::TransferStatus::COMPLETED)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	293 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	294 throw StoragePluginException(std::string("error while writing file ") + path_ + ": response code = " + boost::lexical_cast<std::string>(static_cast<int>(transferHandle->GetLastError().GetResponseCode())) + " " + transferHandle->GetLastError().GetMessage());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	295 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	296 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	297 };
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	298
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	299
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	300 class TransferReader : public DirectReader
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	301 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	302 std::shared_ptr<Aws::Transfer::TransferManager> transferManager_;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	303
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	304 public:
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	305 TransferReader(std::shared_ptr<Aws::Transfer::TransferManager> transferManager, std::shared_ptr<Aws::S3::S3Client> client, const std::string& bucketName, const std::list<std::string>& paths, const char* uuid)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	306 : DirectReader(client, bucketName, paths, uuid),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	307 transferManager_(transferManager)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	308 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	309 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	310
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	311 virtual ~TransferReader()
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	312 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	313
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	314 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	315
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	316 virtual void ReadWhole(char* data, size_t size)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	317 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	318 std::string firstExceptionMessage;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	319
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	320 for (auto& path: paths_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	321 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	322 try
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	323 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	324 // The local variable 'streamBuffer' is captured by reference in a lambda.
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	325 // It must persist until all downloading by the 'transfer_manager' is complete.
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	326 Aws::Utils::Stream::PreallocatedStreamBuf streamBuffer(reinterpret_cast<unsigned char*>(data), size);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	327
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	328 std::shared_ptr<Aws::Transfer::TransferHandle> downloadHandler = transferManager_->DownloadFile(bucketName_, path, [&]() { //Define a lambda expression for the callback method parameter to stream back the data.
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	329 return Aws::New<Aws::IOStream>(ALLOCATION_TAG, &streamBuffer);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	330 });
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	331
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	332 downloadHandler->WaitUntilFinished();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	333
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	334 if (downloadHandler->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	335 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	336 return;
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	337 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	338 else if (firstExceptionMessage.empty())
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	339 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	340 firstExceptionMessage = downloadHandler->GetLastError().GetMessage();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	341 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	342 // getObjectRequest.SetResponseStreamFactory(
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	343 // [data, size]()
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	344 // {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	345 // std::unique_ptr<Aws::StringStream>
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	346 // istream(Aws::New<Aws::StringStream>(ALLOCATION_TAG));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	347
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	348 // istream->rdbuf()->pubsetbuf(static_cast<char*>(data),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	349 // size);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	350
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	351 // return istream.release();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	352 // });
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	353
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	354
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	355 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	356 catch (StoragePluginException& ex)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	357 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	358 if (firstExceptionMessage.empty())
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	359 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	360 firstExceptionMessage = ex.what();
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	361 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	362 //ignore to retry
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	363 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	364 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	365 throw StoragePluginException(firstExceptionMessage);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	366 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	367
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	368 };
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	369
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	370
41 d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	371
d99afdf6d872 ReadRange for AWS Alain Mazy <am@osimis.io> parents: 37 diff changeset	372
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	373 const char* AwsS3StoragePluginFactory::GetStoragePluginName()
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	374 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	375 return "AWS S3 Storage";
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	376 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	377
94 1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	378 const char* AwsS3StoragePluginFactory::GetStorageDescription()
1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	379 {
1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	380 return "Stores the Orthanc storage area in AWS S3";
1bc055199cd2 Added a description for all plugins Alain Mazy <am@osimis.io> parents: 78 diff changeset	381 }
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	382
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	383 static std::unique_ptr<Aws::Crt::ApiHandle> api_;
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	384 static std::unique_ptr<Aws::SDKOptions> sdkOptions_;
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	385
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	386 #include <stdarg.h>
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	387
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	388 class AwsOrthancLogger : public Aws::Utils::Logging::LogSystemInterface
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	389 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	390 public:
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	391 virtual ~AwsOrthancLogger() {}
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	392
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	393 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	394 * Gets the currently configured log level for this logger.
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	395 */
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	396 virtual Aws::Utils::Logging::LogLevel GetLogLevel() const
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	397 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	398 return Aws::Utils::Logging::LogLevel::Trace;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	399 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	400 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	401 * Does a printf style output to the output stream. Don't use this, it's unsafe. See LogStream
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	402 */
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	403 virtual void Log(Aws::Utils::Logging::LogLevel logLevel, const char* tag, const char* formatStr, ...)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	404 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	405 Aws::StringStream ss;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	406
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	407 va_list args;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	408 va_start(args, formatStr);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	409
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	410 va_list tmp_args; //unfortunately you cannot consume a va_list twice
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	411 va_copy(tmp_args, args); //so we have to copy it
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	412 #ifdef _WIN32
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	413 const int requiredLength = _vscprintf(formatStr, tmp_args) + 1;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	414 #else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	415 const int requiredLength = vsnprintf(nullptr, 0, formatStr, tmp_args) + 1;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	416 #endif
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	417 va_end(tmp_args);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	418
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	419 char outputBuff[requiredLength];
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	420 #ifdef _WIN32
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	421 vsnprintf_s(outputBuff, requiredLength, _TRUNCATE, formatStr, args);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	422 #else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	423 vsnprintf(outputBuff, requiredLength, formatStr, args);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	424 #endif // _WIN32
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	425
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	426 if (logLevel == Aws::Utils::Logging::LogLevel::Debug \|\| logLevel == Aws::Utils::Logging::LogLevel::Trace)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	427 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	428 LOG(INFO) << reinterpret_cast<const char*>(&outputBuff[0]);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	429 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	430 else if (logLevel == Aws::Utils::Logging::LogLevel::Warn)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	431 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	432 LOG(WARNING) << reinterpret_cast<const char*>(&outputBuff[0]);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	433 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	434 else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	435 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	436 LOG(ERROR) << reinterpret_cast<const char*>(&outputBuff[0]);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	437 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	438
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	439 va_end(args);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	440 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	441 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	442 * Writes the stream to the output stream.
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	443 */
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	444 virtual void LogStream(Aws::Utils::Logging::LogLevel logLevel, const char* tag, const Aws::OStringStream &messageStream)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	445 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	446 if (logLevel == Aws::Utils::Logging::LogLevel::Debug \|\| logLevel == Aws::Utils::Logging::LogLevel::Trace)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	447 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	448 LOG(INFO) << tag << messageStream.str();
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	449 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	450 else if (logLevel == Aws::Utils::Logging::LogLevel::Warn)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	451 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	452 LOG(WARNING) << tag << messageStream.str();
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	453 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	454 else
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	455 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	456 LOG(ERROR) << tag << messageStream.str();
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	457 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	458
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	459 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	460 /**
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	461 * Writes any buffered messages to the underlying device if the logger supports buffering.
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	462 */
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	463 virtual void Flush() {}
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	464 };
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	465
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	466 IStorage* AwsS3StoragePluginFactory::CreateStorage(const std::string& nameForLogs, const OrthancPlugins::OrthancConfiguration& orthancConfig)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	467 {
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	468 if (sdkOptions_.get() != NULL)
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	469 {
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	470 throw Orthanc::OrthancException(Orthanc::ErrorCode_BadSequenceOfCalls, "Cannot initialize twice");
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	471 }
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	472
15 2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	473 bool enableLegacyStorageStructure;
70 16e419fe80c5 Google & Azure: Added the "EnableLegacyUnknownFiles" configuration option Alain Mazy <am@osimis.io> parents: 68 diff changeset	474 bool storageContainsUnknownFiles;
15 2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	475
77 80792bb9600e new HybridMode Alain Mazy <am@osimis.io> parents: 74 diff changeset	476 if (!orthancConfig.IsSection(GetConfigurationSectionName()))
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	477 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	478 OrthancPlugins::LogWarning(std::string(GetStoragePluginName()) + " plugin, section missing. Plugin is not enabled.");
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	479 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	480 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	481
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	482 OrthancPlugins::OrthancConfiguration pluginSection;
77 80792bb9600e new HybridMode Alain Mazy <am@osimis.io> parents: 74 diff changeset	483 orthancConfig.GetSection(pluginSection, GetConfigurationSectionName());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	484
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	485 if (!BaseStorage::ReadCommonConfiguration(enableLegacyStorageStructure, storageContainsUnknownFiles, pluginSection))
15 2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	486 {
2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	487 return nullptr;
2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	488 }
2a02b21f0a19 migration + storage structure Alain Mazy parents: 6 diff changeset	489
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	490 std::string bucketName;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	491 std::string region;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	492 std::string accessKey;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	493 std::string secretKey;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	494
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	495 if (!pluginSection.LookupStringValue(bucketName, "BucketName"))
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	496 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	497 OrthancPlugins::LogError("AwsS3Storage/BucketName configuration missing. Unable to initialize plugin");
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	498 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	499 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	500
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	501 if (!pluginSection.LookupStringValue(region, "Region"))
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	502 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	503 OrthancPlugins::LogError("AwsS3Storage/Region configuration missing. Unable to initialize plugin");
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	504 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	505 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	506
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	507 const std::string endpoint = pluginSection.GetStringValue("Endpoint", "");
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	508 const unsigned int connectTimeout = pluginSection.GetUnsignedIntegerValue("ConnectTimeout", 30);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	509 const unsigned int requestTimeout = pluginSection.GetUnsignedIntegerValue("RequestTimeout", 1200);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	510 const bool virtualAddressing = pluginSection.GetBooleanValue("VirtualAddressing", true);
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	511 const bool enableAwsSdkLogs = pluginSection.GetBooleanValue("EnableAwsSdkLogs", false);
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	512 const std::string caFile = orthancConfig.GetStringValue("HttpsCACertificates", "");
119 e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	513
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	514
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	515 api_.reset(new Aws::Crt::ApiHandle);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	516
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	517 sdkOptions_.reset(new Aws::SDKOptions);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	518 sdkOptions_->cryptoOptions.initAndCleanupOpenSSL = false; // Done by the Orthanc framework
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	519 sdkOptions_->httpOptions.initAndCleanupCurl = false; // Done by the Orthanc framework
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	520
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	521 if (enableAwsSdkLogs)
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	522 {
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	523 // Set up logging
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	524 Aws::Utils::Logging::InitializeAWSLogging(Aws::MakeShared<AwsOrthancLogger>(ALLOCATION_TAG));
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	525 // strangely, this seems to disable logging !!!! sdkOptions_->loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	526 }
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	527
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	528 Aws::InitAPI(*sdkOptions_);
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	529
e39aa88ec20e AWS: new 'EnableAwsSdkLogs' config + show timings for read/write operations Alain Mazy <am@osimis.io> parents: 113 diff changeset	530
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	531 try
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	532 {
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	533 Aws::Client::ClientConfiguration configuration;
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	534
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	535 configuration.region = region.c_str();
6 393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	536 configuration.scheme = Aws::Http::Scheme::HTTPS;
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	537 configuration.connectTimeoutMs = connectTimeout * 1000;
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	538 configuration.requestTimeoutMs = requestTimeout * 1000;
27 e1f52b851827 Added "VirtualAddressing" configuration option in the AWS S3 plugin (for compatibility with minio) Sebastien Jodogne <s.jodogne@gmail.com> parents: 15 diff changeset	539 configuration.httpRequestTimeoutMs = requestTimeout * 1000;
6 393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	540
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	541 if (!endpoint.empty())
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	542 {
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	543 configuration.endpointOverride = endpoint.c_str();
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	544 }
393fcf337462 AWS: added 3 configurations: Endpoint, ConnectionTimeout, RequestTimeout Alain Mazy parents: 1 diff changeset	545
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	546 if (!caFile.empty())
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	547 {
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	548 configuration.caFile = caFile;
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	549 }
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	550
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	551 bool useTransferManager = false; // new in v 2.3.0
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	552 unsigned int transferPoolSize = 10;
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	553 unsigned int transferBufferSizeMB = 5;
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	554
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	555 pluginSection.LookupBooleanValue(useTransferManager, "UseTransferManager");
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	556 pluginSection.LookupUnsignedIntegerValue(transferPoolSize, "TransferPoolSize");
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	557 pluginSection.LookupUnsignedIntegerValue(transferBufferSizeMB, "TransferBufferSize");
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	558
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	559
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	560 std::shared_ptr<Aws::S3::S3Client> client;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	561
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	562 if (pluginSection.LookupStringValue(accessKey, "AccessKey") && pluginSection.LookupStringValue(secretKey, "SecretKey"))
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	563 {
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	564 OrthancPlugins::LogInfo("AWS S3 Storage: using credentials from the configuration file");
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	565 Aws::Auth::AWSCredentials credentials(accessKey.c_str(), secretKey.c_str());
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	566
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	567 client = Aws::MakeShared<Aws::S3::S3Client>(ALLOCATION_TAG, credentials, configuration, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, virtualAddressing);
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	568 }
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	569 else
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	570 {
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	571 // when using default credentials, credentials are not checked at startup but only the first time you try to access the bucket !
1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	572 OrthancPlugins::LogInfo("AWS S3 Storage: using default credentials provider");
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	573 client = Aws::MakeShared<Aws::S3::S3Client>(ALLOCATION_TAG, configuration, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, virtualAddressing);
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	574 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	575
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	576 OrthancPlugins::LogInfo("AWS S3 storage initialized");
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	577
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	578 return new AwsS3StoragePlugin(nameForLogs, client, bucketName, enableLegacyStorageStructure, storageContainsUnknownFiles, useTransferManager, transferPoolSize, transferBufferSizeMB);
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	579 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	580 catch (const std::exception& e)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	581 {
53 1c3e34f5c5c6 AWS S3: if no access & secret keys are provided, now getting the credentials from the default credentials manager Alain Mazy <am@osimis.io> parents: 41 diff changeset	582 OrthancPlugins::LogError(std::string("AWS S3 Storage plugin: failed to initialize plugin: ") + e.what());
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	583 return nullptr;
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	584 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	585
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	586 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	587
57 ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	588
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	589 AwsS3StoragePlugin::~AwsS3StoragePlugin()
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	590 {
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	591 assert(sdkOptions_.get() != NULL);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	592 Aws::ShutdownAPI(*sdkOptions_);
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	593 api_.reset();
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	594 }
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	595
ba1be668e475 fix initialization of the aws static library Sebastien Jodogne <s.jodogne@gmail.com> parents: 53 diff changeset	596
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	597 AwsS3StoragePlugin::AwsS3StoragePlugin(const std::string& nameForLogs,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	598 std::shared_ptr<Aws::S3::S3Client> client,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	599 const std::string& bucketName,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	600 bool enableLegacyStorageStructure,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	601 bool storageContainsUnknownFiles,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	602 bool useTransferManager,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	603 unsigned int transferThreadPoolSize,
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	604 unsigned int transferBufferSizeMB)
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	605 : BaseStorage(nameForLogs, enableLegacyStorageStructure),
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	606 bucketName_(bucketName),
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	607 storageContainsUnknownFiles_(storageContainsUnknownFiles),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	608 useTransferManager_(useTransferManager),
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	609 client_(client)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	610 {
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	611 if (useTransferManager_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	612 {
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	613 executor_ = Aws::MakeShared<Aws::Utils::Threading::PooledThreadExecutor>(ALLOCATION_TAG, transferThreadPoolSize);
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	614 Aws::Transfer::TransferManagerConfiguration transferConfig(executor_.get());
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	615 transferConfig.s3Client = client_;
113 78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	616 transferConfig.bufferSize = static_cast<uint64_t>(transferBufferSizeMB) * 1024 * 1024;
78c075412ab4 more config for S3 transfer mode Alain Mazy <am@osimis.io> parents: 111 diff changeset	617 transferConfig.transferBufferMaxHeapSize = static_cast<uint64_t>(transferBufferSizeMB) * 1024 * 1024 * transferThreadPoolSize;
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	618
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	619 transferManager_ = Aws::Transfer::TransferManager::Create(transferConfig);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	620 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	621 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	622
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	623 IStorage::IWriter* AwsS3StoragePlugin::GetWriterForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	624 {
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	625 if (useTransferManager_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	626 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	627 return new TransferWriter(transferManager_, bucketName_, GetPath(uuid, type, encryptionEnabled));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	628 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	629 else
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	630 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	631 return new DirectWriter(client_, bucketName_, GetPath(uuid, type, encryptionEnabled));
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	632 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	633 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	634
78 d7295e8678d7 renames Alain Mazy <am@osimis.io> parents: 77 diff changeset	635 IStorage::IReader* AwsS3StoragePlugin::GetReaderForObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	636 {
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	637 std::list<std::string> paths;
c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	638 paths.push_back(GetPath(uuid, type, encryptionEnabled, false));
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	639 if (storageContainsUnknownFiles_)
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	640 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	641 paths.push_back(GetPath(uuid, type, encryptionEnabled, true));
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	642 }
64 c4f56973a279 Fix reading/deleting DCM header files that were saved with plugin v 1.2.0 and Orthanc 1.9.3 which had a .unk extension Alain Mazy <am@osimis.io> parents: 62 diff changeset	643
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	644 if (useTransferManager_)
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	645 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	646 return new TransferReader(transferManager_, client_, bucketName_, paths, uuid);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	647 }
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	648 else
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	649 {
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	650 return new DirectReader(client_, bucketName_, paths, uuid);
c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	651 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	652 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	653
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	654 void AwsS3StoragePlugin::DeleteObject(const char* uuid, OrthancPluginContentType type, bool encryptionEnabled)
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	655 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	656 std::string firstExceptionMessage;
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	657
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	658 std::list<std::string> paths;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	659 paths.push_back(GetPath(uuid, type, encryptionEnabled, false));
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	660 if (storageContainsUnknownFiles_)
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	661 {
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	662 paths.push_back(GetPath(uuid, type, encryptionEnabled, true));
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	663 }
fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	664
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	665 // DeleteObject succeeds even if the file does not exist -> we need to try to delete every path
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	666 for (auto& path: paths)
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	667 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	668 Aws::S3::Model::DeleteObjectRequest deleteObjectRequest;
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	669 deleteObjectRequest.SetBucket(bucketName_.c_str());
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	670 deleteObjectRequest.SetKey(path.c_str());
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	671
106 c9356e42af99 added TransferMode for S3 (currently affected by https://github.com/aws/aws-sdk-cpp/issues/2319 since we are using version 1.9.45) Alain Mazy <am@osimis.io> parents: 94 diff changeset	672 auto result = client_->DeleteObject(deleteObjectRequest);
68 58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	673
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	674 if (!result.IsSuccess() && firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	675 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	676 firstExceptionMessage = std::string("error while deleting file ") + path + ": response code = " + boost::lexical_cast<std::string>((int)result.GetError().GetResponseCode()) + " " + result.GetError().GetExceptionName().c_str() + " " + result.GetError().GetMessage().c_str();
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	677 }
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	678 }
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	679
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	680 if (!firstExceptionMessage.empty())
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	681 {
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	682 throw StoragePluginException(firstExceptionMessage);
58a03fce4897 new option for AWS: EnableLegacyUnknownFiles Alain Mazy <am@osimis.io> parents: 64 diff changeset	683 }
1 fc26a8fc54d5 initial release Alain Mazy <alain@mazy.be> parents: diff changeset	684 }

Mercurial > hg > orthanc-object-storage

annotate Aws/AwsS3StoragePlugin.cpp @ 119:e39aa88ec20e 2.3.0