GetBytesTotalSize() < GetBytesTransferred() when uploading file to S3 #3329
Answered
by
sbiscigl
CuteCuteForever
asked this question in
Q&A
-
|
Hello. I had build the aws-sdk-cpp in macOS and tried to upload a 1mb.txt file to my personal S3 using transfer manager. The file did successfully upload to my S3. However, during assertion in the code, i am facing the uploadHandle->GetBytesTotalSize() is less than uploadHandle->GetBytesTransferred(). Below are my logs. Please advice... |
Beta Was this translation helpful? Give feedback.
Answered by
sbiscigl
Mar 7, 2025
Replies: 2 comments 7 replies
-
|
the main problem i see is that SSL connections are failing on several of your requests So it would appear that your cert is self signed. This is likely a issue with your enviornment. So following this stack overflow i would suggest addressing the issue by setting up a valid ssl cert. Or rather that is what i am gathering from your logs. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Does the extra bytes due to chunking ? or request header metadata ? |
Beta Was this translation helpful? Give feedback.
-
|
how are you running transfer manager? when run it like #include <aws/core/Aws.h>
#include <aws/core/utils/threading/PooledThreadExecutor.h>
#include <aws/transfer/TransferManager.h>
using namespace Aws;
using namespace Aws::Auth;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::Transfer;
using namespace Aws::Utils;
using namespace Aws::Utils::Threading;
namespace {
const char * LOG_TAG = "test-app";
}
auto main() -> int {
SDKOptions options{};
options.loggingOptions.logLevel = Logging::LogLevel::Debug;
InitAPI(options);
{
const auto executor = Aws::MakeUnique<PooledThreadExecutor>(LOG_TAG, std::thread::hardware_concurrency());
TransferManagerConfiguration configuration{executor.get()};
configuration.s3Client = Aws::MakeShared<S3Client>(LOG_TAG);
configuration.transferInitiatedCallback = [](const TransferManager*, const std::shared_ptr<const TransferHandle>&) {
std::cout << "started transfer\n";
};
const auto transferManager = TransferManager::Create(configuration);
transferManager->UploadDirectory("/some/dir/path", "bucket", "test-transfer", {});
transferManager->WaitUntilAllFinished();
std::cout << "finished transfer\n";
}
ShutdownAPI(options);
return 0;
}I see it complete as expected |
Beta Was this translation helpful? Give feedback.
-
|
Yes the code is similar. Can i double confirm that if the status is COMPLETED, do we need to adhere to the assert of
as per the example shown : https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/examples-s3-transfermanager.html ? |
Beta Was this translation helpful? Give feedback.
-
ah sorry, this is correct, it is due to the chunked encoding of the body. so the amount of bytes transferred should be the file size + size of chunked headers and trailer. so thae assertion that the uploaded file bytes == file size is no longer valid. "uploaded byes" will refer to the actual byte stream including the headers. so if we update the assertion to something like #include <aws/core/Aws.h>
#include <aws/core/utils/threading/PooledThreadExecutor.h>
#include <aws/transfer/TransferManager.h>
#include <fstream>
using namespace Aws;
using namespace Aws::Auth;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::Utils::Stream;
using namespace Aws::Transfer;
using namespace Aws::Utils;
using namespace Aws::Utils::Threading;
namespace {
const char * LOG_TAG = "test-app";
const char * BUCKET_NAME = "some-bucket-name";
const char * KEY_NAME = "tsome-key";
}
auto main() -> int {
SDKOptions options{};
options.loggingOptions.logLevel = Logging::LogLevel::Debug;
InitAPI(options);
{
const auto executor = Aws::MakeUnique<PooledThreadExecutor>(LOG_TAG, std::thread::hardware_concurrency());
TransferManagerConfiguration configuration{executor.get()};
configuration.s3Client = Aws::MakeShared<S3Client>(LOG_TAG);
configuration.transferInitiatedCallback = [](const TransferManager*, const std::shared_ptr<const TransferHandle>&) {
std::cout << "started transfer\n";
};
const auto transferManager = TransferManager::Create(configuration);
std::shared_ptr<IOStream> inputStream = Aws::MakeShared<Aws::StringStream>(LOG_TAG,
"1234567890");
const auto handle = transferManager->UploadFile(inputStream,
BUCKET_NAME,
KEY_NAME,
"text/plain",
{});
transferManager->WaitUntilAllFinished();
assert(handle->GetStatus() == Transfer::TransferStatus::COMPLETED);
/*
* Bytes transferred becomes the amount that is encoded in the stream for instance
* "1234567890"
* generates the stream
* "A\r\n1234567890\r\n0\r\nx-amz-checksum-crc64nvme:YCWlgjR7efM=\r\n\r\n"
*
* which is 59 bytes
*
*/
assert(59 == handle->GetBytesTransferred());
std::cout << "finished transfer\n";
}
ShutdownAPI(options);
return 0;
}So the assertion "technically isnt valid", but if you are able to precalculate the streaming amount of bytes added as a trailer, you can add a assertion. Let me update the documentation to reflect this |
Beta Was this translation helpful? Give feedback.
Answer selected by
sbiscigl
-
|
Thanks alot. Your advice is definitely helpful. Maybe you can update the examples too : https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/cpp/example_code/transfer-manager/transferOnStream.cpp One last question, on what condition does it use the different checksum algo ? Sometimes I see crc32 in the logs. Sometimes I see crc64navne |
Beta Was this translation helpful? Give feedback.
2 replies
-
yeah gunna start the process of updating those
in the transfer manager configuration you cant set what checksum you want currently it is default to CRC64 but has a typo in the comment. the checksums should match that configuration |
Beta Was this translation helpful? Give feedback.
-
|
thank sbiscigl for the response. One last question please, i am sending a 1mb file, why does it use chunking? I thought chunking only occur when file size is more than 5mb for it to happen ? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ah sorry, this is correct, it is due to the chunked encoding of the body. so the amount of bytes transferred should be the file size + size of chunked headers and trailer. so thae assertion that the uploaded file bytes == file size is no longer valid. "uploaded byes" will refer to the actual byte stream including the headers. so if we update the assertion to something like